reputation rank

Google has PageRank, and when I first started thinking about this, it seemed like they might have reputation rank too.

For example, consider a reputation system where you could vote for the trustworthiness of other people. Your vote would have a weight that was equivalent to the sum of the votes from other people towards you. In other words, your feedback is weighted by the feedback that people leave for you. But the feedback that people leave for you is weighted by the feedback that each of those people got from everyone else. The reputation system is recursive, and pretty much has to know the reputation rank of everyone before it can calculate the reputation rank of any particular person. ?? How would you get out of a infinite loop like that? Most likely, you'd have to snapshot reputation ranks occasionally and use old reputation ranks to help calculate new reputation ranks. Not perfect, but still pretty workable. And that's pretty much what Google does with PageRank, except it's ranking pages instead of people.

But then I began to wonder: should people have a single reputation rank that applies to the entire universe, or should reputation rank be a function of the relationship between you and the person you're interacting with? As a trite example, I trust George W. Bush a lot less than a lot of people. And it's possible that I trust him more than a lot of people too. The practical reputation rank for George W. Bush should not be the same for all people... otherwise I would not trust it enough to help me make decisions about who to trust. Even an average reputation rank would not work... I want my own. I want to know how George W. Bush's opinions/values/thoughts/etc line up with my own, not how they line up with the general country, or world. This significantly complicates the calculation of reputation rank for a large group.

As always, thinking about it in terms of software helps come to some conclusions. The calculation can't be impossible if our brains are capable of doing it without even thinking. A proof of concept script is probably in order here, one that encapsulates all the complexity of a real reputation system without trying to make it scale to thousands or billions of people and without trying to make it account for all things that reputation currently does.

Here's what I would like to build. If it's compelling enough, and has enough unanswered questions that only a prototype could answer, after I describe it, maybe I'll try to build part of it later today. If all the value seems to come out of the functional spec, so to speak, then I'll leave it at that.

Of course, there would have to be an account manager so that each user of the prototype could have their own space within which to fight for and compensate reputation.

There would have to be a way of making statements. "I have two cats," would be a statement. "There is a God," would be another. Metadata for these statements would include the person making the statement, the time it was made, and maybe even a unique ID so that all statements could be referred to unambiguously. Ideally, these statements would be structured in such a way that they could be stored with semantic meaning. Maybe "I believe there is a God," would work better for that. Statements could be broken up into subject/verb/object triplets and stored as RDF, then you could surf similar statements, and compare the frequency of a statement across the users... but I get ahead of myself. (Even better, statements could be translated into Lojban.)

Statements could be voted on by other users. Actually, votes should turn into new statements: "Erik's statement (I have two cats) is correct." That would mean that you agree that Erik has two cats. And you may say, "Erik's statement (I believe there is a God) is unverified." It may or may not be true, but the fact that Erik says it's true may reflect on his ability to only say correct statements (in your eyes). Other people may then make statements/votes about my original statements, or about the votes of other people on my statements.

Now we have a web of interrelated statements and people. How would you determine the reputation of a particular statement or person in relation to yourself?

It would be easy to determine this if you had actually voted on the reputation of the statement or person in question. But there are going to be tons of statements and people who you have not bothered to vote on explicitly. That's the difference between having a bookmark for "erik benson's weblog" and searching Google for "erik benson's weblog". A good reputation system would similarly give you the same answer "Erik has two cats" even if you had not yet explicitly voted on it yet.

Okay, so you're looking at the statement "Man Versus Himself is a book worth reading." This is a statement that is definitely going to require a reputation rank (and perhaps you've already given it one, using your brain's from-the-manufacturer installation). If you have already come to a conclusion, you could make a statement about it, "Erik's statement (Man Versus Himself is a book worth reading) is a malicious lie." But, if you don't yet know whether or not it's true, this is when the reputation rank's system could show its strength.

First, you would gather together all of the votes/statements that have been made about that statement. Then, for each of those statements, you would look at the person who had made the statement and calculate their reputation rank in relation to you. Their reputation rank would be determined by comparing their votes with your own. Identical votes will improve the reputation score between you two, contradicting votes will decrease the reputation score. The resulting reputation rank will tell you how much you agree with this person about statements of any kind, and it would result in a percentage chance that you will agree with things they say in the future. The result would be a collection of probabilities that you will agree with each person's next statement. Their next statement, in this case, happens to be about whether or not they agree with "Man Versus Himself is a book worth reading." Those who you have a large chance of agreeing with will have more sway on the reputation rank of that statement than those whom you have closer to a 50% chance of agreeing with. If you have a 10% chance of agreeing with someone, and they say that the book is worth reading, that would push the reputation rank of that statement in the opposite direction.

Finally, after doing some math (which I can't exactly figure out right now... maybe Jeff could help out), you have a reputation rank for that statement. That's a lot of work to get a reputation rank for a single statement... most likely if this were done in a prototype, it would have to do a lot of these calculations offline in a build so that you didn't have to sit there and wait forever to get an answer. Then again, if you had 55,000 servers helping to compute the answers for you on the fly, I think a run-time answer would be worth it.

Am I overlooking anything in this very informal functional spec? Will it create reputation of the same quality that our brains do? That's what I'm really after. Sure, there are probably ways to take shortcuts here and get a good approximation (so that the end result is actually buildable without a huge investment) but right now I'd rather limit the scope of the prototype (by creating a system that would have scaling troubles) than the functionality/quality of the prototype.

One thing I just realized I overlooked is reputation rank scope. I might actually have several different reputation ranks for any given person... one for how trustworthy they are on the subject of politics, one for how trustworthy they are on the subject of plant care, one for how trustworhty they are on the subject of writing Perl code, etc. To be compatible with this, we'd need to know the scope that each statement is made in (perhaps another piece of metadata), and when calculating reputation rank we would only consider votes made within that scope. Or maybe we should just give votes within that scope higher weights, just in case there aren't enough to get an accurate score... and backfill it with each person's general reputation from all scopes.

Comments

[ade]

I was actually going to point you towards Advogato and it's trust metric (http://www.advogato.org/trust-metric.html) but a little time on Google revealed you'd mentioned it last year (http://erikbenson.com/index.cgi?node=web%20of%20trust).

So instead take a look at the following discussions:
http://www.advogato.org/article/554.html
http://www.advogato.org/person/danf/diary.html?start=2
http://sccm.stanford.edu/pub/sccm/sccm02-16.pdf
http://www.advogato.org/person/ade/diary.html?start=6

You should also consider that our brains tend to calculate trust with ourselves as the seed. Everybody else's trustworthiness then gets calculated by traversing the graph from ourselves to them.

[Andrej Gregov]

The non tech point
Less feedback on implementation and more on what a reputation system needs to work. Google's pagerank works because it requires no participation from users. They just click around and Google uses this data to create meaningful search results. Amazon does the same thing with personalization. They just watch what people do and then look for patterns. What you're describing involves friction. People have to pysically do something to tell you something about a person. This introduces all sorts of bias in your reputation rank. Of course you could say that a reputation is subjective anyway. But still, the big question for success to me would be: 1) why would people us it? and 2) would it be friction less enough for people to use regularily so that you'll get enough data for a meaningful rank? Quick thoughts :-).

[Erik Benson]

Yeah, I completely agree. I was using the software metaphor more as a way to try to imagine what the ideal system would be. In order to build a truly compelling website that would encourage usage of it and eventually be a success, however, a lot of other factors would need to be taken into consideration.

One interesting point here, though, is that I think a lot of this can be approximated in HTML, as Google has done with PageRank. Also, there are a lot of pseudo-reputation ranks out there already... all of them specific to the site which they're providing reputation for. Amazon has one, eBay has one, slashdot has one, etc. People seem to know what reputation is and are willing to invest themselves in preserving their reputation within these contexts. If someone could create a system that was then adopted by these specific websites, and the reputation was independant from the site, I think that would be one way to bootstrap usage of the system. It might require that someone build the reputation system and create one compelling usage for it... while also offering it up for others to build off of and contribute to. I haven't given it much more thought than that though.

[Andrej Gregov]

Perhaps a first version could some how come up with a reputation score for folks by summing pseudo-reputation ranks from Amazon, Ebay and the like into some overall score. Call it their Internet Rank or iRank (or maybe iRep). This might be your baseline rank and the rest of the stuff you discuss could be optional ways of building a more complete score for users that care to "play." I could see this sort of concept interesting to sites like Amazon/Ebay because they get easy access to competitor data that could add legitimate value to their sites. Just an idea.

[adam]

cory doctorow wrote about a system very similar, "whuffie", in his second newest book, down and out in the magic kindom

[Lucas Fletcher]

Reality Check
If the end goal of a reputation system is not just to establish a personalized trust metric for people but for statements as well (as you describe) then I think it is useful to differentiate between object, verifiable statements and statements of opinion or taste. The former falls more squarely in the purvue of a reputation system, while the latter is more like collaborative filtering.

Interestingly, if you subscribe to the viewpoint that most opinions about the world (say, political opinions) are determined as much or more by the belief in a given subset of the underlying facts surrounding it as by ones values, then a reputation system relating to belief in objective facts can serve to strengthen the confidence of collaborative filtering predictions of opinion.

But be careful not to conflate subjective statements and objective facts as you seem to have done, or else you sound like one of those wishy-washy extreme subjectivist. :)

[Erik Benson]

I am a wishy-washy extreme subjectivist (if I understand the label correctly). I don't believe there is a concrete difference (practically, maybe if you had a real behind-the-scenes knowledge of statements) between subjective statements and objective facts. In other words, I can say "my cat is green" and that might be an opinion or a statement of truth, but you have no way of knowing other than by relying on your level of trust for me.

[Lucas Fletcher]

Good point. Okay then, to ground the reputation rankings, instead of the Page Rank-like snapshot, how about a list of "statements of fact" that you believe to be true? Collaborative filtering could then be used to compare your ground with others, and another metric can relate these statements of fact with correlated statements of opinion. Back when people actually had iterated interactions with one another in a close-knit community environment, reputation was definetely a "behind-the-scenes" verifiable attribute. This was because you could actually witness the results of their handiwork, after-the-fact. Without something akin to this eye-witness accounting, a reputation system would simply magnify people's ignorance and bias along with everything else.

I guess I have always thought some sort of objective grounding was necessary because of my observation that people are easily misled, in that they have a tendency to be swayed by the personality, popularity, and charisma of the individual while glossing over the content of what that person says.