Science of Social blog

(Relatively) Cheat Resistant Rewards and Metrics for Gamification

By MikeW

(Relatively) Cheat Resistant Rewards and Metrics for Gamification

by Lithium Guru ‎10-03-2011 09:50 AM - edited ‎09-07-2012 01:14 AM

Dr Michael WuMichael Wu, Ph.D. is 927iC9C1FD6224627807Lithium's Principal Scientist of Analytics, digging into the complex dynamics of social interaction and group behavior in online communities and social networks.

 

Michael was voted a 2010 Influential Leader by CRM Magazine for his work on predictive social analytics and its application to Social CRM.He's a regular blogger on the Lithosphere's Building Community blog and previously wrote in the Analytic Science blog. You can follow him on Twitter or Google+.

 


 

Scale imbalance_web.gifThis is the sequel to my last post: Beat the Cheat: Stop Gaming the Gamification. In that post, I presented a psychologist/economist’s solution to the problem of cheating in gamification. It turns out that we don’t have to build a bullet proof gamification system. We just have to make it hard enough to game, so the cheaters don’t feel the reward is worth the effort they spent to “game the system.” I talked about two levers that you can pull to play this psychological game:

  1. Decreasing the perceived value of the reward
  2. And/or increasing the effort required to game the system

Today, I’d like to continue this discussion and show you practical ways to affect these two levers.

 

Decreasing the Perceived Value of the Rewards

There are many effective ways to lower the perceived value of the reward, so what I describe here is, by no means, complete. However, they are practical ways that you can decrease the perceived value of the rewards without demotivating the players too much.

  1. First, use purely intrinsic rewards that do not have any transferable values in the real world. This is actually quite challenging, because it requires substantial user profiling to determine the intrinsic motivation of the target audience
  2. Rewards that have greater exchangeable value are more susceptible to gaming. So perks that have a low exchangeable value (e.g. like a cap or a t-shirt) that represents the greater intangible value are more suitable if you want to avoid gaming
  3. Use rewards that have a large perceived-value differential between the target audience and the rest of the population. For example, if LEGO is trying to gamify the word-of-mouth (WOM) sharing of their BrickCon event, they could certainly use cash prizes to do this. However, a much better reward for their gamification effort would be a set of rare LEGO pieces or a set of limited LEGO pieces that few may have. These LEGO pieces will be extremely valuable to LEGO fans, but not valuable to others. That is fine in this case because LEGO fans are probably more effective for WOM than ordinary consumers. So these LEGO pieces have a have a huge perceived-value differential between LEGO fans and ordinary consumers.
    There are many rewards that have perceived-value differentials. Most of them are rare items, or special items that money can’t buy (e.g. special privileges, exclusive access, limited production items, collectables, etc.)

 

Increasing the Effort Required to Game the System

There are many ways to make the gamification scheme harder to cheat. Most rewards in gamification are based on metrics or some cryptic combination of metrics. We can certainly make the combination so complex that the player can’t figure out what they need to do in order to get the reward. This is precisely what Google did to increase the effort required to game their PageRank algorithm. Although this simple “brute force” approach does work, it’s probably not the most economical way. In my experience,there are two key elements which are particularly effective:

 

(A) Metrics That are Less Susceptible to Gaming

The first element is to use metrics that the players do not have direct control over. I’d like to introduce two classes of metrics:

  1. Time-bounded, unique-user-based reciprocity metrics (or TUUR metrics).
  2. Time-bounded, unique-content-based reciprocity metrics (or TUCR metric)

The idea is that we should reward a player based on metrics that are accumulated within a given time frame; and these metrics should measure the number of unique users or the amount of unique reactions to his actions. Let me illustrate what I mean with a couple of examples.

 

The number-of-retweets I receive is a reciprocity metric, because it depends on other people’s reaction to my tweets. But it is NOT unique-user-based, because each tweeter can retweet me multiple times (they can even retweet the same tweet from me multiple times). So I may get 100 retweets, but this wouldn’t be so impressive if they were all from the same user. That is why we need the reciprocity metric to be based on unique-users. So, what if we use the number-of-unique-retweeters? This would be a unique-user-based reciprocity metric. However, it is not time-bounded, because it is ignorant of when a tweeter retweeted me. I may get 100 retweets from 100 unique users, but this metric wouldn’t be so impressive if it is accumulated over a span of a year. That is why we also required the unique-user-based reciprocity to be time-bounded. Therefore, the number-of-unique retweeters-per-week would be an example of a TUUR metric.

 

The second example is from my previous post. Instead of rewarding a player for the number-of-messages he posted in a community (which he can control directly), we should reward him for the number-of-likes (or kudos) he received per month. Every player has direct control over the quantity of messages he posts, but he can’t directly control how many people will “like” his messages since that depends on other players. Because each player can only like a piece of content once, this reciprocity is unique to a piece of content. Therefore the number-of-likes-per-month is a TUCR metric.

 

Although TUUR and TUCR metrics are more resistant to gaming, they are still technically gamable. Players can certainly team up and give each other “likes” or retweet each other. However, to successfully game a gamification scheme that uses these metrics, they would need to coordinate a large number of users over an extended period of time. This is not easy to achieve. Even if some player is able to pull this off, it may not be worth the effort for just getting a few more “likes” or unique-retweeters for the week.

 

(B) Total Transparency and Social Shame

Transparent Footprint Matrix3.jpgThis brings us to the second element, which is to leave a transparent audit trail for everyone to see. Although TUUR and TUCR metrics are technically cheatable, if we make all the reciprocal actions completely transparent down to the atomic event, then people are much less likely to cheat.

 

Total transparency means we need to make visible all the data on who, what, and when (possibly where, if geo-location is available) an event (such as a “like”) took place. For example, if I received 100 likes, then total transparency would allow everyone to see which 100 individuals (who) liked me, which piece of content (what) they like from me, and precisely when they clicked the like button for every like I received. Total transparency is helpful because not only does it make good behavior visible, it also makes any cheating behavior discoverable and sometimes blatantly obvious.

 

Supposed we have two players (say, User123 and User456) on the leader board of some TUUR or TUCR metric. If we make it easy for everyone to see all the events contributing to each player’s TUUR/TURC metric, then it would be pretty easy to discover any coordinated gaming activity. For example, maybe we will discover that 90% of User123’s retweets came from User456 and vice versa. Then it would be pretty obvious that something fishy is going on.

 

When this fraud discovery process is made simple, people’s cheating behaviors can be exposed easily in public (i.e. social shame). Knowing this, a player would probably hesitate before coordinating any dishonest activity. Even if some of the players don’t care about their reputation, other people might not conspire with them knowing the possibility of social shame. This can definitely help limit the amount of collusion.

 

Conclusion

Alright, this concludes our exploration on practical ways to stop cheaters from gaming the gamification system. I talked about several different types of rewards that we can use to reduce its perceived value without attenuating its motivation too much. I also introduced some metrics (i.e. TUUR and TUCR metrics) that we can use to make any gamification system harder to cheat. Finally, to increase the efficacy of these metrics, total transparency and social shame can definitely help to deter cheaters.

 

Next time, let’s talk about something new. As always, discussions are welcome here and always open. See you next time.

 

 

Comments
by Frequent Commentator on ‎10-03-2011 01:27 PM

Hi Mike!

Thx for great post and metrics!

One question. "If we make it easy for everyone to see all the events contributing to each player’s TUUR/TURC metric" - do you have some real examples where this cheaters discovering works (in social networks)? 

by Occasional Commentator matnl on ‎10-05-2011 04:10 AM

Hi Mike!

 

Just a thought: total transparency is fine, but it assumes that people will take the time to delve into the data available. Dissecting 100 likes can be envisioned, but imagine you have 36000 items to go through...I would in this case suggest an 'assisted approach' that mines the patterns and provides the derived insights to other users, who in turn can - based on these insights which they can they verify for themselves - procede to doing 'social shaming'.

 

Give other community members the data and the tools and make it as easy as possible to understand the context, but don't automate the 'social shaming'. Peer pressure is far more effective!

 

Cheers,

Mark

by Lithium Guru on ‎10-06-2011 08:37 AM

Hello Andrei,

 

Nice to see you back and thank you for the question.

 

The only platform that I know makes these metrics transparent is the Lithium platform. For example, we made kudos transparent. You can click on the kudos button to see who has given a post kudos and when. But our platform is pretty well designed in the sense that there is not much cheating, so there is nothing bad to discover.

 

Actually, you can see this comment to my earlier post when I suggest the use of social shame as a deterant for undesirable behavior. And PhilS has discover something quite interesting about kudos I received (see this comment). Not that is is anything bad. But if people doubt, it gives them the ability to look and discover.

 

Anyway, very few platform even track all these metrics. So I really don't know any other examples.

 

Alright, thanks for the question and see you next time.

 

by Lithium Guru on ‎10-06-2011 10:41 AM

Hello Mark,

 

Thank you for stopping by and commenting.

 

Although I totally agree that “assisted approach” would make it easier to discover community misbehavior, but personally, I do not believe in a community that implements or automate social shame. I am advocating having a function that does the social shame. Because, no one will join a community, let alone participate in a community that has a feature that openly shame their members for wrong doings.

 

It’s like the following analogy. Installing publicly visible camera in shopping malls could certainly deter shop lifting. But imagine what would happen if you openly shame shop lifter by showing the video feed to the public in a big screen at the mall entrance? It would certainly stop all future shop lifting, but I think it will also stop people from shopping there completely. The video feed is there for authorized people to discover the misbehaviors, not for automating social shaming. In a community, the total transparency give the opportunity for people to cheating behaviors. We just need to let people know that these informations are tracked and cheating can be discover. That is all.

 

In fact, a community should probably never implement social shame or use social shame at all. It should only provide the perception of a potential social shame. That is often enough to stop the undesirable behaviors. When dealing with human psychology, we must be careful what measure we take, and not over doing it and get counterproductive effects.

 

Anyway, thanks for the comments, and I hope to see you next time.

 

by Tony Ventrice(anon) on ‎10-06-2011 11:35 AM

Mike, this is a common topic we've encountered as well and I think our stance matches yours in a lot of  ways.  I do have a few questions though.

 

It seems that your first point of 'Decreasing the Perceived Value of the Rewards' is a potentially disastrous idea that could cost you users.  Aren't you basically saying:  Make the rewards of exclusive interest and then only your most core fans will be interested in cheating to obtain them?

Many gamification systems aim to engage larger communities by reaching out to more 'peripheral' users.

 

The second point is more valuable but probably doesn't need all the acronyms. I think the message could probably be summarized as:  

'Measure community response, not individual output.'

Regarding the time-limited aspect, it seems like more of a point of situational taste as it could end up feeling 'punishing' in some contexts.  Afterall, whatever happened to 'slow and steady wins the race'?

 

To the third point of exposing the 'who' behind a voting-based metric, this is a fine idea in theory but in practice feels like it undermines one of the primary tenants of gamification, which is the aggregation of behaviors into digestible, meaningful metrics.  Re-expanding those metrics in the interest of full-disclosure would need to be handled very carefully (probably hidden beneath an UI layer) as it could potentially be overwhelming to the majority of your users. 

 

Tony

by Lithium Guru ‎10-11-2011 06:46 AM - edited ‎10-11-2011 07:30 AM

Hello Tony,

 

Thank you for the comment and the question. I apologize for the late response. I’ve been traveling a lot lately.

 

Your points are very valid. Dealing with human psychology is a delicate issue. And success depends on what you are trying to achieve. I’ve described 3 strategies to make the rewards less prone to gaming, only the last one focus on the core fans. And there are situations where that will work better than reaching everyone. But I am not advocating to make rewards of exclusive interest all the time. For people who are not interested in your product or service (then obviously they won’t bother to game it), you will actually need to increase the value proposition, rather than decrease it. It is only for those who already realize the value and want your rewards so badly that they are willing to cheat that you want to reduce the perceived value of the reward. It is a very delicate balancing act.

 

Your second point is very valid too. However, it is important to distinguish community response from reciprocity. The community can respond to a lot of thing that is not an reaction to any individuals output. Communities themselves have dynamics and react to external forcing functions. It is only the part of community response that the users can influence through their action that we need to measure. Community response that users cannot influence is not going to be an effective metric for gamification.

 

The time bounding metrics is mainly to drive competition. And it is important that the oldest users don’t always win just because they are the first to participate and have time to accumulate his reciprocity metrics. So it may seem like punishment in some context, but on the other hand, it also gives other users a chance to win. The time scale at which you time-bound the metric depends on the tradeoff between how much cooperation vs. competition you want to drive. There is no one time scale that works for all situation.

 

Finally, I am not advocating that you impose the raw metrics on users. They are only there for those who like to dig deeper. For example, our kudos system. We made it transparent for users to look at who gave what content kudos, but only if people click into the kudos page. If not, they just see the total number of kudos a piece of content receives. One of our client, giffgaff, mentioned that this transparency let their community members help them discover cheating behaviors. The community manager don’t even have to do too much to stop cheating. The community’s social norm essentially polices itself.

 

There is not hard and fast rules when it comes to preventing cheats. All of these are techniques that must be used carefully, monitor constantly, and handle delicately.

 

Alright, I hope I’ve explain the delicacy of this issue. And I hope I’ve clarify what I meant.

Thanks again for your comment, and hope to see you next time.

 

by Mark(anon) on ‎10-20-2011 10:57 PM

Hi Mike, great article thanks,

 

Accountability, you call it transparency (to a degree), is the key I think. One of our products is a social contest platform. We did a few experiments. Users can only log in with Twitter or Facebook. When the avatar and link to their Twitter/Facebook profile was displayed we found that it was not neccessary to moderate almost at all (for scale some we're talking about upto 800 likes/votes and sometimes 100's comments). When avatars and links (especially) were removed the need for moderation increased, with higher levels of abusive content and attempted "cheating". 

 

Essentially accountability leads to self moderation. Anonymity almost encourages cheating/abuse. Removing it by only allowing direct connection to an existing valued namespace is imperative, in my opinion.

 

Another interesting factor that we noticed is the type of contest being run and the demographic it's aimed at. I've not got time now, but if interested I'll give you some more info.

 

Cheers, Mark.

 

 

by Lithium Guru ‎10-21-2011 09:34 AM - edited ‎10-21-2011 10:02 AM

Hello Mark,

 

Thank you for the nice comment.

 

You are totally right! Accountability is the essence. But what you really want is social accountability, meaning that people or the community can police themselves. Transparency and data presentation that makes cheating behavior easily discoverable is key to making this work.

 

You brought up a good point. Social Network profiles are much harder to fake (see Community vs Social Network), because the connections collectively and uniquely identifies the users. As a result, social networks are much more traceable and much better for social accountability. That is probably the reason for your observation.

 

We would love to see more of your demographic data and how users behaved differently based on the different type of contest being run. Feel free to share the data here. We can discuss this openly and transparently.

 

Cheers and hope to see you again next time.