The precise calculation for computing the interaction score is done in 2 steps:
1. compute the interaction score for each publically accessible thread in the community.
2. average the interaction score over all publically accessible threads in the community.
So you may ask how do we compute the interaction score for each thread then. Well, that is computed by taking the number of unique participants in the thread, subtract 1 from it, then multiply by the base-2 logarithm of the number of messages in that thread.
If we let u=number of unique participants, and m=number of messages in the thread, the interaction score is (u-1)*log2(m). After we computed this for each thread, we just average them across the community, smooth and normalized to get the interaction health factor for the community for that week.
Few things to notices is that this equation is design this way so when there is only 1 users, the interaction should be zero. And when there are 2 users, each posted 1 message. ie, user A ask a question and user B answer him/her. Then the interaction will (2-1)*log2(2)=1, or 1 unit of interaction.
Also, because the number of message is inside a log2, which is a very slow growing function, whether 2 person talk is more important than how much they talk. The point is that if there are 10 more person join in a discussion then the interaction could increase by 10x for that thread, but if 2 person talked 10x more, then interaction would only increase about 3x.
Hope this is not too confusing. But you wanted me to elaborate...
Concerning the fact that there are many possible interaction. That is true. But CHI is not meant to capture all type of interaction. Besides, since it was intended to be a score that any community (including non-Lithium communities) can compute, I was working with a very limited set of metrics that any community platform should have. That is why I was limited to this simpler form.
I hoep this answered your question... Keep the questions coming.
... View more
Thanks for you comment. Glad you like the idea of the new widget.
Let me try to reply to some of your other comments that I think might be useful for other readers here.
"The un-normalized grap only seems to show week 3 and 4 (green and purple), whereas the normalized shows all the weeks/colors. Is something wrong with the graphic, or is that the whole problem with un-normalized data?"
Actually all 4 weeks are on there. They are just so similar that they pretty much lay on top of the earlier weeks. That is why you don't see them. That is another problem with using un-normalized data. The scale for the traffic health factors is so large that the weekly variation in other factors are virtually unnoticeable.
"I totally get the w/o typology form. I didn't get the typology... like most people, it sounds"
Yeah, in retrospect. I think that could be confusion for people who are not statisticians. My apology to you all. But here is how I compute the percentages for week 1. We take the un-normalized predictive factors and divied each by the best of the group in the past 6 month. Since we are only at week 1, the best among the predictive factors is interaction, which has a score of 5. So we divided 1, 5 and 2 by 5 to get that 20%, 100% and 40% for liveliness, interaction, and responsiveness respectively.
Now for week 2. The value drop, the predictive factors are 2, 3 and 3. But the best score among the predictive group is still interaction, which has a score of 5. So we still normalizes 2, 3 and 3 by 5, which gave us the 40%, 60% and 60% for liveliness, interaction, and responsiveness respectively.
Moving on for week 3. Now the best score in the past 6 month among the predictive factors is 8 because interaction jumped to 8. So we divided the week 3 scores: 3, 8 and 2 by 8, which gave 38%, 100% adn 25% for liveliness, interaction, and responsiveness respectively. I think you get the idea.
By normalzing the predictive factors as a group, the relative difference between the 3 predictive factors are preserved. And this difference is precisely what gave us the typology information, which used to be encode in the shape of the CHI compass.
For example, enthusiast communities tend to have have a higher level of interaction relative to their liveliness and responsiveness because the dominant mode of interaction is discussion, where as in a support community the dominant mode of interaction is Q&A. By normalizing the predictive factors as a group we can preserve the fact that interaction is always higher because all three predictive factors are always divided by the same number each week. If they were normalize individually, they would have been divided by different numbers depending on each factor's past performance. In this case, interaction may no longer be the highest score week-to-week. You can see this in the "normalized w/o typology" figure above. So the typology information, that characteristic bulge along the interaction axis is lost in this figure.
I hope this made some sense, rather than made it more confusing.
... View more
Dr Michael Wu Ph.D. is Lithium's Principal Scientist of Analytics, digging into the complex dynamics of social interaction and online communities.
He's a regular blogger on the Lithosphere and previously wrote in the Analytic Science blog.
You can follow him on Twitter at mich8elwu.
Hello everyone, and welcome to Building Community, Lithium's blog about the platform. Just as in the past, I will continue to cover analytic science and research here at Lithium. Today, I will talk about a refinement in the community health report that slightly altered the information displayed on the CHI compass that many community managers are receiving today.
The CHI compass is a radar chart that is a great for giving a compact presentation of multi-dimension data, namely the six health factors (members, content, traffic, liveliness, interaction and responsiveness) that goes into the computation of CHI.
However, some health factors have very different ranges of values, for example, weekly values for members may range in the hundreds, where as content (posts weighted by views) and traffic (views) may have values over 10K or 100K. On the other hand, liveliness, interaction and responsiveness, have much smaller values in the tens or, in some cases, less than one. With such large variation in the scale of the data, it means we have to normalize these values before we display the CHI compass. Otherwise the small weekly variations in the smaller values would not be visible against the large variations in the diagnostic health factors.
Here is an example that I've generated with Excel. In this example, liveliness had a very significant 100% weekly increase (from 1 to 2, 2 to 3, then 3 to 4) during the first 4 weeks. But it is relatively unnoticeable when plotted in the radar chart against the other diagnostic health factors, which have much higher un-normalized values.
One of the insights that came out of Lithium's research was that even though you may launch a community with a specific purpose (such as key benefits like Innovate, Promote, or Support,) depending on the interaction at any particular period in time, you will find that a community can adopt the multiple traits, and these traits can change over time. For example, support communities often behave like enthusiast communities after product launches. Likewise marketing and sales oriented community can also behave like a support community and answer technical questions from the enthusiasts.
The shape of the CHI compass was designed to actually show what type community behavior is taking place. Thus, the shape can tell you whether your community is trending towards an innovation, promotion, or support community. However, this design requires a particular normalization scheme that normalizes the predictive health factors as a group. Because the liveliness factor generally has a smaller range of values than interaction, it appears much smaller when normalized and put on the same scale as interaction. People often interpret this low value as the community not being healthy whereas the community might be perfectly fine. The relative position on the compass is purely an artifact of trying to show the typology information on the CHI compass.
Under this normalization, the same community would look like the following. The outward bulge in interaction is the characteristic signature of an promotion community.
Rethinking the Visualization
So, on to the changes. After hearing many inquiries about how to interpret the CHI compass, we've decided to go with a simpler and more intuitive normalization scheme and remove the community typology information from the CHI compass.
We simply normalize each health factor to its best previous performance (over a 6 month window). If you outperformed the 6 month best score, your health factor will be 100% that week. This will redefine the new standard for which to normalize your future scores. So a growing community should hit 100% periodically, which indicates improvements over the last 6 months.
With the new normalization, the very same community would look as follow.
Some of you may ask..., if the shape of the compass doesn't indicate the community type anymore, where do we get that information? Don't worry, the community typology information is not lost, we will design another widget specifically for displaying that data!
I hope that gives you an insight into the changes in the CHI compass, and why we thought it is important. As always please let me know if you have any questions or would like me to cover particular topics.
... View more
Hello FM Addicted,
Being a scientist, I am some what like that too. Well, I hope you won't lose too much sleep though. Even though Flow is an extremely desirable state, eating and sleeping is important in the long run so we can enjoy more future flow states. Take care and thanks for the comments.
... View more
Hello, and welcome back from the Thanksgiving holiday. As the year comes to an end, we are also planning a restructuring of our blog program for next year. So today, there are two items on my agenda.
First, I like to thank all of you readers out there for following my blog on Analytic Science. I know it is not a subject that appeals to everyone, and numbers and symbols can be, at times, technical and hard to digest. So thank you again for your interest.
Second, I also like to inform you of the upcoming changes to our blog program for the new year.
Announcement: Blog Program Changes
Starting in the New Year we'll be running a new, restructuring blog program on Lithosphere. Full details can be found here where we outline all the changes.
Thanks for following up till now, and we're looking forward to talking with you in the new blogs.
... View more
The SF Bay Area is turning really cold now, and it's been three months, since I was invited to the Lawrence Livermore National Lab (LLNL) as a DOE CSGF alumnus. Today, I'd like to finish sharing the rest of that day with you. If you missed the first part, look here.
After the NIF tour, we took a short bus ride to the Terascale Simulation Facility (TSF). This facility is so highly-classified that one of our CSGF fellows was not allowed on this tour because there wasn't enough time to obtain clearance due to his foreign national status. The TSF houses the world's most powerful supercomputers. And they are constantly building and acquiring the newest, fastest supercomputer in the world. For example, the TSF still houses IBM's Blue Gene/L, which was the fastest supercomputer in the world back in Nov 2007. This supercomputer has a peak performance of 478.2 TeraFLOPS (478.2 trillion Floating-point Operations per Second). Today, Blue Gene/L ranks #5 on the top500 Supercomputer list. The Roadrunner at Los Alamos National Lab takes the lead with a peak performance of 1.1 PetaFLOPS (1.1 quadrillion FLOPS). However, earlier this year, TSF has already installed Dawn, a 500 TeraFLOPS initial delivery system of the Sequoia. Sequoia is the next generation Blue Gene supercomputer capable of computing at 20 PetaFLOPS, and it will be delivered early 2011. Scientists believe that at this rate, supercomputers will be powerful enough to simulate human brain function by 2013 and full-brain neuronal simulation would become possible by 2025.
Figure from LLNL.
The TSF houses its supercomputers in a 48,000 square-foot clear-span vault. The vault entrance has a large panel of electrochromic smart glass that turns transparent with a flip of a switch, but is otherwise completely opaque. This panel is fail-safe, and power must be applied to keep it transparent. So in the event of a power failure, the vault would remain secured. The vault's interior is filled with motion sensors, and we were told that the alarm had been tripped a few times because the computer's cooling fans had blown off small pieces of stickers and tapes from the new computers.
With so many supercomputers, heat becomes a serious problem. The supercomputer vault basically sits above a giant air conditioner. The vault floor is tiled with both perforated and solid aluminum tiles. The precise placement of each type of tile and the arrangement of the computers is optimized so that the airflow from beneath the floor provides maximum cooling efficiency. All computers are also mounted on seismic suspension racks, and they will simply wobble around during an earthquake.
The TSF is critical to the stockpile stewardship program, and it is used to simulate the physics of nuclear weaponries to an extremely high degree of accuracy that could supersede the need for actual underground testing. Basically the simulation must be as good as real. TSF also enables scientists to study many physical phenomena that occur at multiple scales through molecular dynamics. At the conclusion of the tour, Frederick Streitz (director of Institute for Scientific Computing Research at LLNL) showed us 3 amazing large-scale (up to billions of atoms) simulations and visualizations that lead to a better understanding of our physical world. The first was the Kelvin-Helmholtz Instability, for which Dr. Streitz and his team receive the Gordon Bell Prize. And the other two were on the fluctuation of electric fields in plasmas, and the domain formation during solidification.
After the tours, we were treated to wine tasting, dinner, and networking with DOE scientist and officials at the Wente Vineyards. I must say that every year when I gather with the CSGF fellows and alumni, I always feel deeply honored to be a part of this very distinguished group. Although many CSGF fellows do end up working at a national lab, such as LLNL, LANL, NASA, etc, a significant number of them go into academia and joint top rated universities, such as UC Berkeley, and Stanford. The rest work at some of the best companies across all major industries: Google, Microsoft, Intel, Boeing, IBM, HP, Fair Isaac, GE, Procter & Gamble, Goldman Sachs, Boston Consulting Group, AREVA, Exxon Mobil, Merck & Co, and Lockheed Martin, just to name a few. And I am very proud to bring Lithium Technologies to this list among the CSGF employers.
OK, that was the rest of my day at the CSGF Bay Area chapter gathering. After the Thanksgiving holiday, we will go back to community and social analytics. Follow my updates at mich8elwu.
... View more
Thank you for the comment. These are great tips for community managers. I couldn't agree more that you should never give praise that you don't really meant. Once that gets into a habbit, the community will lose credibility at the beginning, and eventually lose trust from their users. When that happens, that is one way that I've seen communities failed. So be honest and sincere!
... View more
Hello Neil, Yeah, we found that communities have lots of applications in a marketing organization as well as a support organization. However, the ROI is not as well quantified. So we move forth with this study. To address your questions: 1. We can definitely identify influencers in the community by user name and enable community managers, moderators, admins, or whoever has the right permission, to contact them. Moreover, we can identify different types of "influence," by different centrality measures. For example, users who are most well connected are not necessarily the same as those who are most well respected. Depending on your marketing needs, we can identify users who are most relevant for propagating WOM for your community. 2. In general, there isn't really a time range beyond which defensive WOM becomes negligible. The worse is probably not trying at all. 3. Personally, I would not wait until the community has reached a healthy level of responsiveness and interaction before embarking on WOM. Because in this study we found that time is really an inelastic resource that has dramatic effects on people's enthusiasm, and therefore their ability to propagate WOM. I would start immediately to keep the enthusiasm going, and then improve the responsiveness and interaction health factor to boost the results. 4. Yes, but keep in mind that the gain from customer acquisition is zero only under the assumption that the 2 brands have equal strength. This is the assumption of our study. Because for everyone customer that brand1 acquired from brand2, brand2 can do the same to brand1 for a different customer. If one brand is stronger than another, then the gain from customer acquisition would be non-zero. I hope this address your questions. Thanks for asking questions and commenting on my blog.
... View more
The Social CRM Virtual Summit is almost upon us - and I am getting ready to take part in two expert chat sessions tomorrow on the Science of Social Analytics, and How you can build Brand Equity through community. This last topic is particularly relevant as last week we published a whitepaper on the value of using your customer network for word-of-mouth (WOM) marketing. This is joint research I conducted with renowned professors in the field of marketing science, including:
Barak Libai, MIT Sloan School of Management and the Recanati School of Business in Tel Aviv
Eitan Muller, NYU Stern School of Business and the Recanati School
Renana Peres, The Wharton School, UPenn and Hebrew University of Jerusalem
This whitepaper is a particularly hot topic, so I will be joining our VP of Product Marketing, Phil Soffer, and chatting on the topic of WOM at 6:15am and 11:15am PST - I look forward to talking with you.
That leads me into my topic for today's post. As I've alluded in previous blog posts, the value of WOM is not particularly tangible. Estimating the ROI on WOM is nontrivial and it is still a research topic for academics.
In a network of hundreds of thousands of customers, the value of WOM really comes down to how we estimate a customer's lifetime value with the effect of WOM and without it. Let's consider the Lithosphere community as an example.
If I told PaulGi about Lithium's mobile community product and he subsequently buys it, then a small fraction of the value that Lithium gains from PaulGi's purchase should be attributed to my WOM interaction. However, if I didn't tell PaulGi about the product, maybe ScottD could have told him about it. Then that small fraction of WOM value should now be attributed to ScottD, instead of me. To complicate things, maybe we both told him about it, and who's to say that people I've spoken with listen to me instead of Scott. So the conundrum is, how should we estimate my WOM value to Lithium?
Working with academics, we use a simulation methodology, which I will refer to as "impact upon removal." In essence, my WOM value to Lithium is the value that Lithium would have lost if I did not tell anyone about their product. So a user's WOM value is the value lost if we blocked all of his interaction with other community members, essentially remove him from the social network. It's like the saying that "You don't really know the value of someone until you lose him/her." In a nutshell, this technique is what allowed us to study the effect of WOM in a customer network.
Now that you know some of insights to our research result, I hope it will encourage you to find out more. If you are intellectually inclined, you can get the details from our whitepaper. Better yet, come by and ask me question during the Virtual Summit tomorrow. You can still register at http://www.bit.ly/vscrmreg. I hope to see you 'virtually' and talk to you tomorrow!
... View more
Last time I talked about what got me interested in social analytics and what is the big community topic that is currently taking up most of my brain cycles. This time, let me give you a bit more detail about my current projects at Lithium Lab.
My research at Lithium focuses on a couple of key areas. First, since a community is all about the people, the first area of research focuses on understanding user behaviors. The goal of this research is to understand the complex interplay between different groups of users through social network analysis (see figure below) and discover the dynamics that drives a healthy and successful community.
Currently I am particularly interested in two groups of users
and the lurkers.
Superusers are obviously interesting because they contribute so much and bring so much value to the community. But why do they contribute? What is their incentive? No one comes to the community as a superuser. Yet, in every community, we observe the rapid emergence of influential superusers. Can we accurately predict who will become a superuser soon after they join the community?
Lurkers are interesting in their ownright because there are so many of them. The majority of the audience - up to 90% of the users - could be lurking. What keeps them engaged even though they don't participate? Can we incent lurkers to change their behavior and start to participate and move up the rank ladder, maybe ultimately becoming a superuser? That is surely a holy grail for community managers.
Another area of focus is research which aims to derive predictive models for business value. The goal of this research is to discover all the mechanisms where the Lithium platform can bring value and then quantify the actual value they bring to the business. There are many mechanisms that our community platform and services can bring value to our client. Just to name a few, for example: call deflections, word of mouth (WOM), collaborative innovation, crowd sourcing, even lurking can bring certain values to our client. Some of these mechanisms, such as call deflection, are well understood and their ROI are readily quantifiable. But the value of WOM, and lurking are less tangible.
Currently I am working a model that quantifies the value of WOM in a community. This is along the road to quantifying the value of a superuser. Superusers actually come in many flavors (product experts, advocates, brand evangelists, opinion leaders, etc) and each type of superusers brings value through different mechanisms. More importantly, different community needs a different mix of superusers. For example, a support community probably needs a lot of product experts and some opinion leaders; where as a marketing community would need more advocates and brand evangelists. What is the optimal mixture of superusers for any given community?
With all that said, I hope you are excited? I certainly am. I am hoping this will give you a little more context for the live-chat at the Social CRM Virtual Summit. I look forward to seeing you there and chatting with you on November 11th. Remember if you haven't registered for the Virtual Summit, I highly recommend it - and you can sign up here.
... View more
Lithium is hosting the Social CRM Virtual Summit on Nov 11, 2009 (you can sign up here), and I was asked to hold a live-chat with the audience at the summit. This will be a great opportunity for me to talk to practitioners and get a sense of what kinds of analytics people want from their communities. To get the conversation started, let me tell you a little bit about what got me into social analytics and what I am working on now.
As some of you know, I was a computational neuroscientist (my bio is here). So what got me interested in social analytics? Honestly, it's all about the data! As a SaaS company, Lithium has recorded a huge data set over the 10 years of its business operation. The data at Lithium is very rich and diverse. Besides the 200+ metrics that Lithium records, there are also loads of conversation data between real people. This is what got me excited about social analytics.
You may ask why I didn't go to some place like Google or Facebook then? Certainly they have also collected a lot of social network data, probably a lot more than Lithium if we are talking about sheer storage volume. But as a statistician, we care about sample size. Facebook may have the biggest social network of 300 millions users, but it is only one network. Lithium has hundreds and the number is growing! This enables benchmarking and cross sectional studies that are not possible anywhere else. It is almost as if you can play god and start the network over and over again hundreds of times with different initial conditions. In statistics terms, this is what gives statistical powers to any inferences we make about the community.
Because Lithium has such a rich set of conversation data, we can also glean much insight from understanding these conversations using advance text analysis tools from machine learning. Because the conversations in a community are highly relevant to the sponsoring company, we do not need to worry about information retrieval and deal with the tradeoff between precision and recall. So we can focus our computing power on understanding the content of the conversation. Personally, I believe this will revolutionize the CRM industry, and this is the topic that I am most excited about.
By listening and comprehending the conversation of their customers, companies can understand customer needs and serve them better. On the flip side, customers can truly make their voices heard! CRM would be much more than an automation system of business processes on top of a database of customers' name, contact, when, what, and where they bought in the past. CRM system would know, for example, is a customer satisfied about the product? Do they like all the features? Which feature didn't they like? What problem did they have when using the product? Are they considering switching to a different brand? Are they considering your brand because of the bad experience with another brand? These are the kinds of insight we can reveal by understanding the conversation within the community.
So now that you know what got me into social analytics and what's in my mind, next week let's can get a little more detail about my research at Lithium Lab and what I am currently working on. Stay tuned at mich8elwu.
... View more
After more than two months, we have finally reached the 6th and final blog on the current and future development of the community health index (CHI) and its health factors. I want to thank all of you readers out there for all the feedback; the analytics team at Lithium takes them seriously in prioritizing what we deliver. I guess this mini-series is getting too long and you are probably a little tired of it by now. So, let's nail it this time! Alright! On with the final health factor: Responsiveness. If you missed any of the previous blog in this series, they are: Traffic, Content, Members, Liveliness, and Interaction.
The Responsiveness Health Factor
In this era of information, there is no shortage of quality information, and because this information lives on the internet, it is nearly free to the end users. However, if this information is not delivered in a timely manner, it will lose its value. Therefore, time is a critical resource that is going to influence a user's experience when he/she is engaging with your community. A community that responds promptly will give the visitors a much better user experience than one that takes a long time to respond.
The responsiveness health factor is a measure of the average elapsed time between responses. It is very similar to the traditional time-to-response metric, which is generally defined as the elapsed time between the first message in a thread and the first response to that message. However, this traditional metric does not take into account the elapsed time between subsequent responses. So, you can think of this health factor as a more accurate version of the time-to-response metric.
Responsiveness also has a significant impact on user retention because visitors will abandon an irresponsive community altogether and spend their valuable time elsewhere. Therefore, this health factor can be interpreted as the perceived quality of engagement for potential visitors.
The Current and Future of Responsiveness
The current formula for computing responsiveness ignores threads that have no response, and the fraction of responded threads (or the response ratio) has no effect on this health factor. The response ratio was originally part of the responsiveness calculation, but it was removed when I found that it is highly correlated to the interaction health factor for the community. Since threads that have no responses will have zero interactions, these unresponsive threads are already penalized by their interaction health factor. However, people seem to inherently expect the response ratio to affect the community's responsiveness. To make CHI and the health factors accessible to a wider audience, I may need to formulate the responsiveness calculation in the future. Besides this minor point, we have not received much troubling feedback concerning this health factor.
Finally, I like to draw your attention to an important point about the three predictive health factors: Liveliness, Interaction, and Responsiveness. Recall that these health factors are predictive because they describe the intrinsic social dynamics within the community, and they tend to be leading indicators of community health. In practice, this means the predictive factors can often serve as an early warning sign to imminent problems. By analyzing these predictive health factors, you can take corrective actions against potential problems before they occur. So the predictive factors are also called actionable analytics.
Well, this concludes our coverage on all the health factors of CHI. Next time we will talk a bit more about how these health factors are combined. Stay tuned at mich8elwu.
... View more
This is the fifth blog in the miniseries on the Health Factors of CHI. Other blogs already in this miniseries are:
Last time we talked about the most troublesome predictive health factor: Liveliness. This time we will discuss Interaction.
The Interaction Health Factor
When you are able to create a lively community, the hard work is half done because by definition a lively community has solved a difficult conundrum of participatory media: How do you get people to participate? However, in a healthy community, it is not enough just to participate. There must be interaction with other users. Otherwise, where is the social of social media?
There are two important dimensions to interaction:
The amount of conversation you have with a particular user.
The number of different users you've communicated with.
Clearly, the more you talk to a user, the more interaction you have with that user, but talking to different users also increases the level of interactions within your community. In fact, the latter has a greater effect on the overall health of the community. The interaction health factor measures the average number of unique participants per topic weighted by the amount of conversation between them. Thus, this health factor provides an estimate of the average number of members a potential user is likely to interact with and the estimated amount of messages that will be exchanged between them. Accordingly, this health factor may be interpreted as the expected scope of the engagement for potential visitors.
Different Modes of Interactions on the Lithium Platform
Because different communities have vastly different kinds of interactions, the interaction health factors also vary greatly across communities. For example, the dominant mode of interaction of a support community consists of a troubled user asking a question and a knowledgeable user answering the question. This kind of interaction usually involves few users with relatively short diagnostic dialogs before arriving at the solution, so support communities tend to have a steady and modest value for the interaction health factor. In contrast, the dominant mode of interaction for an enthusiast community often involves extended discussions among many users. Also, because the topic and amount of discussion is usually event-driven, enthusiast communities tend to have a higher, but more volatile value for their interaction health factor. B2B and internal communities also tend to have lower level of interactions than that of enthusiast communities.
The Current and Future of Interaction
Besides the difference in community purpose, the different web applications, such as forums, blogs, ideas, and tribal knowledge base (TKB), also have drastically different modalities of interaction. However, the current health factors and CHI were developed based on analyses of primarily forum interactions. Blogs, Ideas, and TKB are relatively new products in our platform that do not have 10 years of historical data. Consequently, the expected level of healthy interaction in the current algorithm may be too stringent for these new interaction modalities. As a result, the interaction health factor for communities that use these new applications may appear slightly lower than expected. We have made note of this issue and we will address it in the next revision of our formulae.
I hope this blog gave you a better understanding of interaction and addressed the concerns you may have concerning this health factor. Next time, we will move on to the last health factor, responsiveness. Watch for updates at mich8elwu.
... View more
Thanks for the positive affirmation.
I agree that planning your board structure is a delicate art. Our best practice is to start small, than expand if necessary. See post by one of our CSM: Organic Growth in Your Community. It is certainly easier to just have product specific boards, and they are definitely easier to manage and helps keep conversations organized. However, we need to understand at what point users stop browsing and start to use the search function or simple google for the solution. These user behavioral studies are part of the research that is being carried out at Lithium. If you need further assistance, our CSM teams will be able to help you plan the most appropriate board structure for growing a successful community.
... View more
After a month of digression, let's come back to the topic of health factors. Previously, we have covered the three diagnostic health factors:
In the next three post, we will explore the predictive health factors. Base on all of your valuable feedbacks, I got the impression that Liveliness is the factor that has raised the most questions. So I will begin with this health factor.
The Liveliness Health Factor
A number of people have asked: what is liveliness? The liveliness health factor is a perception of the amount of activity within a given space, so it is a measure for the concentration of activities. This health factor is extremely important, because it strongly influences a user's propensity to participate. In fact, liveliness gives an estimate of the likelihood of active participation for community users. The more lively the community appears to be, the more likely an individual is to participate. It's logical to think that a lively community is more likely to respond to the user's question or comment.
Concentration of Activity = Perception of Liveliness
Although modern communities furnish users with many activities, posting messages is still the most visible action that is going to influence a user's perception of the community's liveliness. Since other activities (such as rating contents, tagging messages, chatting with other members, etc.) are less noticeable, they have much less effect on the perceived liveliness of the community.
Look at the above picture and ask yourself: what make it lively? This picture looks lively to a spectator because the children are concentrated, and there are a lot of visible actions. This playground would be much less lively if the same group of children were dispersed throughout the park.
Since liveliness is a concentration measure, the absolute amount of post is not very important. For example, if your community has 200 posts per week and there are 4 boards, then each board gets 50 posts per week on average. That is more than 7 posts a day! Users would perceive this as a pretty lively community. In contrary, if there are 50 boards (same 200 posts per week), then each board will only get 4 posts per week. That is less than 1 post a day! Even though the absolute amount of posting activity is the same (200 per week), the former example has a much higher concentration of post, and therefore gives the perception of greater liveliness than the latter example.
The Current and Future of Liveliness
The question that is often raised is "Do you excluded private boards, archived boards, read-only boards, etc, when computing the liveliness health factor?" In the current version of CHI, the liveliness calculation only excludes boards that are hidden or under a hidden category. These hidden boards may contain some private or archived boards, but they probably do not encompass all of them, and it certainly does not include read-only boards or announcements boards. Consequently, communities with many of these "non-public" boards may appear to have a lower liveliness, because our current calculation may not excluded all boards that are not intended for public participation.
Like the content health factor, liveliness is currently computed at the community level, which prohibits drilling down to identify of categories or boards that are not lively. These two items are already on our roadmaps of to-do items. If you can think of others, please let me know. Also, if there are any questions about liveliness that I did not cover, please bring it up . Let's try to make this a lively discussion.
... View more
Earlier this month, I was invited to give a workshop at Stanford University. This workshop, titled New Metrics for New Media: Analytics for Social Media and Virtual Worlds, is one of many that are sponsored by Media X. It was held at Wallenberg Hall on Aug 5 and 6.
The workshop was organized by Martha Russell (Associate Director of Media X) and Marc Smith (Chief Social Scientist of Telligent), and consists of panel discussions from industry leaders, various talks, and hands-on tutorial sessions. I was very privileged to be invited by Martha Russell to give a talk on Measuring Social Media & Digital WOM. Besides giving my talk on CHI, and the mechanism and benefit of community word-of-mouth, I also learned a lot from the workshop. Since my days are still overwhelmingly busy, I've decided to write a short blog this week to share with you one of the most interesting metrics that I've learned.
Inferring User Intentions:
One of the most intriguing metric I learned from this workshop is intention: what does a user want to accomplish. Intention is not really a metric in the traditional sense, but it is something that we can potentially infer based on other metrics. Although this might sound a little scary, I believe it is beneficial. Information retrieval system could benefit greatly from understanding exactly what the user is looking for.
Although some business may leverage this information and spam their audience, but a spam to one may turn out to be useful information to another. A key for Google's success in ad targeting is that the ads are sufficiently relevant to the user that they are no longer treated as spam. The reason businesses spam today is precisely because they are unable to accurately infer user intent. If we can truly infer intention, there is no need to spam at all. In fact, spamming would become impractical and no one would do it because its efficacy is so low.
However, intention inference is not easy. The only algorithm I know that is capable of performing such inferential task is belief propagation. However, this algorithm is computationally very expensive. Even with modern cloud style supercomputing, running a large-scale belief propagation may still be infeasible. Yet, I am fascinated by this inference problem from a theoretical point of view. Much can be analyzed before one finds a practical implementation. That is why we need researchers!
OK, that is it! Told you this will be a short one. Let me know what you think about this metric. If someday we discovered an efficient algorithm for intention inference, would you be excited or afraid? How would you use this information?
Photo by Marc Smith
... View more
I asked this exact quetion when I was there. The answer I got was that they are looking at 2-3 year from now to prove this technology (Inertial confinement fusion). As for commercial power, it may take a decades. This is because there are lot of infrastructure that need to be in place before you can build a fusion power plant: policies, safety, economical viability, efficiency etc.
... View more
You might know from my short bio that I am an alumnus of the Computational Science Graduate Fellowship (CSGF) program sponsored by the US Department of Energy (DOE). Last week, the CSGF fellows and alumni in the Bay Area were invited for a tour of the Lawrence Livermore National Lab (LLNL). We visited the National Ignition Facility (NIF) and the Terascale Simulation Facility (TSF), and it was eye opening to learn about the big science and the challenging research that were conducted at LLNL. After the day tours, we were invited to a wine tasting and dinner at the Wente Vineyards. Although this has nothing to do with community or analytics, it was a full day of inspiration for me. So let me share some of my excitement with you.
LLNL is a classified national lab, so I had to send in my personal information for a background check weeks ahead to obtain the proper security clearance. The lab is so secured that no cell phones, cameras, video cameras, or any electronics were allowed beyond the guarded fenced surrounding the facility. Basically, after I picked up my visitor badge outside the fence, I become completely isolated from the outside world. I'd like to draw your attention to the blue casing hanging above the badge. Any guesses on what that might be? It is a radiation sensor. Although I had one of these badges when I was at the Los Alamos National Lab in 2003, I never had one with a radiation sensor, because I was working in the Machine Learning Group under CCS-3 Division, which is safely shielded far away from all radioactive materials.
Alright, let's get on with the tour. Stop #1: NIF. As we arrived at the NIF, we saw a big sign saying "Bringing star power to earth." Let me explain what that means. The NIF is a nuclear fusion research facility that houses 192 lasers, including four of the most powerful lasers in the world. Using the Inertial Confinement Fusion technology developed at LLNL, these lasers are used to ignite a fusion reaction (at a temperature of 100,000,000 degrees) that creates a miniature sun (a medium sized star) inside a 10 meter spherical target chamber (see photo from NIF). The fusion chain reaction in this miniature sun will then generate clean energy much like the sun has power life on earth for millions of years. This animation by NIF explains how these lasers work.
Current nuclear power plants use nuclear fission technology, which produces a radioactive byproduct that remains hazardous for millennia and can be used in nuclear weapons. However, the byproduct of fusion is just Helium-4, a perfectly clean, safe, non-reactive noble gas. Moreover, fusion technology could eliminate our dependence on fossil fuels, because fusion power is much more efficient. The fusion energy released from 1 gram of hydrogen fuel is equivalent to roughly 2 tons of coal!
The NIF is a colossal project! Commissioned in 1993, the construction of NIF began in 1997, took 12 years and $3.5 billion. The last of the 192 laser beams were completed on March 31, 2009. I feel so privileged to be able to see the NIF in its entirety so soon after its completion. Currently, hundreds of engineers and scientists at NIF are using simulation shots to test all the lasers, the diagnostics, and the control system. There is absolutely zero tolerance for errors. Future experiments are aimed at replicating the nuclear ignition process reliably and rapidly enough so that it can be a viable source of commercial energy.
Are you excited? As a scientist, I'm totally thrilled! You may admire the futuristic look of this magnificent structure, but what I see is the culmination of a Herculean scientific endeavor. To me, the NIF is a product of thousands of passionate scientists and engineers from all sorts of backgrounds who have dedicated their lives to working together for the well being of humanity. I just couldn't help but be in awe when I think about the difficulties they must have overcome to reach this stage.
Let's save some of my enthusiasm for later. This is the first time I have posted something unrelated to social analytics, but I thought it might be interesting for you to get a peek at the life of a scientist. Let me know how you like my little adventure. Stay tuned at mich8elwu.
... View more
Thank you for your reply. Concerning attrition: we have to make an assumption that members who have not login for some period of time are considered lost. We can pick a reasonable time frame, say 4 weeks for example (the exact number of days for the window, will have to be set after we examine more data). Then member attrition is just the number of user who has not login for at 4 weeks or more. Since out app tracks the time of every login, this attrition rate can be computed.
Of course, you would want to compare the weekly differential of the above measure of member attrition to the number of registration (member accumulation) in the past to come up with a percentage of loss over the 1 week period. For that you will need the number of registration between 4 weeks ago and 5 weeks ago.
... View more
Thanks for the reply. Your affirmation will ensure that this feature gets the attention it deserves. This is very good and I sincerely appreciate you taking the time to confirm that this is a good feature to have and that it is useful for you.
... View more
Hello, TGIF. I think this is the first time I've try posting a blog on a Friday. I had a hectic week!
So, this is the fifth and the last article in the miniseries Ranks Designed to "Flow". Previous blog articles from this miniseries can be access through these links:
1. Spacing the rungs of your ranking ladder
2. Know your superusers!
3. Flow with your most prolific superusers
4. Give your superusers a little surprise
We've covered a lot of ground, and I've introduced many design principles for building an optimal ranking ladder for engaging your superusers. I must emphasize that it is very important to implement these rank design principle in the order that they are presented. It is meaningless to flow with your superusers, if you don't know your superusers' capability. And it is useless trying to surprise your superusers with special privileges if the gaps between your ranks are so large that it takes them years to get a promotion. They will never get there and never be surprised! However, the first two steps do involve some analytical work, and they are the most difficult and most important step (especially step 2). Once you know your superusers, everything that follows is easy.
Building a complete ranking structure
Until now, I have been talking about the principles for designing a single ranking ladder that rewards the posting behavior of community members. Although message posting is a common participation within online communities, modern community platforms now furnish their members with a host of activities. Consequently, superusers may come in many flavors depending on the kinds of activity they participate in. The superusers that we've considered so far are content creators who excel in posting messages. But superusers may be critics who rate contents by giving kudos and report inappropriate contents, and others may be organizers who label and tag contents. Therefore, an ideal ranking structure should have multiple branches for rewarding different kinds of participation.
To create a multi-branch ranking structure, you simply juxtapose everal ranking ladders together. Each ladder has a set of ranking criteria that is based on different participation metrics. For example, rather than post requirement, some ladders may use kudos requirements, and others may have a tag count requirement. Based on the superusers' participation, they will climb different ladders. Some well-rounded superusers may even excel on several ladders. Moreover, ranking ladders can be merged by creating participation criteria using the logical AND.
So how many branches should you have? According to Forrester Research, online participation pattern can be segmented into 6 categories via the social technographic profile: inactives, spectators, joiners, collectors, critics, and creators. Since inactives do not participate, and spectators only consume contents passively, there are at least 4 categories of active participation that you can reward. But in theory, there is no limit to how many branches you could have in your ranking structure. The more ladders you have, the more unique your superusers will feel about their contribution (and reward). But more ladders required more management. A multi-branch ranking structure should be the last step in the design of your ranking structure. Having many poorly designed ranking ladders is much worse than having one that is well designed. My advice is to start with one ladder for your creators. When you are able to manage steps 1 to 4 with all the yearly adjustment, add one for your critics, then collectors, and finally joiners.
Congratulation! This concludes my miniseries on the optimal design of your ranking structures. Next time we'll explore something different. Have a great weekend.
Photo by Jon Wiley
... View more
Thank you for the comment. I am truly honored to be invited to give a talk at the workshop on New Metrics for New Media: Analytics for Social Media and Virtual Worlds. I will try to cover some of my recent research along with the community health index during the workshop. And if time permits, I'd be happy to go over the art and science of designing an engaging ranking structure. I look forward to participating at the workshop.
... View more
Hello Roxy Thank you for the reply. I totally agree that we should make as much of our best practices the default setting. The difficulty with ranking is that the optimal setting depends on the performance of your community's superusers, which is surely going to be different from one community to another. (See my earlier blog on Know Your Superusers.) Although not impossible, this certainly makes it challenging to have one out-of-the-box default that will work for all communities. Even though we have been launching and managing communities for 10 years, we are also analyzing the data and constantly learning, and we will discover new best practices as the product evolves. It is definitely one of our goals to built-in as many of our best practices in our app’s default as possible, since we are committed to the success of our communities.
... View more
After 3 weeks of Health Factors, I guess (from the fact that my last post received no kudos) you are probably a little fed up with it. So let's take a break!
It's been a while since we talk about rank design, so this is a good time to revisit the topic. This is the 4th article in the miniseries on designing the optimal ranking structure for your community. Previous blog articles from this miniseries can be found here:
1. Spacing the rungs of your ranking ladder
2. Know your superusers!
3. Flow with your most prolific superusers
Give your superusers a little surprise
Part of the fun and the challenge in gaming is the unpredictable elements in games. The player can never truly know the outcome of his play. When the gamer has just figured out the game, he moves into the control state. When he is able to predict the outcome, the game is probably too easy for him. And soon he will find the game boring and move onto something more challenging. Likewise, a fun and challenging ranking ladder should be cryptic, and it is best when there are some elements of surprise built in. In a previous post, I talked about switching after year 2 to a very regular and predictable arithmetic progression (a.k.a. linear progression) for our ranking ladder. The problem is that your smart superusers will most likely figure out this ranking scheme.
So, how do you keep the superusers engaged under such a predictable (boring) ranking criteria? I will describe 4 things that you can do to spice up your ranking ladder even when it is a predictable and boring arithmetic progression.
1. Name your ranks creatively.
It is inevitable that some superusers will eventually figure out the post requirement for promotion, and will expect a promotion at the right time. But if you name your ranks cleverly, they still won't know what they are going to get next. So, don't name your ranks in any obvious progression. For example, a terrible choice of rank names would be bronze, silver, etc. Without even writing it down, it is blatantly obvious that the one after silver is going to be gold, then probably platinum, titanium, and then diamond. Give your superusers some serendipitous joy when they get promoted and use your imagination when naming your ranks.
2. Make some noise.
For statisticians, there is a simple trick to make things less predictable. Just add some noise (or randomness)! Rather than following any formula strictly, you just need to randomly jitter the number a little bit. For example, it is very easy for anyone to figure out the pattern in the following linear progression: 10, 20, 30, 40, 50, 60, 70, etc. By add some random noise, the sequence becomes much harder to predict: 11, 18, 32, 41, 49, 63, 74, etc. Notice that the challenges between the ranks will remain roughly the same (compare the figure of the jagged ranking criteria here with the figure of the smooth ranking criteria from my previous blog post). The shape is virtually the same, but this one is much harder to figure out. You can start jittering the post criteria from the first rank if you wish, but please be sure to "make some noise" when you switch over to the highly regular linear progression.
3. Privileges do matter.
Even with added noise and the most humorous and interesting rank names, superusers may still get tired of the routine rank changes without other incentives. This is especially true after you switch the ranking criteria over to the highly predictable linear progressions to avoid over challenging your superusers. If you follow the recommendation in this miniseries of blogs, this should happen roughly 2 years after the superuser first participated in the community. After 2 years, don't you think it is time to show your superusers that you recognize how special they are? So tell them (it only takes an email) and grant them access and customization privileges that are not available to others. Start with something simple, such as allowing your superusers to use a personalized icon. Then as they move up the rather boring rank ladder, they will be intermittently rewarded with different privileges that are totally unpredictable.
Attaching special privileges to a rank through our permission system is an extremely effective way to engaging your superusers. However, you might want to let time test their loyalty and goodwill before giving out too many privileges. Therefore the perfect time to give out special privileges is after the superusers have moved through the first 24 ranks. However, if you really trust your superusers, you may do this earlier on your ranking ladder. This also applies when switching your superusers into a MVP program.
4. Show your trust.
An important reason to give special permissions to your superusers is to establish trust. A deep relationship built on trust is likely to be more lasting than a superficial relationship that is built on other incentives. So show your superusers that you trust them. Invite them to beta test programs in private boards and treat them as your most valuable assets. Give them power to take action in the community, such as moving posts and perhaps deleting inappropriate content. Let them help you moderate your community. Can you really trust the superusers with that much power? I believe that you should. Because it is highly improbable that a superuser would just throw away all the reputations and privileges they've earned through hard work over long period of time. The more they have invested, the more they will treasure their unique social status.
Now that your rank ladder is all spiced up, you should be able to keep your superusers engaged "indefinitely" (contrast the above figure with the figure from my previous blog post). Next time we'll wrap up with the grand finale: synthesizing all the rank design principles we've explored to build a complete ranking structure. Stay tuned at mich8elwu.
... View more
It is possible! However, this require Lithium specific data, such as kudos and accepted solutions. And if we do that, CHI would no longer be a general metric that is applicable to other community vendors.
... View more
Last time we talk about the content health factor and how we plan to reformulate it to enable drill down capability for the health factors and CHI. This time, I will explore the third health factor: Members.
The Members Health Factor
After achieving a steady stream of traffic and copious amount of high-value content, a healthy community should accumulate registered members, grow, reach critical mass, and then become self-sufficient and self-sustained. The members health factor is intended to measure the growth rate of the community.
Recall that traffic is a measure of passive engagement, and content is a measure of active participation. In Lithium-powered communities, login is not required for passive engagement, but it is required for active participation. This means that users must register to become members of the community and must sign in before they can post. However, because people tend not to register or log in unless they have to, the members health factor is good estimate for the conversion rate from passive engagement to active participation.
What about Member Churns and Lurkers
Although growth is a sign of a healthy community, member attrition is inevitable. Some users register merely to post a question, and they will leave as soon as they have the solution to their problem. Therefore, even healthy communities will have some natural member churn rate. But a high attrition rate is indicative of poor community health because the community is not able to retain members and grow its active core of user base. This issue has been brought to my attention at the Persuasive2009 conference.
Another question I often hear is that "what should we do about lurkers?" If you are not familiar with social media you might think that lurking is an undesirable behavior. But in a series of studies, Prof. Blair Nonnecke and Prof. Jenny Preece have shown that lurkers do provide value, and they are important to the success of community. So let your lurkers lurk! When the time comes--when they have a specific question to ask or some information to contribute--some of them will de-lurk and become members.
The Current and Future of Members
Currently, the members health factor is the same as the weekly registration metric, so it only measures the passive to active conversion rate. To get a complete picture of community health, it is important to track member attrition in addition to member accumulation. So in our next iteration of CHI, the members health factor will account for member churn.
This covers the three diagnostic health factors: Traffic, Content, and Members. They are called diagnostic because they describe the result of user behaviors and tell you the current health of the community. As such, they are lagging indicators of community health. Next time, we will start with the predictive health factors: Liveliness, Interaction, and Responsiveness. They are referred to as predictive because they describe the intrinsic social dynamics within the community, so they tend to be leading indicators of community health.
Let me close by reaffirming that all comments are welcome, so let me know what's on your mind. We do take your comments seriously, and you can help us make CHI more accurate and better suited to everyone's needs.
... View more
The post count is still the biggest factor that will influcence the Content health factor. Because the viewership is a dampen version of page view. It's affect will also be dampened. However, page view generally is a much larger number than post count. If you have a huge boost in Traffic, you can also expect a significant increase in Content. But it would have to be a very huge increase, like orders of magnitudes difference.
... View more
Thank you for the question. Scott was right. CHI was developed with the goal that everyone who like to compute it should have sufficient data to compute it. So we excluded metrics that are specific to Lithium, such as kudos and accepted solutions. But don’t get the impression that encouraging your members to give kudos and mark solutions has no effect on your community health. It does! But it does so indirectly.
Because rated and validated content are more useful, communities that have many kudos and accepted solution tend to drive more traffic and viewership. So even though kudos and accepted solutions do not directly affect CHI, they certainly can drive traffic and affect your community health through the traffic health factor and the content health factor by increasing viewership.
... View more
My previous post focused on the "traffic" health factor - where it is today and where it's headed in the near future in terms of reformulation. Today I'll be talking about another health factor: Content.
The Content Health Factor
Once critical mass is reached in terms of human traffic, the next thing to focus on is building content within the community. Visitors won't return time and time again without an abundance of interesting, useful and highly desirable content. The content health factor is a measure of both the quantity and the quality of the messages posted within your community. Contrary to traffic, which measures the passive engagement of visitors, content measures the active participation of your community members. Because posting (whether it is a message, a reply, or a comment) adds consumable information that is persistent within the community. This is a form of active participation.
Measuring the Quantity and Quality of the Posts
The post count metric provides a straightforward measure of the quantity of posts, but how do you measure their quality? We leave this decision to the readers. Using a marketplace metaphor, when the number of consumers (readers) in a community is large, the "economics" of the community can give an accurate estimate of the relative demand (whether they are useful or interesting) for the posts. Since the demand for a post strongly correlate with its viewership, the demand for a post must be reflected in the page view metric. However, highly viewed pages tend to draw more random views. This snowball effect will inflate the estimate of consumer demand. Therefore, the post quality can be approximated by a dampened version of the page view metric, which we call viewership.
The Current and Future of Content
In the current implementation of CHI, post counts and page views are aggregated over the weekly window, and then computed at the community level. However, this computation prohibits any drill down capability for this health factor. Yet, drilling down to a category or a board and seeing the content health factor at different hierarchy of the community can provide actionable intelligence for the community manager. Because this is a common use-case, and personally I've been asked this specific question at our customer conference earlier this year, I will make sure that we address this issue in the next reformulation of CHI. This drill-down view can be achieved by computing the product before the aggregation.
Can you guess what is coming? Yes, we will talk about the member health factor next. In the mean time, please do tell me if there is a feature you want with regards to measuring community health. Stay tuned at mich8elwu.
... View more
Welcome back. I hope everyone had a relaxing Fourth of July holiday. Last week, Lithium delivered a new report, the community health report, to our clients. This report contains two important pieces of new information:
The community health index (CHI): a score that reflects the "health" of your community.
The community health compass: a radar chart that shows the relative value of the six health factors (traffic, content, members, liveliness, interaction, and responsiveness) that went into the computation of CHI.
Earlier this year we published a whitepaper on CHI. Since then, however, we've refined the CHI algorithm based on data from real-world use cases along with customer feedback. So for this post, I'd like to talk about the current state and future direction of CHI.
The Traffic Health Factor
Let me begin with the health factor you are most familiar with: Traffic. This health factor is most important when you are launching a community. If no one comes to your new community, the community will fail. As the community matures and acquires a large core of active users, the effect of traffic may become less important. This is because traffic reflects only the passive engagement of the visitors with the community. In a public community, visitors can accomplish quite a bit without any active contribution. For example, they can read messages, search for answers, navigate the community, checkout other members' public profiles, etc. These activities are considered "passive" because the visitors merely consume existing community content, but do not add content of their own.
Measuring Human Traffic is Not That Simple
In short, the traffic health factor is intended to measure the amount of human visits to the community. However, traditional page-view metrics also counts non-human visits by web robots and crawlers. Moreover, counting page views on a modern AJAX-filled website is not trivial. Because Web 2.0 technology furnishes users with a wide variety of interactions on a very dynamic webpage, users can potentially visit many places and perform many activities without refreshing the page. As a result, behind-the-scene REST API calls, RSS feeds, or any server-rendered pages go unnoticed to conventional page-view trackers such as Google Analytics.
The Current and Future of Traffic
In the current formulation of CHI, traffic is measured by Lithium's own PageView metric, which includes REST calls and any server-rendered pages. Although our preliminary study suggested that robots and crawlers do not significantly affect the final CHI score of the community, we intend to remove their contribution to our PageView metric in our next formulation of CHI. However, since robots do not always declare their identity (in fact some crawlers may even intentionally disguise themselves as human visitors), it is not possible to completely remove the effect of robots and crawlers. Despite this, through iterative reformulation, we hope to derive a traffic health factor that can track the passive engagement of human visitors more accurately.
I hope this post gave you a better understanding of the traffic health factor. Next time, let's talk about the content health factor. In the meantime, feel free to continue this conversation in the comments section below.
... View more