While searching for some articles on influence, I stumbled upon this blog post by Christopher Penn, where he was challenging readers to dig into a small dataset of Klout and PeerIndex score to explore the differences between these two metrics of influence. I quickly ran some numbers which I included in my comment to his post, but would like also to share here.
The dataset that Chris was kind enough to post contains Klout and PeerIndex scores for 15,737 Twitter users who tweeted at least once in the past month, presumably on topic related to #Marketing (data is available on his blog, in CSV format). I first computed straightforward Pearson's correlation between two scores, which turned out to be .45. The rank-order correlation (correlation between the ranks of the scores instead of their actual values) was slightly lower (.43). Finally, I assigned the scores to quintiles (5 equal groups; think of them as top 20% of users with highest scors, then next top 20%, etc., ending with bottom 20%) and looked at the correlation and cross-tabs between the quintiles. The correlation was .42, the average overlap between two scores (that is, user who was in the top 20% on Klout score wa also in the top 20% on PI score was about 30%). And here is a modified confusion matrix for all 5 quintiles, showing the overlaps between the groups:
As you can see, the top and bottom 20% had higher overlaps. Overall, there is a positive relationship of moderate strength between two metrics. But only moderate. One possible explanation for discrepancies is that Klout and PeerIndex use different reference groups. Remember, the scores are computed, normalized and compared within a larger group of users. Since it is nearly impossible to download and process data on ALL Twitter users, so most likely, these services draw random samples of people on Twitter and then use them as reference groups.
