Modern Metrix

The Rise of Data Scientists? Apparently, Not Fast Enough

As data become more abundant (and messy), there is increasing demand for more specialists who could not only separate noise from proverbial "golden nuggets" of information, but also make those "nuggets" useful and actionable:

"Hilary Mason, chief scientist for the URL shortening service bit.ly, says a data scientist must have three key skills. "They can take a data set and model it mathematically and understand the math required to build those models; they can actually do that, which means they have the engineering skills…and finally they are someone who can find insights and tell stories from their data. That means asking the right questions, and that is usually the hardest piece."

It is this ability to turn data into information into action that presents the most challenges. It requires a deep understanding of the business to know the questions to ask. The problem that a lot of companies face is that they don't know what they don't know, as former U.S. Defense Secretary Donald Rumsfeld would say. The job of the data scientist isn't simply to uncover lost nuggets, but discover new ones and more importantly, turn them into actions. Providing ever-larger screeds of information doesn't help anyone."

Using Consumer Data to Change their Shopping Habits

Fascinating article in NYT magazine on how companies use consumers' data to change and shape their shopping habits. 

"As Pole’s computers crawled through the data, he was able to identify about 25 products that, when analyzed together, allowed him to assign each shopper a “pregnancy prediction” score. More important, he could also estimate her due date to within a small window, so Target could send coupons timed to very specific stages of her pregnancy."

Unfortunately, author makes it sound a bit too creepy and invasive. But yep, that's what we do.

Profiles of E-bay Users According to Hunch

Interesting infographic report from the "personalization graph" company Hunch that developed psychographic and behavioral profiles of e-bay users based on data from 70,000 respondents. Among the most interesting findings: e-bay users are more likely to be happy with their job , prefer impressionalism and surrealism when it comes to art, and read "The New Yorker". Non-users, on the other hand, are more likely to be unhappy with their jobs, prefer realism and romanticism, and are more likely to read "People" magazine. 

4.74 Degrees of Separation

Interesting article in NYTimes about most recent replication of Milgram's famous "small world experiment". The original research introduced concept "six degrees of separation" - that any two people in the world are separated by only six intermediate connections. According to Facebook scientists, who analyzed data on 721 million active Facebook users with total of 69 billion friendships, that number now is down to 4.74. Of course, I would question definition of "friendship" used in the study (it was defined simply as a being "friends" on Facebook), nevertheless, the methodology and analysis are interesting. 

More detailed report is available on Facebook Data page here.  

What Can Your Electric Meter Tell About You? Your Favorite TV Shows!

For those of you who can speak (or read) German, this interesting paper discusses a study on how analyzing data from advanced electric meters can reveal which TV shows are particularly popular in your house.

 

Tweets and biological rhythms

NYT discusses a paper by Scott Golder and Michael Macy from Cornell University on correlation between the emotional tone of people's messages on Twitter and known biological rhytms (aka cyrcadian rhytms). Using text mining software and impressive sample (several millions of tweets), researchers classified random tweets based on their content (usage of words "awesome", "great' and emoticons) as either "positive" or "negative", and then looked at the patterns throughout the day. They found that for majority of people, positive tweets tend to appear in the morning (between 6 am and 9 am), slowly decline towards afternoon, and then rise again after dinner. In the beginning of the workweek the overall mood tends to be lower, with the curve peaking on weekends. The patterns seem to be the same across different cultures.  

 

Image source and more info are here

Googlers vs. Microsofties: Who Buys More Junk Food and Other Shopping Habits

RapLeaf looks at the shopping habits of Google and Microsoft employees in this fascinating study that utilizes Rapleaf's enormous database of identifiable information

Grab some free data on 215 mln. Facebook users, for free!

Pete Warden is making data on over 200 mln Facebook users available to academic research community this week. The fields in the dataset are limited to publicly available info such names, location, fan pages, and friends lists. Pete also shares some preliminary insights:

"In almost every state in the Southern U.S., God is number one most popular Fan page among Facebook users. Among people in the L.A., San Francisco and Nevada regions? "God hardly makes an appearance on the fan pages, but sports aren't that popular either," Warden writes. "Michael Jackson is a particular favorite, and San Francisco puts Barack Obama in the top spot." In the Oregon and Idaho region? Starbucks is number one.

In the Mormon-influenced areas of Utah and Eastern Idaho, the most popular Fan pages are The Book of Mormon, Glen Beck and the vampire book Twilight, which was authored by a Mormon.

The bulk of Warden's posted analysis yesterday was about location networks. People in the western U.S. tend to have Facebook friends all over the country; people in the southern U.S. tend to mostly be friends with people who have remained in the same area."

Eureqa, a computer program for fitting equations to empirical data

Wow, this is awesome! Andrew Gelman from Statistical Modeling, Casual Inference, and Social Science blog writes about computer program called Eureqa that helps to find a curve (and corresponding function) that best fits the given data. If you are tired of regular linear regression, and looking for more fancy ways to describe your data, then check it out!

Peerset: Next Generation Psychographic Hyper-Targeting Tool

Techcrunch profiles Peerset, web startup that offers tool for automated data-driven psychographic profiling and ad placement recommendations based on content analysis.