Modern Metrix

The Rise of Data Scientists? Apparently, Not Fast Enough

As data become more abundant (and messy), there is increasing demand for more specialists who could not only separate noise from proverbial "golden nuggets" of information, but also make those "nuggets" useful and actionable:

"Hilary Mason, chief scientist for the URL shortening service bit.ly, says a data scientist must have three key skills. "They can take a data set and model it mathematically and understand the math required to build those models; they can actually do that, which means they have the engineering skills…and finally they are someone who can find insights and tell stories from their data. That means asking the right questions, and that is usually the hardest piece."

It is this ability to turn data into information into action that presents the most challenges. It requires a deep understanding of the business to know the questions to ask. The problem that a lot of companies face is that they don't know what they don't know, as former U.S. Defense Secretary Donald Rumsfeld would say. The job of the data scientist isn't simply to uncover lost nuggets, but discover new ones and more importantly, turn them into actions. Providing ever-larger screeds of information doesn't help anyone."

How Americans Spend their Money: Break-Down of Expenditures for Average US Household

NPR has interesting visualization that summarizes findings of Bureau of Labor Statistics from their most recent Consumer Expenditures Survey:

The article also looks at how American's spending habits changed over last several decades. In particular, compared to 1949 data, Americans nowadays spend considerably less money on food (15% in 2011 vs. 40% in 1949) and clothes (3.6% in 2011 vs. 11.7% in 1949), but more on housing (41% in 2011 vs. 26% in 1949) and transportation (16.9% in 2011 vs 7.3% in 1949). 

Update: here is an alternative visualization, with more expenditure categories. 

Free Twitter Data!

I am always on the look out for free data to play with, and today stumbled upon awesome depository of Twitter data on Infochimps website. Check it out!

 

Neighborhood Lifestyle Metrics using Foursquare Data!

In this post on their blog, Foursquare engineering team describes how they used Foursquare activity data to compute lifestyle metrics of city neighborhods, and then used these metrics to compare neighborhoods across different cities:

"First, to collect data about neighborhoods, we built some Hive queries to access our large collection of check-ins (stored in S3) and count the number of check-ins per category for every neighborhood in the world. For example, the East Village of New York has 230k check-ins at bars, 57k check-ins at pizza places, 18k check-ins at yoga studios, and 34k check-ins at karaoke places (hipsters like to sing!).

We then used MATLAB to represent each neighborhood by a 400 dimensional vector which specifies the normalized probability distribution of checking in to a place in each category relative to the baseline distribution of the city. This approach allows us to compare neighborhoods with each other using a similarity metric such as cosine similarity, or KL divergence."

What an awesome hackday project!

Impact of Survey Length on Measurement Quality and Participation Rates

This interesting study by Lightspeed Research looked at the relationship between the online survey length and quality of measurement and participation. In summary, study found that longer surveys result in higher drop out rates, and less reliable measures of constructs of interest. Interestingly, there was relationship between certain demographic and probability of dropping out, although that did not necessarily affect the representativeness of the overall sample:

"Respondents who drop out are somewhat different than those who complete the survey. Demographically, they are likely to be older, retired, male and from smaller households without children. In addition, those who drop out are less likely to do all the grocery shopping and tend to purchase snack foods less often. Despite their differences, the dropouts are so few in number that their exclusion does not harm the representativeness of the sample."

The study also found that, possibly, due to the fatigue, the quality of certain measure was affected by the survey length: 

"In the Lightspeed Research study, the concept scores for the 8-minute questionnaire tend to be higher than scores for the longer questionnaires. The 20- and 24-minute questionnaires receive fewer “definitely would buy” ratings compared to the 8-minute version. There are also several notable differences on liking, value, uniqueness, and believability between the 8-minute version and the longer versions (see Table 2). Also, several important differences occur between the 8-minute questionnaire and the two versions with the extra upfront questions."

The position of questions pertaining to the main concept in the survey relatively to other attitudinal and demographic questions may also result in biased and poor quality responses. 

To Rent or To Buy?

Check out this awesome interactive map that visualizes rent vs. buy ratios across 50 largest cities in the US. Yes, I live in New York City. Yes, I rent. No, I don't plan to buy!

Ratio of Single Men to Single Women across NYC Neighborhoods

Check out this awesome map that visualizes the differences in ratios of single men to single women across various part of New York City. The map was created by NYC Economic Development Corporation, based on Census data. Make sure to check out their original blog post with more details, and also similar visualization of cats/dogs ownership rates in NYC. 

New York City Council Frees NYC Public Data

This is great: NY City Council passed yesterday the Intro29A measure that requires all city agencies to make their public data available on NYCOpenData portal:

"Intro 29 would phase in over the next six years, starting with a technical manual to be released by DoITT this summer guiding city agencies on how to format and upload their data. Agencies would have until early next year to upload all public datasets currently on the internet to the new portal. By 2018, all public data would have to be posted. For the time being, DoITT is planning to power the new data portal with Socrata, the service that currently hosts NYC OpenData."

The Rise of Self-Tracking Reports

It looks like life-logging is becoming more popular . I myself have been looking lately into various tools for collecting lifestyle, biometric, psychographic, and sociometric data points on a daily basis, and soon will start posting reviews of some of these tools on my blog. Not sure if I will ever be ready to release the summary reports, though...

 

Using Consumer Data to Change their Shopping Habits

Fascinating article in NYT magazine on how companies use consumers' data to change and shape their shopping habits. 

"As Pole’s computers crawled through the data, he was able to identify about 25 products that, when analyzed together, allowed him to assign each shopper a “pregnancy prediction” score. More important, he could also estimate her due date to within a small window, so Target could send coupons timed to very specific stages of her pregnancy."

Unfortunately, author makes it sound a bit too creepy and invasive. But yep, that's what we do.