Big Data Needs To Think Bigger.

Editor’s note: Guest authorSemil Shah is an entrepreneur interested in digital media, consumer Internet, and social networks. Heis based in Palo Alto and you can follow him on twitter@semilshah

Spend enough time in Silicon Valley, and of all the buzz words you’ll hear neatly tucked in with “graph,” “serendipity,” and “personalization” is one often uttered though, on the whole, not yet fully understood: “Big Data.” On the surface, everyone realizes the opportunity. Data is being generated at lightning speed, the cost of storing is tiny, and new technologies are available to help manage, organize, and secure the data. Earlier this month, LinkedIn co-founder and Greylock partnerReid Hoffman delivered apresentation on this topic at SxSW, and starting next week, GigaOM’s annual big data conference “Structure’” kicks off in NYC.

At the consumer level, while we are wowed by pretty visualizations, the real advancements in big data technologies cover (1) how data is structured and stored, (2) how it is organized and retrieved, and, most interesting to me, (3) how underlying mathematics can be written into algorithms to leverage the data and help discover entirely new things. I’ll paraphrase from one data scientist, LinkedIn’sPeter Skomoroch, who notes onQuora that cheap data storage allows users to leverage asymmetric information, larger data sets increase the likelihood that new insights can be found, and machine learning advancements can be used in entirely new, game-changing ways.

Continues @http://techcrunch.com