Thursday, April 25, 2013

More on big data

For those interested in an in-depth look at big data beyond the brief overview in last week's Disruptive Tech post, I recommend the following Foreign Affairs article:

The Rise of Big Data | Foreign Affairs:

'via Blog this'

As the authors point out, big data represents the Information Age approaching a milestone of maturity. A nexus of processing power and access to data sets so vast as to offer insights on societies and economies unavailable from any other viewpoint. The 90s may have been decade when we bandied about the term "information revolution," but those were very early days in a process of transformative development that may well stretch out over several decades.

Then there's that question of the worth of human intuition in the new data environment.

Me, I like human intuition. It's a powerful tool that quickly keys in on information in noisy natural channels, and lets us make life or death decisions in heartbeats. It's unparalleled in its ability to analyze and sport patterns hidden in social situations involving small to medium groups. It also lets us make important real world choices in situations when there's very little information to go on.

All the same, it's of limited value when making large scale choices, or when confronted at with huge sets of data. When it comes to the abstract and the big, intuition is often harmful, imposing what we expect or desire to see on chaotic information.

There's a reason statistical analysis is a critical component of experimental design in science.

So, will we change ourselves to make human intuition relevant to the new economy?  What does that mean anyways? Augmenting our focus and memory; deadening the desensitization that takes place when looking through page after page, screen after screen of raw data? Tweaking our pattern recognition filters? Adjusting our emotions to find nuts and bolts minutia of big data inherently fascinating?

All interesting possibilities. Or will we rely on machine agents to analyse and explain the findings of big data analytics? Also a very real possibility, and one deserving of exploration.

No comments: