Big Data - The next big thing to come... or is it already here?

This is the trailer from a movie called "Moneyball", which is based on the true story of how a low budget baseball team was formed from players who were picked according to a statistical analysis of many players. This is an example of "big data" application. Big data is the systematic analysis of enormous amounts of information and extraction of knowledge (or wisdom) from this an analysis.

Data are being recorded more than ever. Internet in all forms, mobile phones, sensors, are recording continuously and in digital - standardized format all kinds of data. Heart beat rate of runners, traffic in the cities, searches in the web, weather, news, emotions (through social media), location of smartphones, and many other details that there is no point to list now. Recoding is not producing any knowledge. This amount of information is not even manageable by humans alone. This is the playground of computerized processors! Humans are inventing algorithms that analyze with the help of computers the available data. Of course within the limits of our current imagination.

Data are stored though and are available for ever, with the new technologies of the "cloud". They will be there for generations to come. Computers are starting to learn! Because simply... they have unlimited time to look into and combine all this information. And they are way faster than humans in analyzing huge amounts of raw data. And when a new algorithm is available they can look back at all the stored data with the new "eye" made available and enrich their knowledge. So the possibilities are only expanding.

This is a trend that cannot be reversed. Simply because our lives are improving through this procedure. Some examples:

• Less traffic or plane accidents
• New health procedures, like epidemic analysis and reactions (Personal Monitoring Tech: Invasion of the Body Trackers)
• More convenience in everyday life (for example, location services and personalized information on demand and in real time)
• Longer weather forecasts
• Superhuman fast stock market reactions and complex financial services
• Better pricing and distribution of products at retail (super markets for example can plan their stock according to weather forecast, or customer behavior analysis!)
• Urban crime control through analysis of events in correlation to various parameters like historical arrest patterns, paydays, sporting events, rainfall and holidays!

And for all these you do not need to conduct a survey as in the past. Most likely all the data anyone will ever need are there already. You just have to think of the best way to "dig" in them and extract the knowledge you need.

Of course, as written in a very enlightening New York Times article: "Data is tamed and understood using computer and mathematical models. These models, like metaphors in literature, are explanatory simplifications. They are useful for understanding, but they have their limits. A model might spot a correlation and draw a statistical inference that is unfair or discriminatory, based on online searches, affecting the products, bank loans and health insurance a person is offered, privacy advocates warn." But even this kind of problems can be solved much faster than in the past. Remember... data are there for ever and they only get more and more!

The main issue that must be addresses and resolved is access to the available data. I believe that recording is not bad for anyone, as long as the access to the records are open to anyone! This is the only way that true wisdom can be extracted from all this information. This the "Open Data" movement which is affecting all of us!  The Guardian has been encouraging this movement for some time now.  This a society issue and in my opinion it should be ranking very high in our priorities for the future. I would even call it a political issue!

Of course, there’s a major shortage of analytical talent. According to the McKinsey Global Institute (MGI), this is already an issue in the private sector. MGI predicts that in the US alone there will be an annual shortage of graduates in deep analytical fields of 140,000 to 190,000 by 2018. A very good article on this subject is from Linda Rosencrance of the Spotfire Blogging Team "The Promise and Challenge of Using Big Data to Address World Problems".

Big data in the end is all about wisdom and evolution! And obviously this cannot be left to the hands of few or only in the processing power of computers!

Connected Wearable Computers!

Modern smart-phones have more processing power than the best desktop computers of the '90s. On top of this, they are always connected through GSM or WiFi networks and most of them include, camera, accelerometer and a GPS device. All in one small package that fits in every pocket!

There are two major limitation for the current smartphone technology: Power supply (or batteries), and display "real estate" or size of screen. These two, are also inter-dependant, since larger screens consume more energy. A large battery makes the device bulky and heavy, a large screen makes a smart-phone un-practical as well...

The power supply of a smartphone with all the fancy features described above, allows usually a few hours of usage of all the features. Internet connection and GPS consume a lot and a typical smartphone battery could sustain non stop operation for not more than 4-6 hours. Combined with the larger screen sizes, that enable the user to have access to the information (e.g GPS location on an internet retrieved Google Map) available to him by his phone, can empty a battery very fast.

Lets look at some other devices - apparel that most modern people carry with them everyday. The wrist watch is an example. There are many models of watches that include various sensors, like barometer, thermometer, accelerometer, digital compass or even GPS sensor. The battery of a wrist watch lasts way longer than any of the smart phones. Their screens are very low on energy consumption but also very specific in their use. This new generation of "smart" watches also has some connectivity features, that enable the user to transmit the stored data to a web-service or even retrieve data from remote sensors, like heart rate monitor or power meters in the occasion of a sports watch (like the one at the side picture).

Also many are wearing glasses and sunglasses. This "low" tech apparel is always in front of our eyes, protecting them from harmful light rays, particles or correcting our vision or even for pure aesthetics. Recently we have seen some new products in this segment, starting from the "heavier" sports goggles, where the user can connect wireless to the smartphone and use the goggles as earphones and microphone, or even embed a micro projector to create a Head Mounted Display (HMD) screen.

 

There are many more examples of new products that are exploiting the unprecedented computing power that each one of us has in his pocket with a smart-phone. There are also many sensors that have been introduced, that enhance the capabilities of smart-phones:

• Sports activities sensors (heart rate, power, stride / cadence, etc)
• Health status (heart rate, blood sugar level, etc - Personal Monitoring Tech: Invasion of the Body Trackers)
• Car monitoring services (connectivity to diagnostics - OBDII)
• Connected external mini cameras (Contour+ is such an example)
• and many more...

We should not neglect to take into consideration the market demand for new features which could be served very well by an "expanded" smart-phone. A whole new range of media and marketing possibilities will open-up, like:

• Real life social games
• Augmented reality applications
• Big data statistics and marketing (massive amounts of information coming from recording at a scale never possible in the past)
• Health wellness applications
• Safety on the road
• Public transportation information
• Urban life management
• etc...

With all these things in mind, I believe that the solution to the two major limitations of smart-phone technology, can be found in the modification of the device in the direction of peripheral connectivity and delegation of tasks. The smart-phone will be the main unit, providing connectivity and processing power, while peripheral devices will be deployed according to the needs, and will be supplying the data needed.

Connected wearable computers with standard communication protocols should be able to keep their screens off for power efficiency, while on demand could be presenting information, like caller id and SMS - micro blogging messages at the screen of the connected wrist watch. Eye-wear could work as microphone and headphone and save even more energy for the smart-phone. At an advanced stage special glasses could work as head mounted displays, feeding the user with information according to his needs. It is very important to have a standard communication protocol with all these devices, because today there are at least 3-4 major protocols available making the development and marketing of such applications very complicated. Freeing some of the processing power of the main unit with this delegation of tasks, would allow development of more demanding applications as well, like voice recognition (iPhone Siri is such an example) that would make interaction with the information so much easier.

Consumers would benefit a lot as well! Imagine if you could upgrade only the accessory you need, instead of buying a whole new device (current smart-phones) every time there is a better photo-camera, or more sensitive GPS or more powerful processor for the phone! If you break one thing, you replace only this specific component and not the whole device. It is also a bit more environment friendly set up.

Trying to fit everything in a box is working only at the initial steps of a new technology. From some point on modularity has to be introduced in order to become more efficient, flexible and release the full potential of the new technology. The market will open up as well to so many more companies with new products that will complement the pocket hidden smart-phone.