Big Data - The next big thing to come... or is it already here?

This is the trailer from a movie called "Moneyball", which is based on the true story of how a low budget baseball team was formed from players who were picked according to a statistical analysis of many players. This is an example of "big data" application. Big data is the systematic analysis of enormous amounts of information and extraction of knowledge (or wisdom) from this an analysis.

Data are being recorded more than ever. Internet in all forms, mobile phones, sensors, are recording continuously and in digital - standardized format all kinds of data. Heart beat rate of runners, traffic in the cities, searches in the web, weather, news, emotions (through social media), location of smartphones, and many other details that there is no point to list now. Recoding is not producing any knowledge. This amount of information is not even manageable by humans alone. This is the playground of computerized processors! Humans are inventing algorithms that analyze with the help of computers the available data. Of course within the limits of our current imagination.

Data are stored though and are available for ever, with the new technologies of the "cloud". They will be there for generations to come. Computers are starting to learn! Because simply... they have unlimited time to look into and combine all this information. And they are way faster than humans in analyzing huge amounts of raw data. And when a new algorithm is available they can look back at all the stored data with the new "eye" made available and enrich their knowledge. So the possibilities are only expanding.

This is a trend that cannot be reversed. Simply because our lives are improving through this procedure. Some examples:

• Less traffic or plane accidents
• New health procedures, like epidemic analysis and reactions (Personal Monitoring Tech: Invasion of the Body Trackers)
• More convenience in everyday life (for example, location services and personalized information on demand and in real time)
• Longer weather forecasts
• Superhuman fast stock market reactions and complex financial services
• Better pricing and distribution of products at retail (super markets for example can plan their stock according to weather forecast, or customer behavior analysis!)
• Urban crime control through analysis of events in correlation to various parameters like historical arrest patterns, paydays, sporting events, rainfall and holidays!

And for all these you do not need to conduct a survey as in the past. Most likely all the data anyone will ever need are there already. You just have to think of the best way to "dig" in them and extract the knowledge you need.

Of course, as written in a very enlightening New York Times article: "Data is tamed and understood using computer and mathematical models. These models, like metaphors in literature, are explanatory simplifications. They are useful for understanding, but they have their limits. A model might spot a correlation and draw a statistical inference that is unfair or discriminatory, based on online searches, affecting the products, bank loans and health insurance a person is offered, privacy advocates warn." But even this kind of problems can be solved much faster than in the past. Remember... data are there for ever and they only get more and more!

The main issue that must be addresses and resolved is access to the available data. I believe that recording is not bad for anyone, as long as the access to the records are open to anyone! This is the only way that true wisdom can be extracted from all this information. This the "Open Data" movement which is affecting all of us!  The Guardian has been encouraging this movement for some time now.  This a society issue and in my opinion it should be ranking very high in our priorities for the future. I would even call it a political issue!

Of course, there’s a major shortage of analytical talent. According to the McKinsey Global Institute (MGI), this is already an issue in the private sector. MGI predicts that in the US alone there will be an annual shortage of graduates in deep analytical fields of 140,000 to 190,000 by 2018. A very good article on this subject is from Linda Rosencrance of the Spotfire Blogging Team "The Promise and Challenge of Using Big Data to Address World Problems".

Big data in the end is all about wisdom and evolution! And obviously this cannot be left to the hands of few or only in the processing power of computers!

Consumption and the hijacking of our lives

From the day we are born, some role models are imposed to us and we are trying to be like them. Society accepts those who have a car, own a house in a city, have a phone, a TV, a mobile phone. Society admires people educated to be lawyers, doctors, scientists, engineers, teachers. Trendy clothes and brands have to be worn.

A contrast: What happens when you own a bike? (BBC News - Cycling industry gives economy £3bn boost http://bbc.in/qiwMpM) When you commute either with your own two feet, or with a bike, or using the public transportation for longer distances? How things would be if we did not own the houses we are staying in the cities, but we were renting (maybe as part of the taxes we pay) them from the state according to our income? What if phones and mobile phones were developed under certain standards and specifications and the focus was on the service provided by the operators through them? What if there was only one TV device per house? How about admiring farmers, dancers, painters, musicians, carpenters, palaeontologists and plumbers as much as engineers, mathematicians, doctors and lawyers? What if the clothes were picked according to their functionality and not according to the little label on the chest?

I bet the world would be a much better place! Lets try to imagine:

Houses built according to certain standards by the state. Maintained by the state and demolished and rebuilt  every 50-60 years by the state! Costs covered through taxes we pay. Families would not have to invest from their budget (usually taking 30-40% of the family income) for the housing project. Standards could be observed more closely. Speculation, price inflation and housing market bubbles would be unknown terms! After all, building a house, except for the brief period of the construction, it is not a very productive way to invest ones resources! This model would apply in all cities. Parks and public spaces could be organized better. Long term planning of the community would be much easier. Architecture would have unlimited space to develop new technologies. Energy consumption could be more reasonable, helping the environment. Resources and wealth beyond our imagination which now are being "frozen" in the value of bricks and walls and private swimming pools would be made available for investment to the productive economy!

The commuting vision: People would be able to move around in the cities on safe roads with their bikes or by walking or other human powered vehicles. For longer distances or when the circumstances are not favorable they would be able to use public transportation like trains, trams, electric buses. Accidents costing endless millions to state budget and cause unbelievable personal misery would be reduced! Public health would improve by having less obese citizens and better quality of air. Unreasonable social discrimination based on the kind of vehicle that someone owns would be unknown. Cities would have more public space. State would also save a big chunk of its' budget which goes to maintenance of the roads and for building new ones to accommodate the increasing number of cars and their parking needs. Automotive companies would be focused on engine technologies and not on pointless chassis design projects that offer nothing substantial to our lives. And finally, personal finance resources "blown in the wind" for a new car that is losing it's value as soon as you open the door for the first time, would find better ways of use. Public transportation would require increased budget from the state, but it would be way lower than the requirements for new avenues and their maintenance, or for addressing public health issues, created by the low quality of life in the cities, or by car accidents. 

Singers, dancers, painters, gardeners would be equally important with all the professions. Because the society would have available more free time and more resources to invest in the public welfare and culture. We would prevent the overpopulation in certain professions and allow personalities to develop better, through the education system. Universities and schools could be real cultural centers and not mass production facilities of brains that think in the same way and lose most of their capability to question and improvise!Social discrimination based on ones job title would be reduced, and social value would be attributed mostly to the quality of his / her skills. Excellence in personal level, would be encouraged and rewarded by the society according to the skills and not the perceived value of the profession.

Many more examples could be presented here describing this ideal world. The few, utopic-at-first-sight, cases I described above, highlight some simple imbalances  in our lifestyle. Automotive companies compete on the fields of chassis design and financial solutions (basically loans) for their cars, instead of engineering breakthroughs for the engines that move these cars. People get loans, which will last more than their productive lifetime, to buy bigger and more energy consuming houses than they need. They suppress their creativity and natural predisposition towards the development of skills, in order to be socially accepted. They end up serving non productive positions in multinational companies, without any clear view of the outcome of their day to day work, just because they are socially accepted (probably because of their 6-digit salaries, or collection of credit cards). Because of the intense social focus in certain sectors of economy, value inflation is making this endless pursue of the "dream" even more unobtainable. Consumption without any real need behind it, is the hollow foundation of the development of modern societies.

People need to open their eyes and realize what is the real value of the skills, items, services that they spend their time and resources in their ongoing effort to acquire. We are wasting our lives and resources in pursue of goals that we have not chosen! And we are dragging down with us the whole planet...

(by the way: The housing model described above, has been successfully implemented in Singapore for the past 30 years, the automotive model is the one under which this industry started more than a century ago, and was lost along the way)

Interesting to read: Was Marx Right? - Umair Haque - Harvard Business Review
Interesting to watch: Surviving Progress

Interesting to watch: http://youtu.be/4Z9WVZddH9w