Next 2018: Puzzles vs. Mysteries

The next 2018 Hitachi Vantara event was recently held in San Diego, whose keynote speaker was Malcolm Gladwell, the New Yorker writer.

Malcolm has published five books that have been on the list of the best-selling Times: tipping Point, Blink, Outliers, what the Dog Saw and David and Goliath. Malcolm’s books and articles often provide unexpected ideas that help us understand the events around us and allow us to make better decisions.

In the general session, our CEO Brian Householder did an interview with Malcolm. Some of the ideas that emerged were: a model is as good as the data it contains, which it illustrated with the use of standardized tests of students to measure the quality of the Teachers. Another was to focus on the problem and not panic when circumstances changed. The example here was the music industry, which was frightened by the loss of revenue, but today it is earning more money than ever because of the live performances promoted by its music. I can see a similar transition in our industry, where the cloud was a threat to it vendors, but today revenues increase for it providers due to software and services, making it easier for their customers to develop applications and Generate Information.

Later, I had the opportunity to moderate a question and answer session with Malcolm and a group of VIP Clients. Here we begin with the paradigm of puzzles against Mysteries. Although Malcolm is known for his bestselling books, there are many ideas that are created in his New Yorker articles that are generating even more interest today.

In 2007, he wrote an article “open secrets”, in which he posed the paradigm of the puzzles against the Mysteries.

For example, Osama Bin Laden’s whereabouts were a puzzle. This was finally Settled. The puzzles can be solved by collecting more Data. On the other hand, what happened to Iraq after the fall of Saddam Hussein was a mystery. The mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have very little information, but we have too much. A mystery is a problem caused by an excess of information, and can only be solved by trying to make sense in a more sophisticated way to what is already known. Today we live in an era of information overload and all professions must manage the transition from solving riddles to solving mysteries.

While the address of the questions and answers did not allow him to delve into this issue, he has talked about this in other interviews, where he has said that most companies and industries are designed to solve puzzles, not mysteries. With greater access to data comes a greater Responsibility. Mysteries require a change in thought for which most industries are simply not organized or prepared. The mysteries complex require a different type of thought and analytical skills. You must decide whether the problem is resolving is a puzzle or a mystery. If you think you’re solving a puzzle and collecting more and more data, data overload could bury the pips key that could help you solve the mystery. The data that is used to solve a puzzle should be analyzed differently when solving a mystery. One of the biggest mistakes that companies make is to treat all the data equally by giving all the data the same weight.

This idea goes back to the concept of Nate Silver’s the Signal and the Noise. Some data will tell you exactly what you need to know and help you solve the mystery you are looking for, but most of the data is noise, which distracts you from the answers you are looking for. Today, the trick is not to analyze the data but to understand what data represents the signal to which it pays attention compared to the data that is the noise to Ignore. We have to start classifying the data in terms of its value.

The transition from puzzles to mysteries should resonate with most CIO’s in terms of their data. The databases were great for solving riddles, but the complex nature of today’s business is more of a mystery that requires Big data To be competitive, companies are changing data generation to Data-driven organizations, and Big Data systems are becoming the center of gravity in terms of access and storage Operations. The healing of data is necessary to understand the meaning of it, as well as the technologies that are applied so that the engineers can move and transform the essential data that the consumers Need.and Analysis. Hitachi Content Intelligence y Pentaho Data Integration son herramientas claves para buscar, clasificar, curar, enriquecer y analizar los datos para comprender qué tiene y cómo pueden usar para resolver misterios.

By: Hu Yoshida, Chief Technology Officer de Hitachi Vantara