Big Data is Dead: Long Live Predictive Analysis

Smart Technology

Big Data is Dead: Long Live Predictive Analysis

Businesses are collecting more and more data these days, but are they making the right use of the information they are gathering? The signs of big data indicate that they aren’t, and that in fact all they are creating is more and more complexity. Instead, they should focus on using data to make better decisions.

by Tim Cole

A buzzword, Big Data has been around for almost half a decade, and it was supposed to trigger a true revolution in the way companies view their markets and their customers. It was supposed to hand us the tools needed to exploit the endless growth of data. Instead, many companies and their IT departments complain more loudly than ever about “information overload”, and it’s true: the numbers are staggering. Facebook claims that it processes 2.5 billion pieces of content and over 500 terabytes of data daily. In addition, it collects an average of 2.7 billion “Likes” and 300 million photos a day. Every hour, Facebook scans more than 200 terabytes of data. And that’s just one company, although admittedly a very data-hungry one.
“A customer-focused business with Big Data in its grasp has an unparalleled source of knowledge from an increasing number of sources now; mobile data, social data, transactional data, locational data, financial data, family data, medical data, carbon footprint and consumption data”, writes Theo Priestly in Forbes. But increasingly, experts are worried that companies aren’t asking the right questions, namely the ones that will fill understanding gaps and help them interpret results. “Really effective analysis combines brilliant technologists and cutting-edge code we all recognize with human understanding, social science research, philosophy, and mission expertise”, says Peter B. LaMontagne, a blogger at Huffington Post.
To make matters worse, data is often stored very traditionally and “crunched” the old way, namely in batch processes. This begs the question: what use is that for real-time operational decisions? Another worry is that data is being collected indiscriminately and without any kind of fact-checking or ways to determine the credibility of those providing the information. Faced with a variety of sources, companies understandably choose to cherry-pick their data. Employees tend to select data from the easiest sources, or the ones with the least privacy or classification protections. And analysts are often happiest with sources that support their own view.

“Predictive Analytics: is the Power to Predict Who Will Click, Buy, Lie, or Die”

Eric Siegel
Former professor of data mining and Artificial Intelligence at columbia University

Big Data - Eric Siegel


“Today’s data reliability demands thatcompanies innovate, finding novel ways to pair analyst experience and expertise with automation, overcoming the velocity, volume, and variety of data they see”, LaMontagne maintains. His colleague James M. Connolly of Newsweek says the problem is the focus on bigger and bigger data. In reality, he thinks, it’s just good old data, stupid! “It‘s time to de-emphasize the “big” in “big data”, he writes. By making the whole enterprise analytics concept too complex, rather than focusing on the idea of using data to make better decisions, that type of complexity can turn nasty, he cautions.
In the end, Big Data is just data. Okay, there’s more of it, and it comes in more flavors; it is generated and transmitted faster than ever. But here are some questions we need to ask ourselves if we want to transform these masses of data into intelligent business decisions:
Which pieces of data really help us to create new insight and understanding?

  • Do we know how was this data was sourced, treated and stored?
  • Can we describe the results in terms a manager can understand?

Recently, a whole new field of knowledge management has sprung up which goes by the moniker “predictive analysis”, or PA. This is essentially an intelligence technology that aims to create a predictive score for each customer or organization. PA optimizes activities like marketing campaigns and website behavior to increase customer responses, conversions and clicks, and to reduce churn rates. Based on each customer‘s predictive score, actions can be taken with that customer. In his bestselling book, “Predictive Analytics: the Power to Predict Who Will Click, Buy, Lie, or Die” (Wiley, 2016), Eric Siegel, a former professor of data mining and Artificial Intelligence at Columbia University, sets out examples of what kind of information can be gathered from complex, multifaceted data streams, and how these can affect decisions within companies and government institutions, for instance:
Predicting which people will drop out of school, cancel a subscription, or get divorced before they are aware of it themselves.

  • Why early retirement decreases life expectancy and vegetarians miss fewer flights.
  • How European wireless carrier Telenor, and the Obama‘s 2012 campaign calculated the way to influence each individual.
  • How Target, a retail chain, figures out you‘re pregnant and how HewlettPackard deduces you‘re about to quit your job.

The tools for achieving these kinds of insights usually combine elements from such disparate fields as Data Mining, machine learning, and statistics, to extract information from sets of data in order to find patterns and predict future consequences. These range from expensive professional software solutions like SAS Predictive Analytics or IBM SPSS Statistics, which provide ad-hoc analysis, hypothesis, and model testing (among other features) to add-ins for existing ERP solutions such as SAP Predictive Analysis and even freeware such as R from Revolution Analytics or Orange, an open source analysis tool.
In Predictive Analysis, as in most business cases, one size does not fit all. In fact there are distinctly different approaches depending on the industry involved and the aims companies feel they need to follow. Churn Alert:Many businesses worry about losing customers over time. Bringing in new customers can be expensive; retaining existing customers offers a more affordable solution. Preventing churn by identifying signs of dissatisfaction among customers and identifying those likely to leave is one of the main areas in which Predictive Analysis can benefit companies, for instance in fields such as media, insurance, banking, and telecommunications.

Big Data - Predictive Analytics Chart

Source: © XmPro Inc. 2015,

CLT: Instead of searching for new customers, many companies seek to make existing customers more profitable. This is one of the main areas of focus for Customer Lifetime Value, or CLT. Predictive Analysis can offer marketing departments and top management in fields such as retail, utilities, banking, and insurance new insights that will allow them to target customers that promise the greatest lifetime value.

Product Predilection: Digital marketers are constantly trying to optimize “right offer, right person, right time” through their campaign management solution. So-called “propensity” models offer to improve response and revenue by identifying customers who are “leaning” towards a certain product or service, by analyzing their online behavior in various social media channels

Predictive Maintenance: Enterprises with big investments in infrastructure and equipment such as automotive manufacturers, logistics and transportation companies, or oil and gas suppliers are increasingly demanding the capability to analyze metrics and data that will keep their precious investments up and running at all times. Predictive Analysis enables them to reliably forecast both probable maintenance events and upcoming capital expenditure requirements, reducing maintenance costs and avoiding potential downtime.
Possibly the greatest benefits to be reaped from Predictive Analysis are in up- and cross-selling, where companies need to make smarter and faster decisions about marketing strategy than ever before. Say a shoe store has spent years investing in paid searches, but has only recently begun to explore the possibilities of social media advertising. According to the traditional view of Customer Lifetime Value, the cost for gaining a new customer via social media channels would be prohibitive. But with the help of Predictive Analysis, retailers can determine the true value of an individual customer within days or weeks, thus allowing them to precisely target these customers in ways that were impossible back in the days of paid search. The slightly higher upfront costs can prove to be a bargain in disguise.

Amazon knows what you want to purchase before you even know you want to buy it, and that’s what we’re doing for sales!

Amanda Kahlow
Cofounder and Chief Executive, 6Senese

Big Data - Amanda Kahlow


Forecasting revenue based on historical data is essentially an oldfashioned, backward-looking approach. Much more interesting from a company perspective would be to extrapolate from close observation of new shoppers and combining the results with additional information about the customer mined from a variety of sources, from data brokers to social media platforms, to find out who they are, what channels they prefer to shop through and what demographic group they belong to. This allows smart predictive systems to accurately estimate their probable spending behavior. Hardly any area of business or industry can afford to ignore these and many other possible advantages brought about by Predictive Analysis.

Big Data in the Industry

Cross-selling cable to broadband customers

Denmark‘s leading provider of cable TV and broadband services faces two challenges. First, customer churn is a constant problem. Second, multi-product customers are more profitable and more loyal but many YouSee customers only subscribe to a single product. Using predictive analytics, a call center agent can now make a decision about a cross-sell or retention offer while still on the phone with a customer. YouSee created two models they knew would improve this decision – the likelihood of a successful cross-sell of cable TV services to a broadband subscriber in the next 90 days and the likelihood that a broadband subscriber will churn in the next 90 days.

A bird’s eye view of customer behavior

The home shopping channel based in St. Petersburg, FL, operates affiliate networks all over europe. Its biggest challenge is developing deep relationships with its customers to both attend to and even anticipate their needs without face-to-face interaction. The company‘s electronic and mobile commerce segments have grown at nearly twice the rate of the company’s business as a whole, leading to a tremendous increase in the amount of customer data it needs to handle. Using customer engagement intelligence applications running on in-memory computing provided by SAP, HSe24 is able to target the customer segments that would be most likely to respond to specific campaigns, by querying the data in real-time to detect customer buying patterns. The tool displays the analysis in the form of graphics and charts, making it easy to see and react to data patterns.

Limiting the damage of natural catastrophes

When a city is ravaged by flooding, or an industrial complex is left in ruin by a tornado, Swiss re relies on data to evaluate exactly what kind of damage happened, why it happened, and how to prevent it from happening again. employing powerful analytics on its massive wealth of data, Swiss re uses predictive modeling to educate insurers, cities, and private builders on the possible risks they face when rebuilding after a catastrophe. “We needed a consolidated view of all claims, and our business users needed a way to run their reports in minutes or hours, instead of days“, says reto estermann Head Finance IT General Ledger Systems, Swiss re. Deploying IBmDB2 Analytics Accelerator, he gains actionable insights in hours instead of weeks. The mainframe provides Swiss re a “single source of truth“. Thanks to this system, the reinsurance provider can run queries and generate reports – quickly and efficiently. And the mainframe is a safe, scalable platform for Swiss re’s vital data to reside, moving data in the magnitude of 3 petabytes and housing its biggest database that contains 1.5 billion rows.


No wonder the market for predictive data is booming. Analysts at Gartner predict annual growth of 34 percent by 2017, with revenues projected to reach $48 billion. Venture capitalists have been eager to invest in budding PA startups such as Framed Data (which raised 2 million in seed capital) or 6Sense, which raised $12 million in in Series A equity and debt funding. “Amazon knows what you want to purchase before you even know you want to buy it, and that’s what we’re doing for sales,” says 6Senese’s cofounder and Chief Executive, Amanda Kahlow.

Blue Yonder, a German PA start-up with headquarters in Karlsruhe, gained headlines in 2015 by securing funding of $75 million from the global private equity firm Warburg Pincus, the biggest deal for a predictive analytics company in Europe. CEO Uwe Weiss believes that the need for predictive analytics is independent from traditional economic cycles. “The technology is at the plateau of productivity“, he maintains. “People can use this technology now and produce ROI.”
Blue Wonder’s solution is a cloudbased platform aimed at retail companies and offering them innovative ways of determining pricing and automating merchandise planning. “This means that you can improve turnover, margins and the customer experience, all at the same time,” Uwe believes.
Who would ever dared to predict anything like that?

Big Data Stages of Analytics Blue Yonder

Beyond forecasting Thanks to predictive systems BlueYonder claims it is possible to improve turnover, margins and the customer experience, all at the same time (source: © XmPro Inc. 2015,


Blue Wonder’s solution is a cloudbased platform aimed at retail companies and offering them innovative ways of determining pricing and automating merchandise planning. “This means that you can improve turnover, margins and the customer experience, all at the same time,” Uwe believes. Who would ever dared to predict anything like that?


Comments are closed.