8 Rules for Better Predictions

LAS VEGAS — Nate Silver dove headfirst into the world of data analysis -- and used SPSS, an IBM company's offerings -- at a young age. When he was nine years old, Silver and his father sat down during a rainy day while on vacation in Maine to figure out what attracted people to go to Major League Baseball games.

"Oddly enough, your chances of filling a stadium are greater if you have a good team," he quipped to the crowd on day two of SPSS Directions North American Conference, the predictive analytics company's annual user conference.

Silver, the president of FiveThirtyEight, a company that accumulates and analyzes polling and political data, has drawn on his early days of data analysis to successfully predict the winner of 49 of 50 states during the 2008 presidential election. One of Time magazine's "The World's 100 Most Influential People" this year, he shared with the crowd eight principles he believes can help others make better predictions. "Some are fairly obvious, some lower-level, and others are more advanced," he said.

The eight lessons are:

• make sure you really want to know the truth;
• talent is everything;
• quality beats quantity;
• forget Ockham's Razor, use Ockham's Funnel;
• outliers are inevitable;
• visualize when in doubt;
• the 80/20 rule is overrated; and
• embrace the Tao of data.

Kicking off with what he felt is the most important one of them all, the first lesson is simple: Make sure you really want to know the truth. Noting that many companies commission studies to help back up decisions they have already made, Silver urged attendees to have a healthy level of objectivity and be ready to have hypotheses entirely disproved after careful analysis. "Data analysis is art form and a science," he said. "Coming up with an acceptable answer isn't the same as coming up with an accurate one."

That said, when it comes to data analysis, one must have a talent for the job. Silver believes that there are three qualities that every good data analyst should possess:

• statistical aptitude;
• vision; and
• domain knowledge.

"In some ways, [these skills are] more important now than before because of tools like SPSS, you can do things in a few seconds that took literally hours or days or weeks before when done by hand," Silver said. "Technical proficiency is not as important; you have to be more of an artist than a craftsman."

Along with having quality attributes, quality data should most definitely take precedence over quantity. Silver explained to the crowd that poor quality data is dangerous because it is not randomly distributed among the set. As such, it can have drastic affects on the outcomes you generate.

If an analyst has a question about a chart or graph that has been generated from a particular set, Silver said it is best to use our brains which he said have been wired for pattern recognition since ancient times. "Cavemen protected their families ... by figuring out quickly if a rustling they just heard was either leaves nearby or a lion," he said. "They had to make a good prediction because the consequences for being wrong were fairly catastrophic. Most of our value-add comes through our visual ability."

The final lesson he shared, embracing the Tao of data, is truly where analysis as an art as well as a science takes shape. "The yin and yang of data analysis is a mix of discover and decide, and intuit and inform," he said. "You need to have an intuition but also be objective."

Silver believes that the time is ripe for data analysts, granted that they are willing to accept defeat when necessary. "It's a really good time to be a thinker, but I do think it is a creative field, too," he said. "People who are creative and multitalented can talk themselves into things that are wrong. You must have confidence, but be willing to be humble and admit when you're wrong."

News relevant to the customer relationship management industry is posted several times a day on destinationCRM.com, in addition to the news section Insight that appears every month in the pages of CRM magazine. You may leave a public comment regarding this article by clicking on "Comments" at the top; to contact the editors, please email editor@destinationCRM.com.

Related Articles

Did Web Analytics Just Become One of IBM's Core Values?

Big Blue acquires Web analytics firm Coremetrics for an undisclosed sum, hoping to leverage software-as-a-service and social networks.

IBM to Acquire SPSS for $1.2 Billion

Looking to enhance its business analytics capabilities, Big Blue adds the predictive analytics pioneer to its list of more than a hundred companies acquired in the past decade.

SPSS Collects, Collaborates, and Deploys — All in the Name of Analytics

With new versions of PASW Data Collection and PASW Collaboration and Deployment Services, SPSS helps analysts and business users alike manage the voice of the customer.

SPSS Rebrands Its Analytical Offerings

The vendor updates its software for data mining and text analytics, ditching the "Clementine" brand name for "PASW," an acronym for "predictive analytics software."

Statistics For Everyone!

SPSS's latest release of statistics software aims to suit the novice, the expert -- and everyone in between.

SAS and SPSS Hold the Lead in Gartner Magic Quadrant on Data Mining

Magic Quadrant for Data Mining '08: Packaged applications are all the rage -- but as the pair atop the list remains the same, a Visionary emerges overseas.