Nucleus Research Deemed Presumptuous

Nucleus Research released another inflammatory research study on ROI failure rates. This time it set its sights on SAP, stating 57 percent of customers fail to receive a positive return on investment--an assertion that SAP representatives say is full of holes. The latest report, "The Real ROI from SAP," comes roughly six months after Nucleus lambasted Siebel Systems in a similar report, stating 61 percent of Siebel customers did not achieve positive ROI. According to 21 SAP customers surveyed, 57 percent, or 12 customers, stated they had not achieved a positive ROI from their SAP deployment, the report suggests. Jim Dever, an SAP spokesman, downplays the report's value and importance. "Nucleus surveyed 21 out of our nearly 20,000 customers," Dever says, suggesting that that's not nearly enough customer participants for the study to be a deemed scientific study. Rebecca Wettemann, vice president of research at the three-year old research company, counters that Nucleus approached 93 companies on SAP's Web site to participate in the study, assuming, Wettemann says, that these companies would be happy SAP customers. This is the same assumption used in the Siebel report, "Assessing the Real ROI From Siebel," which stated: "The assumption was that because Nucleus was examining customers that Siebel itself was promoting as examples of successful deployments, Nucleus would find customers that had received better-than-average returns." In a prepared statement Nucleus wrote: "Despite having used the SAP applications for nearly three years and having spent on average more than $5.4 million on license fees and deployment consulting costs, more than half of these SAP reference customers did not realize a positive ROI." However, when asked if Nucleus inquired about a mutually agreed upon set of metrics defined by all the users involved or affected by the SAP implementation, Wettemann said, "No." This is the biggest bone of contention among CRM consultants --without a predefined set of mutually agreed upon metrics for a CRM implementation, or any large-scale implementation for that matter, failure cannot be accurately measured. To prove how the same CRM implementation can be viewed as both a failure and a success, Barton Goldenberg, president of Information Systems Marketing, cites one client that is implementing a sales and marketing CRM project involving more than 3,000 people. "When asked if it was successful, the marketing people said for marketing purposes the global customer profiles were a raving success. The CIO, however, said the CRM project was a dismal failure, because 84 databases were not properly integrated into one holistic data structure," Goldenberg says. Sue Handman, CRM solutions director at RCG Information Technology, agrees: "Failure is the inability to meet mutually defined expectations. A lot of customers haven't set these expectations. Everyone, including IT, salespeople, customers, the software vendor, and integrators, has to agree on what the expectations are, how the project will be implemented, and over what time frame." (For more on this see CRM magazine's July 2002 cover story, "The Truth About CRM Success & Failure" ). Regarding the Nucleus report, Dever says, "This [report] succeeds in doing just enough research to get PR attention, but not enough to provide any helpful information for the industry. I don't think the study has much merit at all."