Nucleus Research, Siebel Spat

Findings issued in a study this week from Nucleus Research have sparked some debate in the CRM space about return on investment. The Wellesley, Mass.-based researcher claims it began the study of Siebel Systems' reference customers to mine best practice information. The reference customers are those featured on Siebel's website, offering testimonials of their positive experience with Siebel and its products. But the study ended up finding that customers touted as happy actually experienced business results that directly contradicted the CRM software giant's marketing claims of a positive ROI in fewer than 10 months. More than 61 percent of the reference customers surveyed did not believe they had achieved a positive ROI from their Siebel investment, citing challenges in training and customization. Seventy-eight percent cited a lack of user-friendliness as a challenge to achieving positive returns, while 65 percent cited rigidity and difficulty in customization as a roadblock to positive ROI, the study says. Siebel executives had originally declined to comment on the survey, instead issuing a verbal statement through its outside public relations people. "No company would take a report that is based on a statistically insignificant survey of 23 companies seriously," the media relations representative says. "In the last year, Siebel has added new customer and maintained a steadfast focus on satisfaction and, according to third party analysts, increased its majority market share. Those are the numbers that count, not a random survey that lacks depth, breath and creditability." According to its website, Siebel claims the average implementation allows customers to achieve a revenue increase of 12 percent, an employee productivity increase of 20 percent, and an increase in customer satisfaction of 20 percent, each of which contribute to positive ROI in less than ten months, the report states. The controversial study began in June with Nucleus contacting the 66 customers featured as testimonials. The original premise of the study was to get information about best practices for achieving a positive ROI from Siebel. Nucleus then intended to use that information to help develop best practice information that would subsequently be sold to its clients. Of the 66 Siebel customers contacted, 23 agreed to participate in in-depth interviews, 12 declined to participate, and 31 never responded to repeated requests for interviews, according to Nucleus. Rebecca Wettemann, vice president of research at Nucleus Research, says that her firm does not consider the study statistically significant or a true sample, but adds that it was not meant to be. "It was just meant as a look at Siebel's best customers," she says. "If anything, we thought that would skew the results to the positive." Those who agreed to participate in telephone interviews conducted by Nucleus analysts were asked questions about factors of their Siebel deployment that would impact ROI, including how, why and when they selected Siebel; how many users are using what functionality; the greatest areas of returns; project spending including software and consulting; customization and deployment strategy; and deployment challenges. "You can't trust the vendor to provide that information," Wettemann says. "Siebel makes some bold claims that aren't true. And they seem to have taken some literary license when it comes to testimonials from Siebel customers. Certainly, when you are looking at reference customers you expect success stories. But the two-third that were willing to talk (to us) were very disappointed." Stacey Wueste, senior director of public relations for Siebel says that all claims and information on the company's website are accurate and that all the testimonials had the blessing and approval of the customer prior to being posted. However, the company has taken down all the testimonials until Siebel can reach all 66 customers and verify that they stand by their original claims and comments. Once that process has been completed the testimonials will be reposted to the site. According to Siebel, for the quarter ended June 30, 2002, the company added 200 new customers, including Key Energy Services Inc., The U.S. Air Force, and The U.S. National Archives and Record Administration. The CRM software developer also expanded existing deals with 250 companies. Overall Siebel has 2,500 customer deployments. Still, Wettemann says that she has not heard from one customer willing to share positive Siebel experiences. "This is a big company that is charging companies a lot of money and not providing value," she said. The Nucleus Research states that Siebel installations cost $18,000 per user per year and that reference customers cited exorbitant license, consulting and maintenance costs as key ROI inhibitors, the study says. According to the report, most of the customers surveyed are using only half of the potential functionality from their Siebel implementation, even after an average implementation time of more than two years. Fifty-seven percent of the Siebel customers interviewed said their deployment took longer than planned, while 55 percent said they spent more than originally budgeted. The report says that some customers have opted for a phased deployment strategy to get users comfortable with the solution, but finds that this approach significantly slows the realization of benefits while increasing and extending costs of consulting, training and personnel. Wettemann says another surprise was that most of those interviewed for the study had to increase their consulting and personnel investment over time to support their Siebel solution. That is the opposite of previous CRM findings that typically show high costs during the first year and progressively lower costs over time. And while all those issues combined can make for difficult times for customers, Wettemann says a lot can be forgiven if customers feel that vendors are responsive and helping them. However, the information Nucleus says it received from the customers about their relationships with Siebel show that customers felt the company was not responsive to their concerns. "Some of these customers are frustrated by the lack of ROI, the feeling that they were oversold, and the lack of support from Siebel to correct this," Wettemann says. Siebel's Wueste countered, saying that Siebel conducts its own surveys every quarter and has found that of the 879 customers interviewed just 65 percent even measured ROI. Siebel's most recent survey conducted in August found that of those measuring ROI, the median increase in customer satisfaction was 23 percent, median revenue increase was 8 percent, median increase in customer retention was 8 percent and the median decrease in operational costs was 13 percent. "We feel that is statistically relevant data," Wueste says. "We use these surveys as part of our day-to-day business and as a management tool."