QUESTION: It disturbs me that analysts don’t have hands-on experience with the products they advise their clients about. How are can they be credible?
ANSWER: Analysts, especially at the larger end-user centric firms like Gartner and Forrester, have access to what they would claim is the world’s largest testing lab – their clients. Analysts covering hot product areas, both software and hardware, are hearing from scores of clients every month who are evaluating products or using them in production. As a consequence, the analyst firms claim that they can develop a series of data points that quite accurately portray the quality of a vendor’s products.
Note: This is a classic SageCircle article from our pre-blogging newsletter. This particular article appeared in March 2001. It is being brought into the blogging era because ZL Technologies in its lawsuit against Gartner criticizes Gartner because it does not do “a single minute of independent testing of the products it purports to evaluate” (page 9 of the original court filing).
- Vendors can help themselves by ensuring that their product evaluation programs for prospects are top notch. This prevents prospects from getting frustrated during evaluations and pilots and passing this frustration on to their favorite analysts.
- Vendors should arrange for reference calls between analysts and their happy customers. The analysts are already hearing from disgruntled users of products, so vendors need to proactively balance this view with satisfied customers.
Bottom Line: Hands on testing, especially of large enterprise-grade hardware and software, has never been part of the advisory analyst business model and research methodology. Knowing that advisory analysts rely on hearing about the experiences with products and service that their end-user clients have should spur vendors into making sure that analysts are provided with a high volume of customer case studies, informal customer stories, customer references, and calls from happy customers.
Question: AR – Do you have a formal customer reference program?