How to use analyst market share numbers after Gartner makes a “huge mistake” with server market share numbers

Rarely do analysts call out another firm on perceived failures in research, but Rob Enderle does just that in Liars, Damn Liars and Statistics: Gartner Goofs on Server Numbers. Money quote:

“…However, the accuracy of these numbers even inside corporations (given how deals are accounted for) would suggest that getting within 5 percent of actual sales would be very difficult, let alone having a high level of confidence that under 1 percent actually signified real market leadership. …”

Rob then goes into an interesting discussion of the shortcomings of market share numbers and the methodologies used to create them. The article is well worth reading. It would be interesting – fun even – if more analysts engaged each other in the marketplace of ideas rather than having a monologue with clients.

SageCircle has long said that market share numbers from the market research analysts can provide interesting insights into the direction a market is going. However, relying on the numbers alone without understanding how they are created or the assumptions that went into their research can lead to the wrong conclusions. To address this problem we wrote SageNote^TM AR81 “A Consumers’ Guide to Market Share Numbers.” You can get a copy of this SageNote by contacting info [at] SageCircle dot com or 650-274-8309. The content has also been incorporated into the Online SageContent^TM Library, which is available to subscribers.

SageCircle Technique: Users of analyst market share numbers should use the following questions with analysts in order to be good consumers of the research:

• What is your market taxonomy with supporting definitions?
• How did you verify the vendor-supplied numbers?
• Do you footnote whether the numbers came from an audited or unaudited source?
• Do you footnote differences in the definitions of numbers provided by the vendors?
• How do you handle a situation when a vendor does not supply numbers?
• Who collects the data, a senior analyst or a research operations staff?
• If research operations, how was the staff trained?

Bottom Line: Users of market share numbers should strive to be informed consumers of analyst research. The best results are obtained by the having a relationship with the market researcher and using that relationship to obtain insights into the quality of the numbers. Consumers should be aggressive in challenging analyst positions and asking for background information on how they developed their numbers.

Question: Market research users – Do you rigorously challenge the analysts about their numbers and methodology?