I had a chance to talk with Brett Waldman (Analyst) and Michael Shirer (Director of Communications) at IDC about the recent press release from IDC about virtualization market share that I had reacted so strongly against. I’m going to paste my general and short notes from the call below and then comment on them after that. Hopefully this will close the chapter on the IDC study.
- Not a sponsored product
- Sponsored reports are one-offs
- Subscription is an on-going thing (quarterly, etc)
- Special Study also exists – not subscription
- Virtualization Tracker was originally a sponsored report
- IDC never puts out a press release for sponsored reports
- Sponsored reports are supposed to say sponsored
- “Virtualization in the Datacenter” was a sponsored report (this was the predecessor to tracker) –> Q407 data
- Measures usage in production
- Doesn’t measure installations
- They specifically ask if you’re using it in production
- It does count Hyper-V beta if they respond it’s in production
- Microsoft gave them a list of TAP customers to go and talk to
- They do not ask about VMware beta (VMware didn’t have a beta going on in Q2)
- semi-annual survey
- Are you using virtualization?
- Which platform?
- What have you installed in the last quarter?
- Determined by new shipments
- Derivative of server tracker
- See note below about upper boundItems Used to Triangulate Number:
- Financials
- Interviews with companies (VMware, MS, etc)
- Competitive information
- Surveys
- Upper bound is new server shipments
- They say that new virt licenses couldn’t possibly exceed new server shipments.On the Survey…
- No idea where the customer list came from
- Used a 3rd party for the survey
- September 2008 survey to ask about Q2 data
- They use this information to get Q3 data as well even though Q3 isn’t over with
- They’ll do the survey again in March 2009 to get Q4 and Q1 data
- They do the surveys twice a year and produce the tracker quarterly
So what does all of this mean to me? In a nutshell it means the numbers are more screwy than I originally reported. The big thing that I have been talking about previously is the need for transparency in these numbers from IDC and Gartner and anyone else that wants to put up a report. I ask for the same thing in the benchmarking world. I always want to see the data to backup claims. During the call I got a little more of that transparency. It looks like the numbers were backed into using 4 things – financials, interviews with companies, competitive information, and a survey.
Financials
My first bone to pick is using financials for any way of calculating units shipped. IDC admits that the numbers take into account free products such as VMware Server and Microsoft Virtual Server. How do you get any clue as to units shipped for these products from financial reports? Even using the reports to get the paid-for side of the equation is skewed. VMware public financial data doesn’t break out actual unit shipments. Now you may decide to take $5,750 (list price for VMware’s most expensive Virtual Infrastrcuture 3 SKU) and divide that into the total revenue for a quarter to come up with a theoretical unit shipment. That would be grossly inaccurate though since VMware sells more than just the VI3 Enterprise SKU and more products than just Virtual Infrastructure.
On the Microsoft side there’s no way to even begin to figure out unit shipment from revenue. From the IDC Link that went out on November 3, 2008 we get a little deeper glimpse into how Hyper-V was calculated:
It is interesting to put some context around the market potential for Hyper-V in terms of Windows Server 2008 shipments. IDC currently projects that Windows Server 2008 will ship 526,000 units during calendar year 2008. Looking at this data on a quarterly basis and taking into account that this is a new product with an initial ramp-up period, and allowing for only 50% of that total to be 64-bit deployments (which include Hyper-V), a conservative estimate would be that there were about 70,000 shipments of Windows Server 2008/64-bit from February until June. This is the potential pool where Hyper-V could be used in conjunction with Windows Server 2008. The stand-alone Hyper-V server product, which does not require Windows Server 2008 at all, would represent additional potential for additional Microsoft Hyper-V footprints. The totals calculated by IDC for Hyper-V in Q2 CY2008 are but a small fraction of this total available pool.
So basically they’re basing Hyper-V share from Windows Server 2008 uptake. The funny thing is that Hyper-V isn’t included in Windows Server 2008. Hyper-V (even the RTM) needs to be downloaded and installed separately even to this date. There’s no telling how many people are doing that download or installing Hyper-V. IDC is basically using the same scientific method as figuring out how many jelly beans there are in a Cadillac – guessing.
Interviews with Companies
Without good numbers from the financials we turn our attention to interviews with the companies. Oh, wait, they didn’t interview the companies – at least they didn’t interview VMware. I’m not sure about the others. During the phone conversation, Brett mentioned that they were at VMworld and they talked to people there. When asked who they couldn’t list a VMware employee or any particular customer. Not sure if they didn’t want to reveal sources or what. I do know for a fact that anyone that really knew exactly how many unit shipments VMware had wouldn’t be talking about it with anyone due to SEC regulations. They could have talked with a sales rep to ask how the quarter was going but that doesn’t give a true market view. They could have talked to Microsoft or other competitors directly but that only further skews the numbers. Basically, this view is also a little warped if there’s even a view here at all.
Competitive Information
This brings me to my favorite point – competitive information. I tried to understand exactly what this was. The way it was explained to me is IDC “knows people”. Hmmm…I know people to and they agree with me that IDC is whack in this report. I guess they could have asked the actual competitors as part of the “interviews with companies” but how accurate do you think that will be? I mean if you ask me how much Hyper-V I’m seeing installed out there I’m not going to shout out that it’s everywhere (which it isn’t) just like Microsoft isn’t going to say they’re seeing VMware everywhere. I’m still trying to figure out how you get any good points out of competitive information. For now this one is in the “we need more transparency” bucket.
The Survey
Last up is the infamous survey of 2,500 people from 35 countries. I’m always skeptical of surveys but let’s just look at these numbers for a little bit. I know you can get good results from a small sample rate (2,500 customers is less than 2% of VMware’s stated pay-for customer base). What we really need to know is who did you ask (VP, techie, CTO, CFO)? What segment were they in (SMB, Enterprise, Global, etc)? What vertical were they in (hosting, tech, financials, etc)? NOTE: There was a surprisingly large showing for Parallels here so there were definitely quite a few hosters interviewed. Without visibility into who participated in the survey all we can do is just give up and trust or not trust the results. IDC doesn’t know who they got to answer either since a 3rd party was contracted to do the survey and the 3rd party came up with the customer list in the blind. I can tell you that this has about the same accuracy as me posting a survey out there. Actually, we’re in luck because Virtualization.info and Techrepublic are both doing surveys in the blind right now as well. The Virtualization.info results aren’t compiled yet. The Techrepublic results as of November 4, 2008 show 111 responses and the results actually match pretty closely with Gartner’s numbers – not IDCs. I’m a betting man and I’ll bet Virtualization.info’s numbers come out about the same as Gartner and Techrepublic. So, if we’re to trust the survey, IDC, then we need to see more transparency. You know, I did a poll with a bunch of people at work and they said IDC was accurate only 23% of the time. See my point?
Bottom Line
The bottom line is I’m tired of looking into this IDC report. The more I pull back the covers the more I see some horrible math from a company that’s supposed to have their act together on numbers. Maybe with even more transparency I’ll be convinced. So far, all of the comments and other reports and surveys out there all say the same thing – IDC is just flat wrong on this one. Time to go back to the drawing board on this report guys. Better luck next time.
-
Alberto
-
Johan
