Scepticism is healthy when market research is considered. It's really important to know what's being counted, how it's being counted, how it was analyized, who funded the research, etc.
My success (or lack of it for that matter), is not at all gaged by what's said in the media. It's gaged by what subscribers say and what they purchase. Media coverage only is tangentially related to subscribers' interest. Helping them make more money, save money, or in some way making their lives better has a much stronger connection with success in this business.
As an aside, it's often true that a journalist calls me up out of the blue, partially listens to what I have to say on a topic, edits it down to fit the space available, and then presents it as if it were exactly what I said. I have to be willing to go on record even though that opinion, as printed, may look silly or flat out wrong a day or so later. It's part of the job. I just live with it.
IDC does its best to be as accurate as possible and all opinions that it publishes are based upon research its conducted. If better data becomes available, we will respond to that change and publish something to update the previous position. Since the net never forgets, the old comment often comes up again and again even though it has been replaced with a new comment when better data becomes available.
I believe that it was Einstein who said "Not everything that counts has been counted, and not everything that has been counted counts."
Please continue to question analysts, consultants, journalists, and anyone else in the public spotlight. In the end, you may agree with them and find their data and opinions useful. If you don't, simply ignore them and go on with life!
The reports IDC subscribers see includes revenues, revenue shipments, shipment installed base for revenue producing products, and an estimate of free downloads or free replicas made of either paid shipments or downloads. Only the revenue shipment data is released to journalists.
IDC's operating environment research starts with SEC financial data on the suppliers. The software revenues (that's software licenses plus software maintenance) are segmented into over 100 types of software, 9 operating environment categories, and then into 6 geograpical regions. The operating system and subsystem revenue category is then used to model shipments. The modeling is based upon the software revenue and survey data on user population for specific products. The modeling is reviewed by representatives of the suppliers.
Once each supplier's revenues and shipments have been tabulated, the numbers are put together into market models. Each operating environment's total revenues and shipments are compared to the total. That's the market share you see published.
IDC is following client operating environments, server operating environments, host operating environments, and embedded operating environments.
No, we use the published finanacial data as the beginning of a process. After segmenting the data into over 100 different categories of software, 9 operating environments and 6 different geograpic regions, we model the shipments based upon survey research.
The modeling of the financal data is reviewed by the suppliers. We listen to their input. We also listen to the input of many other reviewers. After weighing all of the input, we publish something which is supportable based upon all of the research.
Actually, the reports include several tables. One shows shipments for three years. Another shows installed base. The installed base tables typically are only made available to subscribers not to journalists.
My team is made up of fairly senior people. The system software research team is not made up of recent college graduates. Each analyst has quite a number of years in the industry in product management, product market, or IT management.
I beg to differ with you on this. IDC's software research team obtains software revenue information from software supplier's SEC forms or from public comments, executive interviews, and the like of privately held firms. This data is segmented into over 100 different types of software, 9 operating environments and 6 geographical regions. This segmentation is then given to representatives of the companies in question for comment and review.
IDC's numbers are the result of research. They are not produced out of thin air.
No. That would not be an accurate rendition of my comment.
IDC conducts many types of research. A great deal of this research is published use of subscribers of one or more of IDC's 300+ "continous information service" subscribers. It is correct to point out that the subscription fees pay for the research. No subscriber, however, controls the topics researched, the methodology, or the findings beyond the choice to subscribe or not to subscribe.
A small portion of the findings may (or may not) be released in a press release, in a media interview, or in a presentation at an event.
The press releases, interviews, presentations, etc. all serve to publicize IDC's publications and services. In some cases, this encourages individuals or companies to purchase the document or become a subscriber.
Yes, you are right. The resulting revenue allows the company to stay in business and continue conducting research for people.
While conducting interviews for surveys or meeting with clients at their offices, I often ask "How much Linux do you have installed?" or some similar question. Executives often tell me that they have absolutely no Linux running. Later when talking with folks in various business offices, I ask the same question. At times, I'm told by these people that there are quite a few systems running Linux (or BSD) which are supporting Web-based applications, file/print services for a workgroup, or messaging. Software development is another popular use.
When I ask how it is possible for so much Linux (or BSD) to be installed without executives being aware of it, I'm often told something along the lines that within the organization it is much better to get congratulated for doing a great job and solving a problem than to listen to criticism about what tools were chosen to solve the problem.
I believe that a portion of Linux (and BSD) usage is hidden. This means that traditional surveys of IT decision-makers won't find it. Interviews of computer users in business units might turn up clues to its presence.
Your answer is right as far as it goes. IDC's subscriber list includes hardware and software suppiers, suppliers of various types of services, large end-user organizations, investment institutions, and even governmental organizations.
The broad needs they present is the reason that IDC so many analysts looking at so many things.
My original post was written my usual office, you know, seat 16E on Delta flight 1426. I have a tendency to be somewhat terse when trying to type while folded into a pretzel shape.
It's become quite clear to me that being a contortionist is part of the job of an industry analyst. Avoiding the mid-flight, display crunch while typing is another.
I did my best to reply where I thought the reply would add some value to the discussion while taking part in a highly distributed strategic planning meeting. Meeting participants were in many countries.
I think that people should be skeptical about market research and find out more. I agree that people would be best served if they understand what's being counted, how it's being counted, what's included and what's being excluded.
That's why I always begin a presentation with a joke about "lies, damn lies and statistics" or "figures lie and liars figure."
In my 7 years with IDC, the subscribers I enjoyed the most were those who asked those types of questions. We then explored together what was behind the methodology, the data colection, the analysis, and how this information could be of use to the company they represented.
I remember fondly my discussions with the CEO of one company. We'd meet just about every year at that CEO's user group meeting. We'd debate for about 45 minutes, smiling the whole time. In the end, we'd shake hands and promise to do it again next year.
I think that I learned more from that CEO that he learned from me.
That's IDC's estimate of the number of copies of BSD which were sold as packaged software. This, of course, does not include the number of BSD software which was downloaded, no-cost replicas made of the software, or the number of copies of BSD which were hidden inside of an appliance server.
As others have noted, only a small percentage of BSD is packaged software.
BSD shows up in IDC's operating environment surveys but, this data is not used to create supply side reports. Only revenue data and the resulting paid shipments are mentioned.
IDC also publishes customer profiles, vendor profiles, surveys and other types of reports. IDC subscribers see these documents.
If a person paid to download the product, it would produce revenue and the revenue shipment would be counted.
The only way to track free shipments and replicas made of either paid or free shipments is to do a survey.
Since there is no way to independently verify the numbers provided by the mirror sites, just calling them up and reporting those numbers isn't enough.
IDC has done enough operating environments surveys in the last 5 years that we finally have enough data to project free shipments and replicas made of either free or paid shipments. This data is reported in the report, "Linux Software Market Forecast and Analysis, 2001-2005".
I believe that I've said this several ways but, I guess, I haven't been clear enough. Let me try again. Since market share information is based upon supply-side research, not a survey, a survey instrument wasn't needed or used. So, there are no questions to share with the Slashdot community.
Revenue data and the shipment data which is modeled based upon revenue data comes from published financial reports for public companies and public statements and private interviews of executives of privately held companies. The methodology used to go from revenue data to shipment data has been discussed earlier in this string.
If you'll look again, you'll see that I didn't say anything about the number of servers running BSD in my reply. I said that over 8,000 copies of BSD software sold.
As others have noted, the free downloads and replicas made of either free downloads or paid copies are not included in the total.
IDC publishes installed base information to its subscribers but does not provide that type of information to the media.
I'm sorry that I appeared to be providing "fluffy" answers to your questions. Let me try a more direct approach.
IDC's software research group has an editorial review board. All forecasts are reviewed before they are published. Since IDC's name and reputation are on the line, every attempt is made to make sure that the assumptions are reasonable and that they have been applied in a supportable way. They still, by the way, may prove to be wrong in interesting and highly public ways.
IDC analysts review previous year's forecasts each year when the report is published and discuss whether or not they were accurate. If they were not, the analyst tries to point out how market trends differed from those projected in the previous report. This is an attempt to help the subscriber gain some value from what the analyst learned from thet time the first forecast was published.
The bottom line, however, is the review provided by subscribers. If they renew their subscription, they found value in the program. If they don't, then they didn't.
I believe that was answered in my replies to the questions sent to me. Let me repeat it here.
IDC software anaylsts stationed in every major geographic area obtain software revenue data from every major companies in the software market. Companies having over $10 million dollars/year and up are included This is done on a quarterly basis and includes data from both public and private companies.
IDC segments that revenue into many (102)different software markets using a well defined taxonomy. The segmentation is then reviewed by company representaives to make sure the segmentation is as accurate as possible.
Operating system shipments are modeled using operating system snd subsystem (one of IDC's taxonomic categories) revenue data and survey data on the average shipment value. This model is also reviewed by company representatives.
Reports are then written which review how the revenues and shipments in each operating environment segment stack up.
So, the revenues generated by Windows, UNIX, Linux, NetWare, etc. are compared in one table. Shipments are compared in a different table. The data is segmented several ways including adoption in each of 6 geographic regions, client vs. server useage, etc.
Subscribers to the research services for which my team and I write see the data, see the assumptions used in the analysis, see a description of the methodology, and then see the ressults of the analysis. No one is expected simply to take the results and accept them as THE TRUTH.
On more than one ocassion I've been told by a subscriber that they think that my assumptions and analysis are useless to them. They purchase the reseach just for the raw data and then do their own analysis.
One even told me that they consider me a reverse bell weather. If I say suggest one thing in a report, they do the opposite!
My goal is to help subscribers by providing solid information for their planning processes not to be popular or to see my opinion win out.
Thanks for the kind words. I'm a partially reformed software engineer. In earlier lifetimes, I worked on operating systems, database and networking software. I worked for a hardware supplier for 15 years and was responsible for everything from custom development of applications for customers, product marketing, product management and business management.
IDC has been conducting research on the market for information technology products for over 20 years. As open source operating systems started to appear, they were included in the research. IDC used the same methododology it has used for Windows, UNIX, NetWare, Mac OS, OS/400, OS/390, etc. so that the same things would be compared in the same way. The report, "Linux Software Market Forecast and Analysis, 2001-2005", includes an estimate for free shipments. This estimate was developed using several large scale surveys.
IDC screens the lists of potential survey respondents very carefully. Every reasonable effort is made to make sure that the surveys or focus groups include the appropriate people in the study. Since the studies usually include hundreds of respondents and in some cases thousands of respondents, the impact of a single person lying is not likely to be very large.
Slashdot Mirror
Here's a pointer to my bio on IDC.com
0 00 199
http://www.idc.com/getdoc.jhtml?containerId=PRF
Scepticism is healthy when market research is considered. It's really important to know what's being counted, how it's being counted, how it was analyized, who funded the research, etc.
My success (or lack of it for that matter), is not at all gaged by what's said in the media. It's gaged by what subscribers say and what they purchase. Media coverage only is tangentially related to subscribers' interest. Helping them make more money, save money, or in some way making their lives better has a much stronger connection with success in this business.
As an aside, it's often true that a journalist calls me up out of the blue, partially listens to what I have to say on a topic, edits it down to fit the space available, and then presents it as if it were exactly what I said. I have to be willing to go on record even though that opinion, as printed, may look silly or flat out wrong a day or so later. It's part of the job. I just live with it.
IDC does its best to be as accurate as possible and all opinions that it publishes are based upon research its conducted. If better data becomes available, we will respond to that change and publish something to update the previous position. Since the net never forgets, the old comment often comes up again and again even though it has been replaced with a new comment when better data becomes available.
I believe that it was Einstein who said "Not everything that counts has been counted, and not everything that has been counted counts."
Please continue to question analysts, consultants, journalists, and anyone else in the public spotlight. In the end, you may agree with them and find their data and opinions useful. If you don't, simply ignore them and go on with life!
The reports IDC subscribers see includes revenues, revenue shipments, shipment installed base for revenue producing products, and an estimate of free downloads or free replicas made of either paid shipments or downloads. Only the revenue shipment data is released to journalists.
IDC's operating environment research starts with SEC financial data on the suppliers. The software revenues (that's software licenses plus software maintenance) are segmented into over 100 types of software, 9 operating environment categories, and then into 6 geograpical regions. The operating system and subsystem revenue category is then used to model shipments. The modeling is based upon the software revenue and survey data on user population for specific products. The modeling is reviewed by representatives of the suppliers.
Once each supplier's revenues and shipments have been tabulated, the numbers are put together into market models. Each operating environment's total revenues and shipments are compared to the total. That's the market share you see published.
IDC is following client operating environments, server operating environments, host operating environments, and embedded operating environments.
No, we use the published finanacial data as the beginning of a process. After segmenting the data into over 100 different categories of software, 9 operating environments and 6 different geograpic regions, we model the shipments based upon survey research.
The modeling of the financal data is reviewed by the suppliers. We listen to their input. We also listen to the input of many other reviewers. After weighing all of the input, we publish something which is supportable based upon all of the research.
We tend to examine supplier input very carefully.
Actually, the reports include several tables. One shows shipments for three years. Another shows installed base. The installed base tables typically are only made available to subscribers not to journalists.
I've pointed out repeatedly that the numbers IDC publishes are based upon either supply or demand side research and are not plucked out of the air.
Another point is that what you see in print may only have a small relationship to what I actually said.
My team is made up of fairly senior people. The system software research team is not made up of recent college graduates. Each analyst has quite a number of years in the industry in product management, product market, or IT management.
Dan Kusnetzky
I beg to differ with you on this. IDC's software research team obtains software revenue information from software supplier's SEC forms or from public comments, executive interviews, and the like of privately held firms. This data is segmented into over 100 different types of software, 9 operating environments and 6 geographical regions. This segmentation is then given to representatives of the companies in question for comment and review.
IDC's numbers are the result of research. They are not produced out of thin air.
Dan Kusnetzky
No. That would not be an accurate rendition of my comment.
IDC conducts many types of research. A great deal of this research is published use of subscribers of one or more of IDC's 300+ "continous information service" subscribers. It is correct to point out that the subscription fees pay for the research. No subscriber, however, controls the topics researched, the methodology, or the findings beyond the choice to subscribe or not to subscribe.
A small portion of the findings may (or may not) be released in a press release, in a media interview, or in a presentation at an event.
The press releases, interviews, presentations, etc. all serve to publicize IDC's publications and services. In some cases, this encourages individuals or companies to purchase the document or become a subscriber.
Yes, you are right. The resulting revenue allows the company to stay in business and continue conducting research for people.
While conducting interviews for surveys or meeting with clients at their offices, I often ask "How much Linux do you have installed?" or some similar question. Executives often tell me that they have absolutely no Linux running. Later when talking with folks in various business offices, I ask the same question. At times, I'm told by these people that there are quite a few systems running Linux (or BSD) which are supporting Web-based applications, file/print services for a workgroup, or messaging. Software development is another popular use.
When I ask how it is possible for so much Linux (or BSD) to be installed without executives being aware of it, I'm often told something along the lines that within the organization it is much better to get congratulated for doing a great job and solving a problem than to listen to criticism about what tools were chosen to solve the problem.
I believe that a portion of Linux (and BSD) usage is hidden. This means that traditional surveys of IT decision-makers won't find it. Interviews of computer users in business units might turn up clues to its presence.
Your answer is right as far as it goes. IDC's subscriber list includes hardware and software suppiers, suppliers of various types of services, large end-user organizations, investment institutions, and even governmental organizations.
The broad needs they present is the reason that IDC so many analysts looking at so many things.
My original post was written my usual office, you know, seat 16E on Delta flight 1426. I have a tendency to be somewhat terse when trying to type while folded into a pretzel shape.
It's become quite clear to me that being a contortionist is part of the job of an industry analyst. Avoiding the mid-flight, display crunch while typing is another.
I did my best to reply where I thought the reply would add some value to the discussion while taking part in a highly distributed strategic planning meeting. Meeting participants were in many countries.
Thanks for the comments.
I think that people should be skeptical about market research and find out more. I agree that people would be best served if they understand what's being counted, how it's being counted, what's included and what's being excluded.
That's why I always begin a presentation with a joke about "lies, damn lies and statistics" or "figures lie and liars figure."
In my 7 years with IDC, the subscribers I enjoyed the most were those who asked those types of questions. We then explored together what was behind the methodology, the data colection, the analysis, and how this information could be of use to the company they represented.
I remember fondly my discussions with the CEO of one company. We'd meet just about every year at that CEO's user group meeting. We'd debate for about 45 minutes, smiling the whole time. In the end, we'd shake hands and promise to do it again next year.
I think that I learned more from that CEO that he learned from me.
That's IDC's estimate of the number of copies of BSD which were sold as packaged software. This, of course, does not include the number of BSD software which was downloaded, no-cost replicas made of the software, or the number of copies of BSD which were hidden inside of an appliance server.
As others have noted, only a small percentage of BSD is packaged software.
BSD shows up in IDC's operating environment surveys but, this data is not used to create supply side reports. Only revenue data and the resulting paid shipments are mentioned.
IDC also publishes customer profiles, vendor profiles, surveys and other types of reports. IDC subscribers see these documents.
If a person paid to download the product, it would produce revenue and the revenue shipment would be counted.
The only way to track free shipments and replicas made of either paid or free shipments is to do a survey.
Since there is no way to independently verify the numbers provided by the mirror sites, just calling them up and reporting those numbers isn't enough.
IDC has done enough operating environments surveys in the last 5 years that we finally have enough data to project free shipments and replicas made of either free or paid shipments. This data is reported in the report, "Linux Software Market Forecast and Analysis, 2001-2005".
Boring title to be sure.
I believe that I've said this several ways but, I guess, I haven't been clear enough. Let me try again. Since market share information is based upon supply-side research, not a survey, a survey instrument wasn't needed or used. So, there are no questions to share with the Slashdot community.
Revenue data and the shipment data which is modeled based upon revenue data comes from published financial reports for public companies and public statements and private interviews of executives of privately held companies. The methodology used to go from revenue data to shipment data has been discussed earlier in this string.
If you'll look again, you'll see that I didn't say anything about the number of servers running BSD in my reply. I said that over 8,000 copies of BSD software sold.
As others have noted, the free downloads and replicas made of either free downloads or paid copies are not included in the total.
IDC publishes installed base information to its subscribers but does not provide that type of information to the media.
I'm sorry that I appeared to be providing "fluffy" answers to your questions. Let me try a more direct approach.
IDC's software research group has an editorial review board. All forecasts are reviewed before they are published. Since IDC's name and reputation are on the line, every attempt is made to make sure that the assumptions are reasonable and that they have been applied in a supportable way. They still, by the way, may prove to be wrong in interesting and highly public ways.
IDC analysts review previous year's forecasts each year when the report is published and discuss whether or not they were accurate. If they were not, the analyst tries to point out how market trends differed from those projected in the previous report. This is an attempt to help the subscriber gain some value from what the analyst learned from thet time the first forecast was published.
The bottom line, however, is the review provided by subscribers. If they renew their subscription, they found value in the program. If they don't, then they didn't.
I believe that was answered in my replies to the questions sent to me. Let me repeat it here.
IDC software anaylsts stationed in every major geographic area obtain software revenue data from every major companies in the software market. Companies having over $10 million dollars/year and up are included This is done on a quarterly basis and includes data from both public and private companies.
IDC segments that revenue into many (102)different software markets using a well defined taxonomy. The segmentation is then reviewed by company representaives to make sure the segmentation is as accurate as possible.
Operating system shipments are modeled using operating system snd subsystem (one of IDC's taxonomic categories) revenue data and survey data on the average shipment value. This model is also reviewed by company representatives.
Reports are then written which review how the revenues and shipments in each operating environment segment stack up.
So, the revenues generated by Windows, UNIX, Linux, NetWare, etc. are compared in one table. Shipments are compared in a different table. The data is segmented several ways including adoption in each of 6 geographic regions, client vs. server useage, etc.
Subscribers to the research services for which my team and I write see the data, see the assumptions used in the analysis, see a description of the methodology, and then see the ressults of the analysis. No one is expected simply to take the results and accept them as THE TRUTH.
On more than one ocassion I've been told by a subscriber that they think that my assumptions and analysis are useless to them. They purchase the reseach just for the raw data and then do their own analysis.
One even told me that they consider me a reverse bell weather. If I say suggest one thing in a report, they do the opposite!
My goal is to help subscribers by providing solid information for their planning processes not to be popular or to see my opinion win out.
You left one out that is equally important.
"Not everything that counts can be counted and not everything that can be counted counts." Albert Einstein
Thanks for the kind words. I'm a partially reformed software engineer. In earlier lifetimes, I worked on operating systems, database and networking software. I worked for a hardware supplier for 15 years and was responsible for everything from custom development of applications for customers, product marketing, product management and business management.
IDC has been conducting research on the market for information technology products for over 20 years. As open source operating systems started to appear, they were included in the research. IDC used the same methododology it has used for Windows, UNIX, NetWare, Mac OS, OS/400, OS/390, etc. so that the same things would be compared in the same way. The report, "Linux Software Market Forecast and Analysis, 2001-2005", includes an estimate for free shipments. This estimate was developed using several large scale surveys.
IDC screens the lists of potential survey respondents very carefully. Every reasonable effort is made to make sure that the surveys or focus groups include the appropriate people in the study. Since the studies usually include hundreds of respondents and in some cases thousands of respondents, the impact of a single person lying is not likely to be very large.