In March, 2009, the Boston Consulting Group released a report titled “The Innovation Imperative in Manufacturing“. BCG co-wrote the report with the National Association of Manufacturers and the Manufacturing Institute of the US.) The goal was to assess the level of innovation among US firms in the area of manufacturing, both across the different states of the US as well as in comparison to other countries. The strength of this report is its rich dataset, which combines a survey of over 1000 companies with hour-long interviews with 30 senior executives.
The portion of the report that compares across different states of the US is quite interesting. Unsurprisingly, it shows states like California, Connecticut, Massachusetts and New York leading in both innovation inputs and innovation performance, while other states like Alaska, Florida and Maine lag on both dimensions (see page 13 of the report). There are also states with high levels of innovation inputs but low innovation performance, and vice-versa. The most useful parts of the report are Section 4 (What Drives Innovation Success?) and Section 5 (The Role of Government). Both these sections contain interesting snippets from interviews with executives, presenting a clear description of what they view as the issues to be addressed by firms and governments.
In contrast to the within-US comparison, the international comparison raises a greater number of concerns. At the heart of the analysis is a newly devised ranking system which attempts to measure the “innovation friendliness” of 110 countries and the 50 US states. Based on the rankings, the authors conclude that the US is “falling short in its commitment to innovation and in its innovation performance”. The US appears at number eight in these rankings, ahead of Japan, Canada and the European countries, but behind Singapore (#1), South Korea (#2), Switzerland (#3), Iceland (#4), Ireland (#5), Hong Kong (#6), and Finland (#7).
There are two main issues with this ranking system, and they have to do with measurement. A good measure should have high reliability as well as construct validity (see Judd et al., Research Methods in Social Relations, Chapter 3. ISBN 978-0030311499). Reliability refers to the extent to which a measure is free from error, such as when it is measured again and independently produces the same result, or when multiple questions are asked about the same item being measured and yield highly correlated responses. Validity concerns whether the measure really captures what is claimed to be measured. For example, if a group of students were told to take a mathematics test in a foreign language they lacked fluency in, the test scores may not be valid measures of the students’ mathematical abilities.
Herein lies the problem with the BCG report: exactly how “innovation friendliness” was measured is not clearly described. As such it is difficult to ascertain whether the measure is reliable and/or valid, and it is upon this measure that their key conclusions are drawn. It would have been more reassuring to the reader had they included an Appendix describing in a little detail how “innovation friendliness” was measured. What were the questions asked in the surveys and interviews? How were the many different dimensions of innovation (Exhibit 1 on page 9 of the report) combined into a single-dimensional scale of “friendliness”? How were the results of these individual-level surveys aggregated to form a country-level index? Aggregation issues might explain why the top positions in the ranking are dominated by small countries, including Singapore, Switzerland, Iceland, Ireland and Hong Kong. Is it reasonable to compare a tiny city-state like Singapore to a country like the US? Perhaps it might be more useful to compare manufacturing innovation in Singapore to that within a particular metropolitan area (e.g., the San Francisco Bay Area), rather than to the whole of the United States.
The issues of reliability and validity are especially important because this report is based on interviews and survey data. It would be difficult for an independent third party to validate those measures, unlike in reports that utilize publicly available data. An example of an index based on public data is the IPRIA Innovation Index, which uses data on patents from the US Patent Office, R&D expenditure from the OECD, and trade data from the World Bank (http://www.ipria.org/publications/reports.html). We do not imply that the IPRIA report is better; every approach has tradeoffs, and while the IPRIA report may exhibit greater transparency, the BCG report offers analysis based on a richer set of data than would be available publicly.
Another issue concerning the BCG report is that it does not explore in sufficient depth the observed differences in “innovation friendliness” across country. Firms, especially multinational ones, choose to locate different manufacturing activities in different geographic areas due to a variety of reasons. Hence, one would expect a wide range responses within each country depending upon firm-level needs, the competitive environment a firm operates in, intellectual property protection, and the type of innovative activity it engages in. It is unclear from the report whether the differences across countries are of a significant magnitude. For example, how significant is the difference between 1.80 as scored by the US versus 1.88 scored by Ireland (data from pg 25 of the report)? Is that difference of 0.08 really enough to say that the US is much worse off than Ireland? Also, not every country strives to be “innovation friendly” to the same degree. They face different tradeoffs, being at different stages of economic development and with manufacturing playing different roles within their economies. Is greater “innovation friendliness” necessarily better? Countries that want to be more highly ranked would have to incur a cost of doing so (e.g., by increasing innovation-friendly tax credits). Not all countries may benefit in the same way (if at all) from a higher ranking. In the example above, if the US spends a hypothetical sum of $1 billion to improve its score from 1.80 to 1.88, it would match Ireland in the rankings, but would it recoup enough benefit to justify the added expenditure?
In conclusion, the BCG report contains a valuable description of what executives think, while raising many interesting questions about the appropriate level of “innovation friendliness” for each country.
Original Report: The Innovation Imperative in Manufacturing: How the United States Can Restore its Edge