|Home > Research > Research Publications|
Current Issues in Economics and Finance
Second District Highlights
The Predictive Abilities of the New York Fed's
Empire State Manufacturing Survey
|January 2005 Volume 11, Number 11||
|JEL classification: E23, E27, R11||
Authors: Richard Deitz and Charles Steindel
Business surveys often give early signals of the direction and magnitude of economic activity. One release, the relatively new Empire State Manufacturing Survey, is demonstrating an ability to provide information ahead of U.S. production and employment trends. In fact, the predictive power of this survey appears to be at least equal to that of two established manufacturing surveys.
In July 2001, the Federal Reserve Bank of New York launched a monthly survey of New York State manufacturers designed to gauge economic conditions in the state. Public releases of the results of the Empire State Manufacturing Survey began a year later. Particularly notable among the first set of monthly releases were the results issued at 8 a.m. ET on Monday, June 15, 2003. Coming on the heels of a severe contraction in manufacturing activity nationwide, the June survey results indicated a significant improvement in New York State manufacturing conditions for the second consecutive month-news that promptly set off a considerable stir in the financial markets. The constant-maturity yield on ten-year U.S. government debt jumped 7 basis points that day, and the stock markets saw substantial gains. The Dow Jones Industrial Average rose 201 points, or 2.2 percent; the Standard and Poor's 500 Composite Index climbed 22.1 points, also 2.2 percent; and the NASDAQ Composite Index gained 40 points, or 2.5 percent.
The Empire State Manufacturing Survey represented the only noteworthy economic or financial news released that day, and the financial press credited it as the impetus for market movements (Browning 2003). Market participants evidently interpreted the survey results as a signal that national economic conditions had to some extent turned a corner. Indeed, in the weeks following the survey's release, Federal Reserve data on U.S. manufacturing conditions confirmed a sharp improvement. In June 2003, manufacturing production had risen for the first time in four months and has since been on a sustained upswing.1 Moreover, manufacturing employment began to expand in 2004.
This issue of Second District Highlights considers the performance of the Empire State Manufacturing Survey in tracking national manufacturing conditions from the survey's July 2001 inception to June 2004. We begin by comparing the structure of New York State's manufacturing sector with that of the U.S. manufacturing sector. By doing so, we show that factory conditions in the state should provide a reasonable guide to those in the nation. We also describe the history and construction of the Empire State Survey and analyze the survey's record as an indicator of U.S. manufacturing production and employment patterns. Finally, we compare the Empire State Survey's predictive abilities with those of two other widely followed surveys of business conditions: the Federal Reserve Bank of Philadelphia's Business Outlook Survey and the Institute for Supply Management's ISM Survey.2
We find that despite its relatively short history, the Empire State Survey provides a useful early signal of developments in production and employment in the U.S. manufacturing sector. Moreover, the survey appears to offer information that is at least comparable in its predictive value to the content of the Business Outlook and ISM surveys.
Manufacturing in New York and the United
Despite the decline in the state's manufacturing employment, reports from New York factories can provide insight into national trends. The makeup of New York's manufacturing sector is fairly comparable to the national composition. In particular, the share of total manufacturing employment claimed by each manufacturing industry is generally similar in New York and the nation (Table 1).5 Given this broad similarity, manufacturing trends in New York should be comparable to national trends; thus, a survey of the state's manufacturers could well provide useful information on national developments.
A comparison of another U.S. region with the United States offers a solid basis for our expectation. Since 1968, the Philadelphia Fed has collected data from manufacturers in the Third Federal Reserve District (eastern Pennsylvania, southern New Jersey, and Delaware) for its monthly Business Outlook Survey. The composition of manufacturing employment in the Third District is, like that of New York State, roughly comparable to the national makeup (Table 1). Significantly, studies have shown that the Business Outlook Survey's results track U.S. developments (Harris 1991; Schiller and Trebing 2003).6 Thus, a survey of New York State producers, like the one conducted in the Third District, could potentially yield results that correlate with national trends.
Needless to say, however, while an area's manufacturing mix may resemble that of the United States, a regional survey may fail to offer insight into U.S. conditions. Conversely, a survey of a seemingly unrepresentative region's manufacturers may indeed be informative about national trends. Thus, an analysis of the relationship between survey results and manufacturing conditions must follow a more objective-in our case, statistical-approach. We begin this approach by examining how the Empire State Survey is constructed.
Construction of the Empire State
Manufacturing companies with 100 or more employees or annual sales of at least $5 million are asked to participate in the Empire State Survey. On the first day of each month, the New York Fed sends a questionnaire to about 250 firms across New York State. Firms are added to the pool monthly to replace those that drop out.7 The same individual completes the survey each time, in many instances the CEO or another high-level representative. The New York Fed receives approximately 100 responses each month-roughly 90 percent via the Internet, the remainder by mail. The Fed publishes the survey results on the fifteenth of the month, or on the first business day following if that date falls on a weekend or holiday. Responses that cannot be incorporated into the report are included with revised figures released in the following month.8
The questionnaire asks participants to note the direction of change-"increase," "decrease," or "no change"-but not the magnitude, of a variety of business indicators, starting with those associated with current conditions. Participants indicate the direction of general business activity, which is not necessarily specific to their companies; this indicator serves as an overall barometer of business conditions. The remaining indicators pertain to company-specific conditions, including shipments, new and unfilled orders, delivery times, inventories, prices paid and received, employment, and hours worked. Survey recipients also identify the expected direction of these indicators six months ahead. There is also a question on capital spending, but because such plans typically involve longer term commitments, the question is posed for a six-month horizon.
For each indicator, the New York Fed subtracts the percentage of respondents reporting a decrease from the previous month from those reporting an increase. This exercise results in the creation of a "diffusion index" for each indicator; an index with a positive value means that the percentage of respondents reporting that an indicator had risen from the prior month exceeded the percentage reporting that the indicator had fallen.
Manufacturing conditions can vary because of such seasonal events as model-year adjustments and holiday demand. Accordingly, survey data are adjusted to control for these seasonal variations.9 The Fed adjusts the "increase" and "decrease" percentage components of the diffusion indexes separately, and it computes the seasonally adjusted "no change" component by subtracting from 100 the sum of the adjusted increase and decrease. Seasonal adjustment factors are recalculated for all historic data each December and projected for the upcoming year.
The Empire State Survey's somewhat short history makes it necessary for the New York Fed to splice raw data with data from the same questions in the Business Outlook Survey; this process yields a sufficiently long history for estimating seasonal adjustment factors. Each year, as more Empire State Survey data are added and more Business Outlook Survey data are deleted, seasonal adjustments are aligned more accurately with the New York survey's history.
The Informational Value of the Empire
Chart 1 plots the monthly change in U.S. manufacturing production against the three surveys' measures of general business conditions between July 2001 (when the first Empire State Survey was compiled) and June 2004.10 The diffusion indexes for the Empire State Survey and the Business Outlook Survey are centered at zero, meaning that values greater than zero are consistent with a rising indicator; the comparably constructed ISM Survey's diffusion indexes are centered at 50, meaning that values above 50 are consistent with a rising indicator. The chart illustrates the broad correlation between the surveys and business conditions: the survey indexes are higher in the months when U.S. production growth generally is faster. A similar correlation for manufacturing employment can be observed in Chart 2: when the survey indexes are higher, employment levels improve (that is, they deteriorate less, because virtually no month saw gains in manufacturing payrolls in the period we examine).
As one might expect, these correlations alone do not demonstrate the usefulness of survey information; other data available when the surveys are released may offer equally correlative, and hence predictive, power. For example, predictive information for manufacturing production and employment can be derived from recent movements in the series. These readings are readily available well ahead of ISM Survey data, shortly in advance of Business Outlook Survey data, and more or less at the same time as Empire State Survey data.11
Accordingly, we control for the role played by recent movements in a series in order to assess a survey's ability to provide extra explanatory power. For U.S. manufacturing production and employment, we find that the indexes of all three surveys have such power.
We draw this conclusion from an analysis of the data using the well-established statistical technique known as linear regression. Here, we estimate regressions relating production and employment growth to their past values, to control for the series' recent trends, and to the general business conditions and employment index values of each survey (Tables 2 and 3). The regressions were estimated over the July 2001-June 2004 period using each survey's respective indexes, and an additional regression was estimated to include data from all three surveys. The tables demonstrate how the three surveys-alone or in combination-explain the dynamics of manufacturing production and employment growth in this period. The criterion we use to evaluate the surveys' explanatory power is a statistical measure known as the standard error of the estimated regression. The standard error gauges how close the estimated value of a given series typically is to the actual value of the series.12 The smaller the standard error, the closer the relationship between the values-and the stronger the correlation between the surveys and U.S. manufacturing conditions.
The standard error of the regression relating manufacturing production growth to its past values is slightly more than 0.4 percentage point (Table 2, row 1). This means that between July 2001 and June 2004, the relationship between manufacturing production growth and its past values yielded an estimate of growth typically within 0.4 percentage point of the actual change. The regressions relating production growth to each of the three surveys' indexes of general business conditions have standard errors less than 0.4 percentage point-strong evidence that each index adds explanatory power beyond that provided by production growth's past values. In fact, the regression for the Empire State Survey index has the lowest standard error (.279). Moreover, adding the other two survey indexes does not reduce this standard error further. Thus, over the brief period since mid-2001, the general business conditions index of the Empire State Survey provides more information about current production than do the corresponding indexes of the Business Outlook Survey and the ISM Survey.13
Regressions on manufacturing employment growth lead to roughly similar results (Table 3). As we saw with manufacturing production, all three indexes provide information on employment growth over and above what is supplied solely by past values. A comparison of rows 2-4 reveals that the Empire State Survey measure reduces the standard error more than the other two indexes do. However, contrary to our results in the case of production, the standard error produced when we include all three indexes in the regressions on employment growth is lower than what we obtain when we include individual indexes only (row 5).14
Of note are the values of each survey index that are consistent with "no change" in production or employment. Readings above these "break-even" values could suggest that upcoming data will show an increase during the month. One might expect break-even values of zero for the Empire State and Business Outlook surveys and 50 for the ISM Survey. We say this because at those values of the indexes, the numbers of respondents who indicate improving conditions (rising employment) equal those observing deteriorating conditions (falling employment). When evaluated statistically, however, such an expectation does not hold.
From the regressions, we compute alternate break-even values that are statistically associated with "no change" in either production or employment. These are presented, respectively, in the second columns of Tables 2 and 3.15 The Empire State Survey's break-even value for manufacturing production is 7.2 (Table 2). Thus, a general business conditions index value of 7.2 or higher in this survey is consistent with growth in manufacturing output during the period. The Philadelphia Fed survey's value of 4.0 is closer to zero, and the ISM Survey's value of 51.5 is also near its center of 50.16 The statistical break-even value of the Empire State Survey is substantially larger than the value of zero expected based on the construction of the diffusion indexes. The relatively high Empire State Survey value might result from differences in the manufacturing sectors between New York State and the Philadelphia Fed's Third District and the national economy, or from systematic differences in the characteristics of the respondents to the surveys.17
The break-even values for employment growth are also above the respective center points for all surveys (Table 3). The value of the employment index of the Empire State Survey is 15.1, a long way from zero. The Business Outlook Survey's value is a fairly comparable 13.2, and the ISM Survey's break-even value is 55.2, well above a center of 50.
Our statistical analysis reveals that the Empire State Survey potentially offers significant information about trends in U.S. manufacturing production and employment.18 Fairly large positive readings for the survey are associated with gains in these series. In addition, the survey's predictive power appears to be at least as effective as the abilities of the Business Outlook Survey and the ISM Survey. However, given the Empire State Survey's relatively short history, the robustness of these results is still undetermined. For instance, because the survey has yet to cover a full business cycle, the survey's performance could stem from some other factor that coincidentally has a positive effect on both manufacturing conditions and the survey readings during the early stages of a business cycle expansion. In time, the Empire State Survey's explanatory power relative to that of other surveys may change, along with the magnitude of the break-even values.19
The Empire State Manufacturing Survey
is available at http://www.newyorkfed.org/
|About the Authors
Richard Deitz is a senior economist in the Buffalo Branch of the Federal Reserve Bank of New York; Charles Steindel is a senior vice president in the Business Conditions Function of the Research and Statistics Group.
1. On December 22, 2004, the Board of Governors of the Federal Reserve System released its annual revision to the industrial production data. We use data available before the release, but this revision would have no significant effect on our analysis.
2. The Business Outlook Survey is the model for the Empire State Survey. The Institute for Supply Management is the former National Association of Purchasing Management (NAPM).
3. Figures are from the New York State Department of Labor.
4. Figures are from the U.S. Commerce Department's Bureau of Economic Analysis.
5. New York's larger share of apparel employment is a key exception, as the table shows.
6. Other Federal Reserve districts conduct similar surveys. Harris, Owens, and Sarte (2004) and Keeton and Verba (2004) discuss the relationship between manufacturing surveys in the Fifth and Tenth Districts, respectively, and manufacturing employment in those districts.
7. This practice potentially biases the survey in a positive way because some firms may drop out as they fail. The bias, however, is likely to be small.
8. These responses actually come in before the cutoff date of the fifteenth. However, they arrive too late on the fourteenth to make the report's release on the next day.
9. For details on seasonal adjustment procedures,
10. We focus on the ability of the surveys' general conditions indexes to predict production even though each survey has its own production index. The general conditions indexes are consistently more correlated with production than are the production indexes.
11. In eleven of the months in a given twelve-month period, data on recent movements in these series are available well in advance of all three survey readings. The statement in the text refers only to the twelfth, or last, month of the given period examined.
12. The standard error of a regression is inversely related to its adjusted R2, another common summary statistic. The adjusted R2 is the fraction of the variance in the series explained by the included variables, after adjusting for their number.
13. Schiller and Trebing (2003) also compare the ISM Survey and Business Outlook Survey as predictors of manufacturing output. They find that the general conditions indexes of both surveys improve the forecasting power of an equation explaining manufacturing production growth when they control for the last twelve values of production growth. The Philadelphia measure improves the fit slightly more than the ISM measure does. Harris, Owens, and Sarte (2004) show that the Philadelphia survey, along with the survey compiled by the Federal Reserve Bank of Richmond, predicts movements in the ISM general conditions index. The latter is found to predict changes in GDP. Keeton and Verba (2004) find that the survey index compiled by the Federal Reserve Bank of Kansas City adds explanatory power to the ISM general conditions index in a regression explaining the growth of industrial production. However, the Kansas City employment index does not add any explanatory power to the ISM employment index in a regression explaining the growth of manufacturing employment. Moreover, the Kansas City index is released after the ISM measure.
14. Harris (1991), controlling for consensus employment forecasts, finds that the NAPM employment index adds explanatory power for aggregate payroll employment gains.
15. The break-even values are computed by assuming no change in either production or employment for twelve months and calculating the value for the index that suggests that the condition will persist for yet another month.
16. Harris (1991), using quarterly data, also finds a break-even value of 51.5 for the NAPM index. In contrast, Koenig (2002), also using quarterly data, computes a break-even value of 47.8 for the ISM index. However, neither Harris nor Koenig controls for lagged growth in output. Over the 2001-04 period examined in this article, the break-even value for industrial production for the ISM index without lagged growth is slightly more than 50 (50.6 using Harris' formulation, 50.5 using Koenig's).
17. Companies responding to the Empire State Manufacturing Survey tend to be smaller than respondents to the Business Outlook and ISM surveys because the Empire State Survey allows firms with fewer than 100 employees to participate if their annual revenues exceed $5 million.
18. There are many other factors that may explain
production growth. However, we find that the addition of variables
such as the change in existing claims for unemployment insurance,
the growth in the manufacturing component of the S&P; 500,
and the change in motor vehicle assemblies does little or
nothing to reduce the regression's standard error further,
while the statistical significance of the Empire State
Survey's variable is left intact by the changes.
19. Strictly speaking, these results reflect only the ability of the surveys' revised values to anticipate the revised values of production and employment, not the ability of the initial releases of the surveys to forecast the first estimates. Given the relatively short history of the Empire State Survey, few data are available to make more stringent "real-time" tests, which involve reestimating equations with the receipt of each month's data.
Browning, E. S. 2003. "Investors Take Good News and Run with It." Wall Street Journal, June 17, p. C1.
Harris, Ethan S. 1991. "Tracking the Economy with the Purchasing Managers' Index." Federal Reserve Bank of New York Quarterly Review 16, no. 3 (autumn): 61-9.
Harris, Matthew, Raymond E. Owens, and Pierre-Daniel G. Sarte. 2004. "Using Manufacturing Surveys to Assess Economic Conditions." Federal Reserve Bank of Richmond Economic Quarterly 90, no. 4 (fall): 65-92.
Keeton, William R., and Michael Verba. 2004. "What Can Regional Manufacturing Surveys Tell Us? Lessons from the Tenth District." Federal Reserve Bank of Kansas City Economic Review 89, no. 3 (third quarter): 39-70.
Koenig, Evan F. 2002. "Using the Purchasing Managers' Index to Assess the Economy's Strength and the Likely Direction of Monetary Policy." Federal Reserve Bank of Dallas Economic and Financial Policy Review 1, no. 6: 1-14. Available at http://www.dallasfedreview.org/articles/v01_n06_a01.html.
Schiller, Timothy, and Michael Trebing. 2003. "Taking the Measure of Manufacturing." Federal Reserve Bank of Philadelphia Business Review, fourth quarter: 24-37.
The views expressed in this article are those of the authors and do not necessarily reflect the position of the Federal Reserve Bank of New York or the Federal Reserve System.