Current Issues in Economics and Finance

The Predictive Abilities of the New York Fed's
Empire State Manufacturing Survey

January 2005 Volume 11, Number 11
Contact Authors
E-mail authors
JEL classification: E23, E27, R11
View PDF version
PDF 7 pages / 268 kb

Authors: Richard Deitz and Charles Steindel

Business surveys often give early signals of the direction and magnitude of economic activity. One release, the relatively new Empire State Manufacturing Survey, is demonstrating an ability to provide information ahead of U.S. production and employment trends. In fact, the predictive power of this survey appears to be at least equal to that of two established manufacturing surveys.

In July 2001, the Federal Reserve Bank of New York launched a monthly survey of New York State manufacturers designed to gauge economic conditions in the state. Public releases of the results of the Empire State Manufacturing Survey began a year later. Particularly notable among the first set of monthly releases were the results issued at 8a.m. ET on Monday, June 15, 2003. Coming on the heels of a severe contraction in manufacturing activity nationwide, the June survey results indicated a significant improvement in New York State manufacturing conditions for the second consecutive month-news that promptly set off a considerable stir in the financial markets. The constant-maturity yield on ten-year U.S. government debt jumped 7basis points that day, and the stock markets saw substantial gains. The Dow Jones Industrial Average rose 201 points, or 2.2percent; the Standard and Poor's 500 Composite Index climbed 22.1points, also 2.2percent; and the NASDAQ Composite Index gained 40points, or 2.5percent.

The Empire State Manufacturing Survey represented the only noteworthy economic or financial news released that day, and the financial press credited it as the impetus for market movements (Browning2003). Market participants evidently interpreted the survey results as a signal that national economic conditions had to some extent turned a corner. Indeed, in the weeks following the survey's release, Federal Reserve data on U.S. manufacturing conditions confirmed a sharp improvement. In June 2003, manufacturing production had risen for the first time in four months and has since been on a sustained upswing.1 Moreover, manufacturing employment began to expand in 2004.

This issue of Second District Highlights considers the performance of the Empire State Manufacturing Survey in tracking national manufacturing conditions from the survey's July 2001 inception to June 2004. We begin by comparing the structure of New York State's manufacturing sector with that of the U.S. manufacturing sector. By doing so, we show that factory conditions in the state should provide a reasonable guide to those in the nation. We also describe the history and construction of the Empire State Survey and analyze the survey's record as an indicator of U.S. manufacturing production and employment patterns. Finally, we compare the Empire State Survey's predictive abilities with those of two other widely followed surveys of business conditions: the Federal Reserve Bank of Philadelphia's Business Outlook Survey and the Institute for Supply Management's ISM Survey.2

We find that despite its relatively short history, the Empire State Survey provides a useful early signal of developments in production and employment in the U.S. manufacturing sector. Moreover, the survey appears to offer information that is at least comparable in its predictive value to the content of the Business Outlook and ISM surveys.

Manufacturing in New York and the United States
Manufacturing employment in New York has undergone a severe long-term contraction. The state's labor market was much more manufacturing-based than it is now: In 1960, 30 percent of the workforce was employed in manufacturing in both New York and the nation; today, the state's figure has fallen to just about 10percent-well below the nation's 14percent.3 Nonetheless, New York State accounts for a substantial share of the country's manufacturing activity, producing about 5percent of U.S. output and employing roughly 4percent of workers.4

Despite the decline in the state's manufacturing employment, reports from New York factories can provide insight into national trends. The makeup of New York's manufacturing sector is fairly comparable to the national composition. In particular, the share of total manufacturing employment claimed by each manufacturing industry is generally similar in New York and the nation (Table1).5 Given this broad similarity, manufacturing trends in New York should be comparable to national trends; thus, a survey of the state's manufacturers could well provide useful information on national developments.

A comparison of another U.S. region with the United States offers a solid basis for our expectation. Since 1968, the Philadelphia Fed has collected data from manufacturers in the Third Federal Reserve District (eastern Pennsylvania, southern New Jersey, and Delaware) for its monthly Business Outlook Survey. The composition of manufacturing employment in the Third District is, like that of New York State, roughly comparable to the national makeup (Table 1). Significantly, studies have shown that the Business Outlook Survey's results track U.S. developments (Harris 1991; Schiller and Trebing 2003).6 Thus, a survey of New York State producers, like the one conducted in the Third District, could potentially yield results that correlate with national trends.

Needless to say, however, while an area's manufacturing mix may resemble that of the United States, a regional survey may fail to offer insight into U.S. conditions. Conversely, a survey of a seemingly unrepresentative region's manufacturers may indeed be informative about national trends. Thus, an analysis of the relationship between survey results and manufacturing conditions must follow a more objective-in our case, statistical-approach. We begin this approach by examining how the Empire State Survey is constructed.

Construction of the Empire State Survey
The Empire State Manufacturing Survey is based on the same methodology and asks the same questions as the Philadelphia Fed's Business Outlook Survey.

Manufacturing companies with 100 or more employees or annual sales of at least $5million are asked to participate in the Empire State Survey. On the first day of each month, the New York Fed sends a questionnaire to about 250 firms across New York State. Firms are added to the pool monthly to replace those that drop out.7 The same individual completes the survey each time, in many instances the CEO or another high-level representative. The New York Fed receives approximately 100 responses each month-roughly 90 percent via the Internet, the remainder by mail. The Fed publishes the survey results on the fifteenth of the month, or on the first business day following if that date falls on a weekend or holiday. Responses that cannot be incorporated into the report are included with revised figures released in the following month.8

The questionnaire asks participants to note the direction of change-"increase," "decrease," or "no change"-but not the magnitude, of a variety of business indicators, starting with those associated with current conditions. Participants indicate the direction of general business activity, which is not necessarily specific to their companies; this indicator serves as an overall barometer of business conditions. The remaining indicators pertain to company-specific conditions, including shipments, new and unfilled orders, delivery times, inventories, prices paid and received, employment, and hours worked. Survey recipients also identify the expected direction of these indicators six months ahead. There is also a question on capital spending, but because such plans typically involve longer term commitments, the question is posed for a six-month horizon.

For each indicator, the New York Fed subtracts the percentage of respondents reporting a decrease from the previous month from those reporting an increase. This exercise results in the creation of a "diffusion index" for each indicator; an index with a positive value means that the percentage of respondents reporting that an indicator had risen from the prior month exceeded the percentage reporting that the indicator had fallen.

Manufacturing conditions can vary because of such seasonal events as model-year adjustments and holiday demand. Accordingly, survey data are adjusted to control for these seasonal variations.9 The Fed adjusts the "increase" and "decrease" percentage components of the diffusion indexes separately, and it computes the seasonally adjusted "no change" component by subtracting from 100 the sum of the adjusted increase and decrease. Seasonal adjustment factors are recalculated for all historic data each December and projected for the upcoming year.

The Empire State Survey's somewhat short history makes it necessary for the New York Fed to splice raw data with data from the same questions in the Business Outlook Survey; this process yields a sufficiently long history for estimating seasonal adjustment factors. Each year, as more Empire State Survey data are added and more Business Outlook Survey data are deleted, seasonal adjustments are aligned more accurately with the New York survey's history.

The Informational Value of the Empire State Survey
Manufacturing surveys such as the one conducted by the New York Fed can provide early indications of conditions before official U.S. data are released. Empire State Survey data, as well as Business Outlook Survey and ISM Survey data, are published before the data on U.S. manufacturing conditions. Significantly, all three surveys are shown to be broadly correlated with national trends in production and employment (Charts1 and 2).

Chart1 plots the monthly change in U.S. manufacturing production against the three surveys' measures of general business conditions between July 2001 (when the first Empire State Survey was compiled) and June 2004.10 The diffusion indexes for the Empire State Survey and the Business Outlook Survey are centered at zero, meaning that values greater than zero are consistent with a rising indicator; the comparably constructed ISM Survey's diffusion indexes are centered at 50, meaning that values above 50 are consistent with a rising indicator. The chart illustrates the broad correlation between the surveys and business conditions: the survey indexes are higher in the months when U.S. production growth generally is faster. A similar correlation for manufacturing employment can be observed in Chart2: when the survey indexes are higher, employment levels improve (that is, they deteriorate less, because virtually no month saw gains in manufacturing payrolls in the period we examine).

As one might expect, these correlations alone do not demonstrate the usefulness of survey information; other data available when the surveys are released may offer equally correlative, and hence predictive, power. For example, predictive information for manufacturing production and employment can be derived from recent movements in the series. These readings are readily available well ahead of ISM Survey data, shortly in advance of Business Outlook Survey data, and more or less at the same time as Empire State Survey data.11

Accordingly, we control for the role played by recent movements in a series in order to assess a survey's ability to provide extra explanatory power. For U.S. manufacturing production and employment, we find that the indexes of all three surveys have such power.

We draw this conclusion from an analysis of the data using the well-established statistical technique known as linear regression. Here, we estimate regressions relating production and employment growth to their past values, to control for the series' recent trends, and to the general business conditions and employment index values of each survey (Tables2 and 3). The regressions were estimated over the July 2001-June 2004 period using each survey's respective indexes, and an additional regression was estimated to include data from all three surveys. The tables demonstrate how the three surveys-alone or in combination-explain the dynamics of manufacturing production and employment growth in this period. The criterion we use to evaluate the surveys' explanatory power is a statistical measure known as the standard error of the estimated regression. The standard error gauges how close the estimated value of a given series typically is to the actual value of the series.12 The smaller the standard error, the closer the relationship between the values-and the stronger the correlation between the surveys and U.S. manufacturing conditions.

The standard error of the regression relating manufacturing production growth to its past values is slightly more than 0.4percentage point (Table2, row 1). This means that between July 2001 and June 2004, the relationship between manufacturing production growth and its past values yielded an estimate of growth typically within 0.4percentage point of the actual change. The regressions relating production growth to each of the three surveys' indexes of general business conditions have standard errors less than 0.4percentage point-strong evidence that each index adds explanatory power beyond that provided by production growth's past values. In fact, the regression for the Empire State Survey index has the lowest standard error (.279). Moreover, adding the other two survey indexes does not reduce this standard error further. Thus, over the brief period since mid-2001, the general business conditions index of the Empire State Survey provides more information about current production than do the corresponding indexes of the Business Outlook Survey and the ISM Survey.13

Regressions on manufacturing employment growth lead to roughly similar results (Table3). As we saw with manufacturing production, all three indexes provide information on employment growth over and above what is supplied solely by past values. A comparison of rows 2-4 reveals that the Empire State Survey measure reduces the standard error more than the other two indexes do. However, contrary to our results in the case of production, the standard error produced when we include all three indexes in the regressions on employment growth is lower than what we obtain when we include individual indexes only (row 5).14

Of note are the values of each survey index that are consistent with "no change" in production or employment. Readings above these "break-even" values could suggest that upcoming data will show an increase during the month. One might expect break-even values of zero for the Empire State and Business Outlook surveys and 50 for the ISM Survey. We say this because at those values of the indexes, the numbers of respondents who indicate improving conditions (rising employment) equal those observing deteriorating conditions (falling employment). When evaluated statistically, however, such an expectation does not hold.

From the regressions, we compute alternate break-even values that are statistically associated with "no change" in either production or employment. These are presented, respectively, in the second columns of Tables2 and 3.15 The Empire State Survey's break-even value for manufacturing production is 7.2 (Table2). Thus, a general business conditions index value of 7.2 or higher in this survey is consistent with growth in manufacturing output during the period. The Philadelphia Fed survey's value of 4.0 is closer to zero, and the ISM Survey's value of 51.5 is also near its center of 50.16 The statistical break-even value of the Empire State Survey is substantially larger than the value of zero expected based on the construction of the diffusion indexes. The relatively high Empire State Survey value might result from differences in the manufacturing sectors between New York State and the Philadelphia Fed's Third District and the national economy, or from systematic differences in the characteristics of the respondents to the surveys.17

The break-even values for employment growth are also above the respective center points for all surveys (Table3). The value of the employment index of the Empire State Survey is 15.1, a long way from zero. The Business Outlook Survey's value is a fairly comparable 13.2, and the ISM Survey's break-even value is 55.2, well above a center of 50.

Our statistical analysis reveals that the Empire State Survey potentially offers significant information about trends in U.S. manufacturing production and employment.18 Fairly large positive readings for the survey are associated with gains in these series. In addition, the survey's predictive power appears to be at least as effective as the abilities of the Business Outlook Survey and the ISM Survey. However, given the Empire State Survey's relatively short history, the robustness of these results is still undetermined. For instance, because the survey has yet to cover a full business cycle, the survey's performance could stem from some other factor that coincidentally has a positive effect on both manufacturing conditions and the survey readings during the early stages of a business cycle expansion. In time, the Empire State Survey's explanatory power relative to that of other surveys may change, along with the magnitude of the break-even values.19

Conclusion
Business surveys often give early signals of the direction and magnitude of economic activity before official data are available. One such release, the New York Fed's Empire State Manufacturing Survey, is demonstrating an ability to provide useful information ahead of developments in the U.S. manufacturing sector. In fact, this information looks to have at least as much predictive power as the content of two established manufacturing surveys: the Institute for Supply Management's ISM Survey and the Federal Reserve Bank of Philadelphia's Business Outlook Survey. Although the Empire State Survey has already compiled a solid track record of prediction, the survey is fairly new. Future experience will reveal whether its noteworthy performance will continue.

The Empire State Manufacturing Survey is available at http://www.newyorkfed.org/
research/regional_economy/empiresurvey_overview.html
.


About the Authors
Richard Deitz is a senior economist in the Buffalo Branch of the Federal Reserve Bank of New York; Charles Steindel is a senior vice president in the Business Conditions Function of the Research and Statistics Group.

Notes

1. On December22, 2004, the Board of Governors of the Federal Reserve System released its annual revision to the industrial production data. We use data available before the release, but this revision would have no significant effect on our analysis.

2. The Business Outlook Survey is the model for the Empire State Survey. The Institute for Supply Management is the former National Association of Purchasing Management (NAPM).

3. Figures are from the New York State Department of Labor.

4. Figures are from the U.S. Commerce Department's Bureau of Economic Analysis.

5. New York's larger share of apparel employment is a key exception, as the table shows.

6. Other Federal Reserve districts conduct similar surveys. Harris, Owens, and Sarte (2004) and Keeton and Verba (2004) discuss the relationship between manufacturing surveys in the Fifth and Tenth Districts, respectively, and manufacturing employment in those districts.

7. This practice potentially biases the survey in a positive way because some firms may drop out as they fail. The bias, however, is likely to be small.

8. These responses actually come in before the cutoff date of the fifteenth. However, they arrive too late on the fourteenth to make the report's release on the next day.

9. For details on seasonal adjustment procedures, see http://www.newyorkfed .org/
research/regional_economy/empiresurvey_season.html
.

10. We focus on the ability of the surveys' general conditions indexes to predict production even though each survey has its own production index. The general conditions indexes are consistently more correlated with production than are the production indexes.

11. In eleven of the months in a given twelve-month period, data on recent movements in these series are available well in advance of all three survey readings. The statement in the text refers only to the twelfth, or last, month of the given period examined.

12. The standard error of a regression is inversely related to its adjusted R2, another common summary statistic. The adjusted R2 is the fraction of the variance in the series explained by the included variables, after adjusting for their number.

13. Schiller and Trebing (2003) also compare the ISM Survey and Business Outlook Survey as predictors of manufacturing output. They find that the general conditions indexes of both surveys improve the forecasting power of an equation explaining manufacturing production growth when they control for the last twelve values of production growth. The Philadelphia measure improves the fit slightly more than the ISM measure does. Harris, Owens, and Sarte (2004) show that the Philadelphia survey, along with the survey compiled by the Federal Reserve Bank of Richmond, predicts movements in the ISM general conditions index. The latter is found to predict changes in GDP. Keeton and Verba (2004) find that the survey index compiled by the Federal Reserve Bank of Kansas City adds explanatory power to the ISM general conditions index in a regression explaining the growth of industrial production. However, the Kansas City employment index does not add any explanatory power to the ISM employment index in a regression explaining the growth of manufacturing employment. Moreover, the Kansas City index is released after the ISM measure.

14. Harris (1991), controlling for consensus employment forecasts, finds that the NAPM employment index adds explanatory power for aggregate payroll employment gains.

15. The break-even values are computed by assuming no change in either production or employment for twelve months and calculating the value for the index that suggests that the condition will persist for yet another month.

16. Harris (1991), using quarterly data, also finds a break-even value of 51.5 for the NAPM index. In contrast, Koenig (2002), also using quarterly data, computes a break-even value of 47.8 for the ISM index. However, neither Harris nor Koenig controls for lagged growth in output. Over the 2001-04 period examined in this article, the break-even value for industrial production for the ISM index without lagged growth is slightly more than 50 (50.6 using Harris' formulation, 50.5 using Koenig's).

17. Companies responding to the Empire State Manufacturing Survey tend to be smaller than respondents to the Business Outlook and ISM surveys because the Empire State Survey allows firms with fewer than 100 employees to participate if their annual revenues exceed $5 million.

18. There are many other factors that may explain production growth. However, we find that the addition of variables such as the change in existing claims for unemployment insurance, the growth in the manufacturing component of the S&P; 500, and the change in motor vehicle assemblies does little or nothing to reduce the regression's standard error further, while the statistical significance of the Empire State Survey's variable is left intact by the changes.
We also find no evidence of a statistically significant relationship between other survey indexes and their corresponding data series, including price indexes and various measures of U.S. producer prices, employment and general conditions indexes and New York State manufacturing employment (although both indexes yield a limited relationship with total New York State employment), and future conditions indexes and growth in U.S. manufacturing output over the following six months.

19. Strictly speaking, these results reflect only the ability of the surveys' revised values to anticipate the revised values of production and employment, not the ability of the initial releases of the surveys to forecast the first estimates. Given the relatively short history of the Empire State Survey, few data are available to make more stringent "real-time" tests, which involve reestimating equations with the receipt of each month's data.


References

Browning, E. S. 2003. "Investors Take Good News and Run with It." Wall Street Journal, June17, p.C1.

Harris, Ethan S. 1991. "Tracking the Economy with the Purchasing Managers' Index." Federal Reserve Bank of New York Quarterly Review16, no.3 (autumn): 61-9.

Harris, Matthew, Raymond E. Owens, and Pierre-Daniel G. Sarte. 2004. "Using Manufacturing Surveys to Assess Economic Conditions." Federal Reserve Bank of Richmond Economic Quarterly90, no.4 (fall): 65-92.

Keeton, William R., and Michael Verba. 2004. "What Can Regional Manufacturing Surveys Tell Us? Lessons from the Tenth District." Federal Reserve Bank of Kansas City Economic Review89, no.3 (third quarter): 39-70.

Koenig, Evan F. 2002. "Using the Purchasing Managers' Index to Assess the Economy's Strength and the Likely Direction of Monetary Policy." Federal Reserve Bank of Dallas Economic and Financial Policy Review1, no.6: 1-14. Available at http://www.dallasfedreview.org/articles/v01_n06_a01.html.

Schiller, Timothy, and Michael Trebing. 2003. "Taking the Measure of Manufacturing." Federal Reserve Bank of Philadelphia Business Review, fourth quarter: 24-37.


Table 1
Composition of Manufacturing Employment, 2002
Percent
Industry New York
State
Third Federal Reserve Districta United States
Heavy metals 24.5 22.9 29.8
Fabricated metals 9.3 10.9 10.2
Machinery 9.2 7.0 8.0
Transportation equipment
and motor vehicles
6.1 5.0 11.6
High-tech 12.1 7.5 9.6
Computer and
electronic products
12.1 7.5 9.6
Textiles and apparel 10.0 5.6 6.0
Apparel 7.2 2.6 2.5
Textile mills 1.3 1.4 1.9
Textile product mills 1.0 1.1 1.3
Leather 0.5 0.5 0.4
Resource-based 7.7 13.3 11.4
Paper 3.4 4.2 3.5
Primary metals 2.1 4.8 3.2
Wood 1.8 3.3 4.0
Petroleum and
coal products
0.4 1.0 0.8
Other manufacturing 42.7 47.7 40.0
Chemicals 9.1 11.9 6.0
Food 8.3 9.9 10.0
Miscellaneous 7.6 5.3 4.9
Printing 6.2 6.1 4.9
Rubber and plastics 4.4 6.0 5.6
Furniture 3.1 3.1 4.1
Glass, clay, and
nonmetallic minerals
3.0 4.2 3.3
Tobacco and beverages 1.1 1.1 1.3

Sources: U.S. Department of Commerce, Bureau of Economic Analysis;
authors' calculations.

aOur calculations are based on the three-state aggregate of Pennsylvania,
New Jersey, and Delaware; the aggregate does not correspond exactly to
Third District boundaries.


Chart 1
Survey Measures of General Business Conditions and
the Monthly Change in U.S. Manufacturing Production

July 2001 to June 2004

Sources: Federal Reserve Bank of New York, Empire State Manufacturing Survey; Federal Reserve Bank of Philadelphia, Business Outlook Survey; Institute for Supply Management, ISM Survey; Board of Governors of the Federal Reserve System;
authors' calculations.

Notes: The ISM series was transformed to have a mean of zero and a similar
range of variation. A reading of 50 for the ISM series would correspond to
a value of 0 on the chart.


Chart 2
Survey Measures of Employment and the Monthly
Change in U.S. Manufacturing Employment
July 2001 to June 2004

Sources: Federal Reserve Bank of New York, Empire State Manufacturing Survey; Federal Reserve Bank of Philadelphia, Business Outlook Survey; Institute for Supply Management, ISM Survey; U.S. Department of Labor, Bureau of Labor Statistics; authors' calculations.

Notes: The ISM series was transformed to have a mean of zero and a similar
range of variation. A reading of 50 for the ISM series would correspond to
a value of 0 on the chart.


Table 2
Results of Regressions of General Business Conditions
Survey Indexes on Manufacturing Production Growth

July 2001 to June 2004

Survey Standard Error
(Percent)
Break-Even Value
No survey 0.414 N.A.
Empire State Manufacturing Survey 0.279 7.2
Business Outlook Survey 0.321 4.0
ISM Survey 0.319 51.5
Three surveys combined 0.288 N.A.

Sources: Federal Reserve Bank of New York, Empire State Manufacturing
Survey
; Federal Reserve Bank of Philadelphia, Business Outlook Survey;
Institute for Supply Management, ISM Survey; authors' calculations.

Note: Regressions include a constant and twelve lagged values of the
dependent variable.


Table 3
Results of Regressions of Employment Survey Indexes
on Manufacturing Employment Growth

July 2001 to June 2004

Survey Standard Error
(Thousands)
Break-Even Value
No survey 23.6 N.A.
Empire State Manufacturing Survey 20.9 15.1
Business Outlook Survey 23.3 13.2
ISM Survey 21.8 55.2
Three surveys combined 20.0 N.A.

Sources: Federal Reserve Bank of New York, Empire State Manufacturing
Survey
; Federal Reserve Bank of Philadelphia, Business Outlook Survey;
Institute for Supply Management, ISM Survey; authors' calculations.

Note: Regressions include a constant and twelve lagged values of the
dependent variable.


Economic Trends in the Second District

Sources: New York, New Jersey, and Connecticut Departments of Labor; U.S. Department of Labor, Bureau of Labor Statistics; Federal Reserve Bank
of New York.

aUpstate New York comprises the four metropolitan areas listed as well as Binghamton, Elmira, Glens Falls, Jamestown, and Utica-Rome.

bThe northern suburbs of New York City comprise Dutchess, Orange, Putnam, Rockland, and Westchester Counties, New York, and Pike County, Pennsylvania.

cNorthern New Jersey comprises Bergen, Essex, Hudson, Hunterdon, Mercer, Middlesex, Monmouth, Morris, Ocean, Passaic, Somerset, Sussex, Union, and Warren Counties.


Disclaimer

The views expressed in this article are those of the authors and do not necessarily reflect the position of the Federal Reserve Bank of New York or the Federal Reserve System.