The Employment Forecast Survey 9781487592820

Citation preview

THE EMPLOYMENT FORECAST SURVEY

CANADIAN STUDIES IN E C O N O M I C S A series of studies, edited by V. W. Bladen, sponsored by the Canadian Social Science Research Council, and published with financial assistance from the Canada Council. 1. International Cycles and Canada's Balance of Payments, 1921—33. By Vernon W. Malach. 2.

Capital Formation in Canada, 1896-1930. By Kenneth Buckley.

3.

Natural Resources: The Economics of Conservation. By Anthony Scott.

4.

The Canadian Nickel Industry. By O. W. Main.

5.

Bank of Canada Operations, 1935-54. By E. P. Neufeld.

6.

State Intervention and Assistance in Collective Bargaining: The Canadian Experience, 1943–1954. By H. A. Logan.

7.

The Agricultural Implement Industry in Canada: A Study of Competition. By W. G. Phillips.

8.

Monetary and Fiscal Thought and Policy in Canada, 1919-1939. By Irving Brecher.

9.

Customs Administration in Canada. By Gordon Blake.

10.

Inventories and the Business Cycle with Special Reference to Canada. By Clarence L. Barber.

11.

The Canadian Economy in the Great Depression. By A. E. Safarían.

12. Britain's Export Trade with Canada. By G. L. Reuber. 13.

The Canadian Dollar, 1948-1958. By Paul Wonnacott.

14.

The Employment Forecast Survey. By Douglas G. Hartle.

THE

EMPLOYMENT FORECAST SURVEY

BY

Douglas G. Hartle

UNIVERSITY OF TORONTO PRESS

Copyright, Canada, 1962, by University of Toronto Press

PREFACE

THIS MONOGRAPH marks the completion of another phase of an investigation of the Employment Forecast Survey which has been conducted by the Economics and Research Branch of the Canadian Department of Labour since 1946. In the summer of 1953 I was engaged by the Department as a research assistant to Professor William C. Hood of the University of Toronto who had been asked to make a general appraisal of the survey that summer. The report which resulted from that investigation was concerned with administrative and policy matters as well as more technical considerations, but it constituted the foundation of the present study. In 1956—7 I was employed by the Department of Labour again. Among other duties I was responsible for further studies of the survey and for its administration. This stage of the investigation was devoted almost exclusively to an analysis of the predictions which had been derived from the EFS establishment forecasts, and to an evaluation of the improvements which might have been made in their reliability. The results were embodied in my doctoral dissertation which was accepted by Duke University in April, 1957. After joining the staff of the University of Toronto in the fall of 1957 I undertook an analysis of the individual establishment forecasts. This aspect of the EFS had been relatively neglected in the earlier investigations largely because of the limited computational resources available. The present monograph incorporates the findings of all of the foregoing investigations. The EFS industry employment predictions, which are one of the main objects of this investigation, were published in confidential bulletins circulated within the Canadian government. The establishment forecast data upon which these predictions were based, and which were another important object of analysis, have been retained on the restricted files of the Economics and Research Branch of the Department of Labour. Without the active co-operation of the Department in giving access to this material, and in granting me permission to publish the results, this investigation could not have been undertaken. I particularly wish to acknowledge the co-operation and help afforded by Dr. George V. Haythorne, Dr. William R. Dymond, and Dr. Gil Shonning. Professors Frank A. Hanna and Joseph J. Spengler of Duke University were particularly helpful with respect to my dissertation, and Dean Vincent W. Bladen, formerly Chairman of the Department of Political Economy, University of Toronto, provided support and encouragement in undertaking the final phase of the study. Professor William C. Hood has, over the years, most generously provided of his time and advice. His interest has been unfailing. To all of them I would like to express my warmest appreciation. Financial assistance was provided by the Commonwealth-Studies Center of Duke University, the Canada Council, and the Rockefeller grant of the University of Toronto. I am pleased to acknowledge this assistance with gratitude.

VI

PREFACE

The clerical assistance provided by Miss Joan Bindoff of the Department of Labour and the computational assistance provided by Mr. J. Csima of the Computational Centre of the University of Toronto has been invaluable. D. G. H. University of Toronto

CONTENTS PREFACE

v

TABLES

x

CHARTS

xii

I. INTRODUCTION II.

DESCRIPTION OF THE EMPLOYMENT FORECAST SURVEY AND OTHER SURVEYS OF EMPLOYMENT EXPECTATIONS 1. The United States Bureau of Employment Security's Surveys of "Employers' Forecasts of Labor Requirements" 2. The Dun and Bradstreet "Surveys of Business Expectations" 3. The "Business Test" of the IFO Institute 4. The Canadian Department of Labour's "Employment Forecast Survey" 5. Differences among the Surveys

III. REVIEW OF PREVIOUS RESEARCH 1. The Bureau of Employment Security's Surveys of "Employers' Forecasts of Future Labor Requirements" 2. Dun and Bradstreet's "Surveys of Business Expectations" ... 3. The "Business Test" of the IFO Institute 4. The "Employment Forecast Survey" of the Canadian Department of Labour 5. The Purpose of the Present Study IV. EVALUATION AND ANALYSIS OF PREVIOUS EFS PREDICTIONS . . . . 1. Description of the Errors of the EFS Predictions 2. Evaluation of the Errors of the EFS Predictions V. THE SOURCES OF PREDICTIVE ERRORS 1. Conceptual Framework 2. The Estimation of Sampling and Forecast Errors VI. THE SAMPLING PROBLEM 1. A Description of the Sampling Errors 2. The Determinants of the Sampling Problem 3. The Resolution of the Sampling Problem 4. Summary and Conclusion VII. THE FORECAST PROBLEM 1. Description and Analysis of the EFS Sample Forecast Errors

3 4 4 6 8 9 17 18 19 21 23 24 26 28 28 38 43 43 44 49 50 57 60 62 63 63

VIII

CONTENTS

VIII. THE PREDICTIVE VALUE OF THE ESTABLISHMENT FORECASTS . . . . 1. Forecasts of Employment Levels by Individual Establishments 2. Predictions of the Direction of Non-Seasonal Changes by Individual Establishments 3. The Adequacy of the Forecasts Submitted by the Group of Sample Establishments 4. Conclusions IX.

THE SOURCES AND REDUCTION OF ESTABLISHMENT FORECAST ERRORS 1. The Sources of Forecast Errors 2. The Reduction of Establishment Forecast Errors

X.

IMPLICATIONS FOR DERIVATION OF MORE RELIABLE INDUSTRY PREDICTIONS 1. Evaluation Assuming Sample Forecast Errors Were Reliable Estimates of Population Values 2. The Reliability of the Sample Forecast Errors as Estimates of Population Values 3. Implications for the Derivation of More Reliable Industry Predictions by the Ratio Estimate Approach 4. Implications for the Derivation of More Reliable Industry Predictions by the Diffusion Index Approach 5. Conclusions

XL

GENERAL IMPLICATIONS

APPENDIXES A. Industrial Classification Used in the Employment Forecast Survey, 1952-6 B. Description of the Employment Forecast Survey Sample ... C. Employment Forecast Survey Shuttle Card D. The Errors of the EFS Predictions of the Direction of the NonSeasonal Changes in DBS Employment Indexes, Classified by Quarterly Target Date, Manufacturing Industries, January 1, 1952–July 1, 1956 E. The Percentage of EFS Predictions which Underestimated the Magnitudes of the Non-Seasonal Declines and Gains in DBS Employment Indexes, Manufacturing Industries, January 1, 1952-July 1, 1956 F. The Errors of the EFS Predictions of the Magnitudes of the NonSeasonal Changes in DBS Employment Indexes, Classified in Terms of the Direction of the Corresponding Actual NonSeasonal Changes, Manufacturing Industries, January 1, 1952July 1, 1956 G. The Errors of the EFS Predictions of the Direction of the NonSeasonal Changes in DBS Employment Indexes, Manufacturing Industries, January 1, 1952-July 1, 1956

72 73 75 77 86 87 87 90 95 97 99 102 102 104 105 109 111 113 118

119

122

124 126

CONTENTS

H. Comparison of the Errors of the EFS and Hypothetical Predictions of the Magnitudes of the Non-Seasonal Changes in DBS Employment Indexes, Manufacturing Industries, January 1, 1952-July 1, 1956 I. Percentage of Hypothetical Predictions with Smaller Errors than Comparable EFS Predictions, Manufacturing Industries, January 1, 1952-July 1, 1956 J. Determinants of the Reliability of EFS Predictions K. The Estimation of the EFS Sample Size Requirements with an Alternative Design

IX

128 129 130 137

NOTES

139

BIBLIOGRAPHY

148

INDEX

151

TABLES I. Number of Sample Establishments which Submitted Forecasts for the April 1 Target Date, Selected Years, 1947–55 II. Employment of Sample Establishments as a Percentage of the Employment of the Population of Establishments, Selected Years, 1947-55 III. The Errors of the EFS Predictions of the Magnitudes of the NonSeasonal Changes in DBS Employment Indexes, for the Manufacturing Industries, January 1, 1952–July 1, 1956 IV. The Errors of the EFS Predictions of the Magnitudes of the NonSeasonal Changes in DBS Employment Indexes, Classified by Quarterly Target Date, Manufacturing, January 1, 1952–July 1, 1956 V. Percentage of Manufacturing Industries for Which the Signs of the Errors of the EFS Predictions Made for the Same Quarterly Target Date Remained Unchanged, January 1, 1952-July 1, 1956 . ... VI. The Frequency with Which the Magnitudes of the Non-Seasonal Changes in DBS Indexes were Underestimated, Manufacturing Industries, January 1, 1952-July 1, 1956 VII. Frequency of Incorrect Predictions of the Direction of Non-Seasonal Changes in DBS Employment Indexes, Manufacturing Industries, January 1, 1952-July 1, 1956 VIII. Comparison of the Errors of the EFS and Hypothetical Predictions of the Magnitudes of the Non-Seasonal Changes in DBS Employment Indexes, Manufacturing Industries, January 1, 1952–July 1, 1956 IX. The Number of Incorrect EFS Predictions of the Direction of the Non-Seasonal Changes in DBS Employment Indexes, Manufacturing Industries, January 1, 1952-July 1, 1956 X. Mean (absolute) Errors of the Original and Revised EFS Predictions, Manufacturing Industries, April 1, 1953–July 1, 1955 ... XI. Estimated Sampling Errors, Manufacturing Industries, April 1, 1953-July 1, 1955 XII. The Relationship between the Actual Changes in DBS Employment Indexes and the Estimated EFS Sampling Errors, Manufacturing Industries, April 1, 1953–July 1, 1955 XIII. Incorrect Estimates of the Direction of the Non-Seasonal Changes in DBS Employment Indexes Derived from the Changes in the Actual Employment of the Sample Establishments, Manufacturing Industries, April 1, 1953–July 1, 1955 XIV. Relationship between the Coverage of the EFS Sample and the Reliability of the Sample Estimates, Manufacturing Industries, April 1, 1953-July 1, 1955

12 13 32

34

35 36 37

39

42 48 50 51

52 58

TABLES

XV. Estimated Forecast Errors of the EFS Sample, Manufacturing Industries, April 1, 1953–July 1, 1955 XVI. The Relationship between the Actual Changes in the Employment of the EFS Sample and the Estimated EFS Sample Forecast Errors, Manufacturing Industries, April 1, 1953-July 1, 1955 XVII. Incorrect Predictions of the Direction of the Non-Seasonal Changes in the Employment of the Sample, Manufacturing Industries, April 1, 1953-July 1, 1955 XVIII. Proportion of Sample Establishments which Submitted Six-Month Forecasts with No Net Predictive Value with Respect to the Prediction of their own Employment Levels, Manufacturing Industries, October, 1953–October 1957 XIX. Proportion of Sample Establishments which Submitted Six-Month Forecasts Which Failed to Correctly Predict the Direction of the Non-Seasonal Changes in their own Employment at least Half of the Time, Manufacturing Industries, October, 1953-October, 1957 XX. Comparison of the Average Relative Errors of All of the Forecasts Submitted by All of the Establishments with the Errors of Comparable Estimates, Manufacturing Industries, October, 1953–October, 1957 XXI. Summary of the Number and Proportion of Correct Establishment Predictions of the Direction of Non-Seasonal Change, by Type of Interval, Manufacturing Industries, October, 1953-October, 1957 XXII. The Proportion of Establishments which Submitted Six-Month Forecasts with No Net Predictive Value with Respect to the Prediction of their Employment Levels, Classified by Size of Establishment, Manufacturing Industries, October, 1953-October, 1957 XXIII. Proportion of Establishments which Submitted Six-Month Forecasts which Incorrectly Predicted the Direction of the Non-Seasonal Changes in their own Employment at least Half of the Time, Classified by Size of Establishment, Manufacturing Industries, October, 1953-October, 1957 XXIV. The Net Change in the Errors of the Revised EFS Predictions Resulting from the Elimination of Sampling Errors, Manufacturing Industries, April 1, 1953-July 1, 1955 XXV. Mean Percentage Forecast Errors of EFS Establishments, by Size of Establishments, by Length of Forecast, and by Target Date, January 1, 1951–July 1, 1955

XI

64 65 66

74

76

78 80

82

84 98 100

CHARTS 1. The EFS and DBS Employment Indexes for Manufacturing, Target Dates January 1, 1949, to July 1, 1956 2. Predicted (EFS) and Actual (DBS) Year-to-Year Percentage Changes in Manufacturing Employment Indexes, for EFS Predictions Based on SixMonth Forecasts, Target Dates January 1, 1949, to July 1, 1956 . . . . 3. The Actual Year-to-Year Percentage Changes in the DBS Employment Indexes and the Employment in the EFS Sample, Manufacturing, Target Dates April 1, 1953, to July 1, 1955 4. The Actual Year-to-Year Percentage Changes in the DBS Employment Indexes and the Employment in the EFS Sample, Basic Materials Industry Group, Target Dates April 1, 1953, to July 1, 1955 5. The Actual Year-to-Year Percentage Changes in the DBS Employment Indexes and the Employment in the EFS Sample, Consumer Finished Goods Industry Group, Target Dates April 1, 1953, to July 1, 1955 ... 6. The Actual Year-to-Year Percentage Changes in the DBS Employment Indexes and the Employment in the EFS Sample, Producer Finished Goods Industry Group, Target Dates April 1, 1953, to July 1, 1955 7. The Actual and Forecast Year-to-Year Percentage Changes in the Employment of the EFS Sample, Manufacturing Industry Group, Target Dates April 1, 1953, to July 1, 1955 8. The Actual and Forecast Year-to-Year Percentage Changes in the Employment of the EFS Sample, Basic Materials Industry Group, Target Dates April 1, 1953, to July 1, 1955 9. The Actual and Forecast Year-to-Year Percentage Changes in the Employment of the EFS Sample, Consumer Finished Goods Industry Group, Target Dates April 1, 1953, to July 1, 1955 10. The Actual and Forecast Year-to-Year Percentage Changes in the Employment of the EFS Sample, Producer Finished Goods Industry Group, Target Dates April 1, 1953, to July 1, 1955

29 31 53 54 55 56 67 68 69 70

THE E M P L O Y M E N T FORECAST SURVEY

This page intentionally left blank

CHAPTER ONE

INTRODUCTION IN 1946 THE DEPARTMENT OF LABOUR of the Canadian government initiated the Employment Forecast Survey as one technique1 by which it could obtain the information necessary to carry out the economic stabilization programme adopted in the previous year.2 The original conception of the survey, or the EFS as it has been known, was comparatively simple. It seemed reasonable, a priori, that the individual decisionmakers within firms would be able to forecast the future employment of their own firms with considerable reliability. From forecasts3 submitted by a sample that included the most dynamic firms in each industry, it appeared plausible that adequate industry predictions might be derived. A stable sample of approximately 800 establishments4 was therefore selected, and a respondent in each establishment was asked to report the actual employment of the establishment on the three preceding months and to forecast the employment of the establishment three and six months in the future. Although the composition of the sample has changed somewhat over the years, generally speaking the same establishments have continued to submit the same information down to the present time. For the first ten years of the survey, that is to say until the end of 1956, the forecasts submitted by the sample establishments, aggregated by industry, were used to project the latest available indexes of the actual employment of certain industries to the future target dates.5 These projected employment indexes, which were termed EFS employment indexes or EFS employment predictions, provided quantitative estimates of future industry employment levels and trends. In 1957, as the result of the first phase of this investigation,6 the original aggregation procedures were abandoned, and another technique, which was an adaptation of the diffusion index approach, was adopted. The number of industries for which predictions were derived was also substantially reduced. At that time it appeared that predictions based solely on the proportion of establishments forecasting yearto-year gains in their own employment provided equally reliable information at a substantially smaller administrative cost. This more restricted and economical procedure has since been maintained by the Department of Labour. With the exception of a few brief remarks in chapter ix the present monograph does not attempt to analyse or evaluate the employment predictions derived by the new procedure; instead this study constitutes a report of the findings of the first phase of the investigation and of the results of the additional research which has been conducted over the past two years.

CHAPTER TWO

D E S C R I P T I O N OF THE E M P L O Y M E N T FORECAST SURVEY AND OTHER SURVEYS OF E M P L O Y M E N T EXPECTATIONS THE EPS WAS only one of several surveys of businessmen's expectations born during or shortly after World War II.1 With the exception of the surveys of the "National Forecasts of the Regional Shippers Advisory Boards, " which have been conducted continuously by the Association of American Railroads since the late 1920's, virtually all surveys of this type in North America were initiated in the period 1942-8. The survey of "Employers' Forecasts of Labor Requirements," which is conducted by the Bureau of Employment Security of the United States Department of Labor, and the "Survey of Plant and Equipment Programs," which is carried out jointly by the Office of Business Economics in the United States Department of Commerce and the United States Securities and Exchange Commission, were both initiated during World War II. These two surveys probably marked the beginning of the post-war enthusiasm for the expectational approach. The Dun and Bradstreet "Survey of Businessmen's Expectations" was begun in 1948, and appears to have been about the last major forecast survey undertaken on this continent for which results have been published regularly. Interest in the forecast survey seems to have spread to the rest of the world a few years later. In 1950, an elaborate "Business Test" was started by the IFO Institute of Munich, Germany. Shortly thereafter about eight other countries, including Japan, Austria, France, and the Union of South Africa, adopted surveys patterned after the IFO model. Although nearly all of the published expectational surveys were begun within a few years of each other, many important differences exist among them -with respect to the selection of respondents, the manner and form in which the data are solicited, the variables considered, and the procedures by which the establishment forecasts are aggregated. Even among the four published surveys which include employment forecasts, important differences are found.2 To a brief description of these surveys we now turn. 1. THE UNITED STATES BUREAU OF EMPLOYMENT SECURITY'S SURVEYS OF "EMPLOYERS' FORECASTS OF LABOR REQUIREMENTS"3 During the defence production period which preceded World War II, the United States Employment Service initiated employers' forecasts of their labour requirements as one informational aspect of a man-power allocation and mobilization programme. Key establishments were asked to forecast their labour requirements by detailed occupation for a date thirty days in the future. As a result of investigation and experiment, forecast intervals were established at two and six months. Occupational data were omitted, although questions relating to specific critical occupations were attached to regular questionnaires4 from time to time. As a result of the "federalization" of the Employment Service early in the war, state agencies were

DESCRIPTION OF EPS AND OTHER SURVEYS

0

required to submit uniform forecast labour requirement reports for key firms. At the end of hostilities, the Employment Service was returned to the states. However, through the means of a uniform reporting programme, State employment security agencies were required to provide the Bureau with aggregated establishment forecasts of labour requirements in their labour market reports. (Report ES 219) This programme has gradually been contracted so that, at the present time, the Bureau of Employment Security (BES) only requires bimonthly labour market reports for 149 of the largest labour market areas. The state agencies are also required to report (Report 211) the detailed labour requirements of key individual establishments in each area, in one or two selected industries, each month. The aggregate forecast and actual data provided in the bimonthly reports (ES 219) of the state agencies are based on a sample of establishments in each industry in each area. The Bureau has established minimum standards of sample adequacy which must be met by each state agency. These are defined in terms of the proportion of total industry employment (actual) employed by the sample establishments. This is a type of cut-off sample. For example, the current actual aggregate employment of the sample establishments in the manufacturing industry division is required to constitute at least 65 per cent of total manufacturing employment in the area. The methods by which the state agencies gather the actual and forecast data from establishments are apparently diverse, both within and between agencies. Uniformity is restricted to the agency reports to the Bureau and the minimum sampling requirements. It is important to note that information is requested of employers not as a forecast of the level of employment which will prevail in the future, but as an "indication of the number of workers the establishment expects to employ to meet its production schedule, without regard to the availability or non-availability of qualified workers."5 On the basis of the bimonthly reports submitted to the Bureau for each area, the Bureau classified each area with respect to each of several criteria, including the "net change in required non-agricultural employment... 2 and 4 months hence." This classification appears under the section entitled "Employment Indicators," as the "Employers' Forecasts of Labor Requirements" in the bulletin, Labor Market and Employment Security. With a net change of +3.0 per cent or more in forecast labour requirements, the area is classified as type I, with respect to this variable. With net changes of +1.5 to +2.9 or +0.1 to +1.4 per cent the area is classified as type II or type III, respectively. Where no changes, or negative changes, in labour requirements are forecast, the area is classified as type IV. On the basis of the classifications assigned the area with respect to the forecast labour requirements, and of the classifications assigned to the area with respect to other variables, such as the labour force and the current unemployment situation, the area is classified with respect to the "adequacy of the labor supply. " As has been mentioned before, the Bureau has another report (ES 211) which the state agencies are required to submit. This report contains individual establishment data, including the establishment's forecast labour requirements. Each half year the Bureau provides the state agencies with a list of industries—one or two industries for each of the coming months. The schedule also indicates the minimum size of firm (employment size) for which returns are to be made. These reports, which are not made public, are used by the state agencies and the Bureau for industry labour market analysis and to provide the Bureau with data useful in improving the Bureau's industry relations.

6

THE EMPLOYMENT FORECAST SURVEY

The American and Canadian surveys are not entirely comparable. The BES survey seeks to assess the values of the causal factors (of which an employer's future labour requirements are but one) which determine the future labour market. The Canadian survey seeks to measure the employer's forecast of his own actions in the labour market, and by an aggregation procedure to come directly to an assessment of the future labour market. The Canadian survey is therefore considerably more mechanical than the BES survey, and thus lends itself to greater specificity (although not necessarily greater reliability). In one respect the BES survey provides more information than does the EPS survey. It is possible for the Bureau to assess (assuming that the forecasts of labour requirements are accurate) the extent to which employment levels will be, or are, limited by labour shortages. This additional information is not yielded by the Canadian survey, but is secured at some cost to the Bureau—a cost that may outweigh the advantages of the approach. Because the Bureau's sample establishments are asked to forecast the labour they would have to employ to meet their production schedules, rather than the labour they believe that they will employ, it is not always possible to compare the forecast labour requirements for a given date with the actual labour force of the establishment on that date, and to term the difference "forecast error. " Such a comparison would not be meaningful when there were labour shortages. This makes it difficult (impossible under some circumstances) to evaluate the adequacy of the establishment forecasts. The adequacy of the forecasts of the area labour market as a whole will depend not only upon the adequacy of the sample and the forecasts of the sample establishments but also upon the adequacy of the Bureau's evaluation and analysis of the various causal factors. There would seem to be no straightforward means by which the various sources of errors can be separated. The use of qualitative rather than quantitative estimates of the future labour market tends to avoid the implication of spurious reliability. On the other hand, it is also possible that useful information has been discarded in the process. Unfortunately, it is difficult to provide a statement of past reliability which can serve as a guide to the interpretation of the adequacy of current forecasts. The apparent multiplicity of approaches made to individual establishments by state agencies, in an effort to secure forecast data, would appear to contribute needlessly to response variability. A notable advantage of the American survey is the area and industry detail which can be provided, although the industry breakdown is not published in the Bureau's bulletin. 2. THE DUN AND BRADSTREET "SURVEYS OF BUSINESS EXPECTATIONS" As the result of a request by the Congressional Committee on the Economic Report, Dun and Bradstreet has undertaken, since 1947, a survey of the expectations of a sample of businessmen concerning sales, net profits, new orders, number of employees, level of inventories, and level of selling prices. The businesses whose executives are interviewed consist of manufacturers, wholesalers, and retailers on which Dun and Bradstreet prepares what are called "analytical reports. " These reports are made on concerns "rated for the most part at $ 75,000 or more in tangible net worth and in which there is an active credit interest. "6 Roughly 45,000 of these reports are revised by Dun and Bradstreet twice each year, although all of the firms or establish-

DESCRIPTION OF EPS AND OTHER SURVEYS

/

ments are not personally interviewed every six months, and only the interviewed executives are asked to make forecasts. During recent years Dun and Bradstreet officials state that between 1,000 and 1,300 establishments have reported their expectations each quarter, while a slightly higher percentage have reported actual movements (in the preceding quarter) of the above-mentioned variables. The interviews for the expectational data are conducted over a two-week period in the first month of each quarter. Respondents are asked to report whether the values of the variables in the past quarter were equal to, greater than, or less than their values in the same quarter in the previous year. Respondents also are asked to predict whether they believe that, two quarters hence, the values of these variables will be equal to, greater than, or less than their values the same date in the previous year. While the executives are asked to estimate year-to-year changes in percentage terms, as well as in terms of direction of change, both actual and forecast, these approximate percentages are not tabulated by Dun and Bradstreet on the grounds that they are too unreliable. Dun and Bradstreet reports thus provide two frequency distributions : the percentage of interviewed executives reporting "no change," "greater than," or "less than" changes in each variable over the same quarter in the previous year, and matching distributions which show, for each variable, the percentage of the respondents who predict "no change," "greater than," or "less than" changes in magnitude for the coming quarter, over the values of the variables on the same dates in the preceding year. Two general criticisms have been levelled at the form and method of the Dun and Bradstreet survey. It is claimed, on the one hand, that because Dun and Bradstreet is in the credit-rating business there will be some pressure on certain respondents to give optimistic forecasts. On the other hand, even though the population of firms from which the Dun and Bradstreet sample is chosen is large, and the choice may be "approximately random," the Dun and Bradstreet population is less than the total industry population. Dun and Bradstreet's population of establishments does not include the smaller establishments nor does it include the larger corporations. There is virtually no credit interest in the former, apparently, and credit information on the larger corporations is available from other sources. In some industries this restriction of the Dun and Bradstreet population may be extremely serious. For example, many establishments manufacturing clothing have a tangible net worth of less than $ 75,000, and hence could not be covered. At the other end of the scale, in industries made up principally of large corporations, for example the pulp and paper or the chemical industries, the Dun and Bradstreet population, much less its quarterly sample, presumably would inadequately represent the actual population of establishments. Some writers have suggested that the frequency distributions of the directions of change realized and expected should take into account, in some way, the size of the firms, so that the analyst could apply weights to the various changes. A comparison of the EPS and Dun and Bradstreet surveys suggests several important differences. An important advantage in the Dun and Bradstreet survey would seem to lie in the fact that the respondents are requested to make forecasts on several related variables. This gives the analyst a more complete picture of the "sentiment" of the respondents. Unfortunately, the manner of presentation of the data in the Dun and Bradstreet reports makes it impossible to relate the forecast changes in one vari-

8

THE EMPLOYMENT FORECAST SURVEY

able with the forecast changes in other variables, made by the same respondent. A second advantage which accrues to the Dun and Bradstreet survey would seem to derive from the fact that, because the information requested is not quantitative (or at least not exactly so) and is derived only from interviews with senior executives, there is more likelihood that the forecasts constitute the best estimates available within the firm. This advantage is gained at the expense of precise forecasts. It is impossible to determine, for example, the "threshold to change" in the minds of different executives. "No change" may mean very different things to different executives. A serious deficiency in the Dun and Bradstreet survey would seem to exist in the fact that it is difficult, if not impossible, to relate the forecasts to any given statistical series. The analyst is not able to determine, should the expectations of the respondents be correct, and should they represent adequately the forecasts of the Dun and Bradstreet population, what the aggregated forecasts imply with respect to changes in the relevant official statistical series. Also it is extremely difficult to determine what the reliability of the forecasts has been, for there is no bench mark against which the forecasts can be measured. The fact that the sample changes each quarter makes it virtually impossible to assess whether the differences in the published forecasts are caused by changes in business sentiment or by changes in the sample. 3. THE "BUSINESS TEST" OF THE IFO INSTITUTE? Beginning in 1950 the IFO Institute of Munich began a survey of the expectations of businessmen in Western Germany concerning several economic variables, including employment. This survey, which resembles the Dun and Bradstreet survey in several respects, has been copied in whole or in part by several other nations. The questions included in the questionnaire relate to both actual and expected directions of change in "specified economic variables that are involved in the decisionmaking activities of the firms." Executives are asked to forecast the direction of change in the specified variables from the time at which the forecasts are made to a point in time two months hence. There are certain variables, in which seasonal variations are large, for which the executives in the trade classifications are requested to forecast the year-to-year change at the two-month target date. While only the direction of change is forecast by respondents, the IFO Institute publishes the results of the surveys as weighted percentages of the number of forecasts which are "no change," "positive changes," and "negative changes" to the total number of forecasts which were made. Each executive forecast (one forecast per firm) is weighted by the number of employees of the firm ("turnover" in retail trade) and the weighted results are aggregated to give the relative importance of each of the three directions of change which have been forecast. The weighted aggregates in each direction are so computed as to yield three numbers, the sum of which equals 1.0. The IFO "Business Test" covers three groups of businessmen : industry, wholesale trade, and retail trade. The industrial executives are asked to indicate the actual direction of change which occurred over the past months in the following variables : production, raw material supply, orders received, orders from abroad, movements of stocks, personnel, daily working hours, selling prices, and payments received. The same respondents are asked to forecast the direction of change which they expect

DESCRIPTION OF EPS AND OTHER SURVEYS

9

will take place over the coming two months in production, personnel (employment), and selling prices. It may be noted once again that a survey requests forecasts by respondents on three variables at the same time, so that it is possible to publish the forecasts of each of these variables simultaneously and thereby provide analysis with more general view of executive sentiment. Like the Dun and Bradstreet survey, the IFO Institute has restricted the forecasts to the direction of change in a conscious attempt to make the forecasts sufficiently simple and direct so that executives will be able to complete the questionnaire without recourse to records (which frequently means that the forecasts ultimately are made by less qualified employees). The weighted aggregation procedure followed by the Institute also meets one criticism which has been levelled at the Dun and Bradstreet survey. It is possible, from the IFO forecasts, to evaluate the relative importance of the forecasts of a particular direction, when importance is defined to include frequency of forecasts of a particular direction and the size of the firms (employment) which forecast changes in that direction. However, the weighted percentages published by the IFO Institute are open to the same criticism as that suggested for the Dun and Bradstreet survey; it is not possible to determine the magnitude of the changes, positive or negative. The analyst also is unable to determine the range of actual changes which the respondents would classify as no change. In addition, it is not possible to relate, in a direct manner, the changes in the aggregated variables of the sample establishments and the changes in official statistical series, assuming that the sample is representative, and that the forecasts are correct. This is a serious defect, for it makes it extremely difficult to determine whether the firms, in aggregate, are forecasting anything of importance, or if they are forecasting it well. 4. THE CANADIAN DEPARTMENT OF LABOUR'S "EMPLOYMENT FORECAST SURVEY"® As the result of a small pilot study conducted by the National Selective Service,9 the Administrative Board of that agency requested, on February 14, 1944, that an inter-departmental committee be convened to consider the scope and technique of a study of the probable post-war employment situation in Canada. This committee recommended, on April 5, 1944, that a continuing survey of employer opinion about prospective post-war employment conditions be immediately undertaken by the Economics and Research Branch of the Department of Labour. This recommendation was promptly accepted by the departments concerned. The first survey, which was known as the "Post-War Employment Survey, First Phase," began in July, 1944, and was completed within a few months. Reports were received from about two thousand establishments which comprised, it was believed, all establishments in Canada employing two hundred or more persons, excluding construction firms, government offices, National Defence establishments, Crown companies, and hospitals. The sample establishments had a combined employment, as of June 1, 1944, of approximately 1,276,000 employees, or about 30 per cent of the labour force.10 The establishments included in the Post-War Employment Survey were interviewed and asked to answer a lengthy questionnaire. Although precise statistics are not available, the records suggest that an extremely high response rate was achieved. This high response rate may be attributable, at least in part, to the fact that the National Selective Service had had wide powers and author-

10

THE EMPLOYMENT FORECAST SURVEY

ity during the war. Employers may have considered this questionnaire a "command" rather than a "request" for information. Consistent with the original recommendation that the Post-War Employment Survey should be of a continuing character, the second phase of the survey was begun in January, 1945. A preliminary report based on the data collected was ready by May of that year. In the fall of 1944, and the early spring of 1945, consideration was given to the possibility of obtaining regular forecasts of employment from establishments. As early as September 11, 1944, those responsible for the survey investigated whether a continuing survey might be conducted by the Dominion Bureau of Statistics, as part of its regular monthly census of the actual employment of establishments employing fifteen or more persons. It was decided, however, that it would be undesirable to relate the two questionnaires. As the result of several inter-departmental meetings held in February through April of 1945,11 it was decided that the Economics and Research Branch of the Department of Labour should be responsible for conducting, through the regional and local offices of the Unemployment Insurance Commission, a regular survey of the employment forecasts of employers for dates three and four months in advance. It was generally agreed that the sample should cover: (1) the larger mining and manufacturing companies which depend, to a substantial degree, on export markets; (2) the larger companies engaged in the output of producer durable goods; (3) and some of the larger companies engaged in the consumer durable and semi-durable goods industries. A sample of about five hundred firms was considered sufficient to provide reliable estimates. The view was expressed that direct forecasts from these three groups of employers, whose collective level of operations constitutes an important dynamic force in the determination of the general level of Canadian economic activity, when used in conjunction with other types of forecasts, might prove to be an effective means of forecasting future conditions. It was felt that, if the Government was expected to take offsetting action to an anticipated decline, and to take it at the most effective time, adequate forecasts of national economic activity must be available. In an effort to determine the possible reaction of businessmen to the proposed survey, twelve executives were invited to attend a conference in Ottawa in June, 1945. The group generally agreed that forecasts of employment were feasible ; and they expressed the belief that industry would be willing to make them for the Government. It was suggested that it would be best to try to secure a three- and a six-month forecast each quarter. A pilot study was launched in November, 1945, when letters introducing the survey signed jointly by the Deputy Ministers of Labour and of Reconstruction were mailed to establishments in the Toronto area. Interviewing in that area began in December, 1945. Letters were sent to employers in the Montreal area in January, 1946, and interviewing began shortly thereafter. By January 26, 1946, the "cut-off" date, about ninety-eight returns had been received. Forecasts based on this pilot study were made for eight industry groups in the manufacturing category. Forecasts were made for April 1 and July 1, 1946, target dates. It should be noted that the quarterly target dates established in the first survey have prevailed since that time, with forecasts made for April 1, July 1, October 1, and January 1 of each year. In the second quarter survey for 1946, coverage was extended to the Prairie Provinces and British Columbia, and in the third quarter of 1946, the Maritime Provinces were covered.

DESCRIPTION OF EFS AND OTHER SURVEYS

11

Industrial Classification

For reasons which are no longer clear, the classification used in the survey has never coincided closely with the original specifications. In the EFS reports for the year 1946 only some manufacturing industries were included, and these were grouped into "industries ancillary to consumer non-durables," and "producers goods industries." In each year, 1947-51, substantial changes were made in the classification of the manufacturing industries. Moreover, the following non-manufacturing industries were successively added to the survey: mining, logging, electric power, communications, and department stores and mail order houses. The construction, service, transportation, and trade industries (with the exception mentioned above) were excluded from the survey, on the grounds that the sampling problems were insuperable, given the existing budget. The industrial classification used in current EFS reports was adopted, with a few important exceptions, in late 1951.12 It is based, with but one exception,13 upon the industrial classification used in the compilation14 of the employment indexes published in Employment and Payrolls. The Employment and Payrolls classification is in turn derived, with a few modifications, from the Canadian Standard Industrial Classification. In the balance of the study, intensive analysis of the EFS industry predictions generally is restricted to the period 1952-5. The choice of the earlier date was dictated by the difficulties inherent in the determination of comparable actual and predicted employment indexes prior to that date, because of classificatory changes. Because the industry predictions were abandoned at the end of 1956 substantially all of the comparable data is included. Sampling Procedures

Unfortunately, there is no clear evidence available with respect to the method used to select the EFS sample. In memoranda written before the sample was selected, it was suggested that the sample should consist of some or all of those establishments which were "large, " could reliably predict their own employment, and were subject to autonomous changes in their employment levels. On the other hand, it is stated in the Introduction to the first Employment Forecast Survey Report that the sample consisted of so-called leading establishments. Although not specifically defined those establishments in each homogeneous (with respect to product) group which, it was claimed, tended to influence the behaviour of all of the other establishments in the group were designated as "leaders." Apparently an attempt was made to include them in the sample. The contemporary recollection of two persons who were engaged in the selection of the EFS sample nearly ten years ago suggests that the "leader" concept was, in practice, a "size" concept. They believe that an attempt was made simply to include the largest establishments in each industry. If this latter view is correct, it would appear that the EFS sample was selected on the basis of a crude cut-off sample design.15 The size composition of the present EFS sample for manufacturing, which is described in detail in Appendix D, strongly supports the opinion that a cut-ofF sample design was either implicitly or explicitly adopted. Although the EFS sample of manufacturing establishments included only about six per cent of those reporting

12

THE EMPLOYMENT FORECAST SURVEY

to Employment and Payrolls, it included about 250 establishments which employed 500 workers or more as of November 1, 1953. This was over 60 per cent of the population of manufacturing establishments in this size category.16 TABLE I NUMBER OF SAMPLE ESTABLISHMENTS WHICH SUBMITTED FORECASTS^") FOR THE APRIL 1 TARGET DATE, SELECTED YEARS, 1947-55 Year

Manufacturing establishments

1947 1949 1951 1953 1955

542 639 591 566 588

Non-manufacturing establishments 140 139 157 150 150

Total

682 778 748 716 738

SOURCE : Quarterly tabulations made by the Employment Forecast Survey Unit, Employment and Labour Market Section, Economics and Research Branch, Department of Labour, Ottawa. M Three- and/or six-month forecasts.

As shown in Table I, the majority of the establishments which were to comprise the EFS sample had been added by April 1, 1947. By April 1, 1949, when the peak sample size was attained, the total sample consisted of nearly 780 establishments of which over 80 per cent had been drawn from the manufacturing industries. A few additional establishments were brought into the sample in 1955 (between 40 and 50), but the majority of the changes in the sample after 1949 were attributable to mergers of existing establishments, the opening of new establishments by companies covered by the sample, changes in the industrial classification of sample establishments, establishments going out of business, and the non-co-operation of establishments.17 The proportion of the total employment of the sample establishments to the total employment of the comparable industries, as reported to DBS, has remained comparatively stable over the time period considered. The actual employment of the respondent establishments drawn from the manufacturing industries has represented between 36 and 45 per cent of the total employment in the manufacturing industries, as Table II shows. In the selected non-manufacturing industries18 even greater coverage was attained, ranging between 60 and 80 per cent. Unfortunately, precise information is not available concerning the numbers of firms and/or establishments approached by the EFS representatives which refused to co-operate in the survey. The comments of those who were involved in the sampling procedure suggest that the percentage of establishments that refused to co-operate in any way was a small fraction of those approached ; this fraction has been roughly estimated as no greater than 10 per cent. Survey Procedure

Each establishment included within the Post-War Employment Surveys was interviewed by a representative of one of the regional offices of the Unemployment Insurance Commission. It was originally intended that this same procedure would

DESCRIPTION OF EPS AND OTHER SURVEYS

13

TABLE II EMPLOYMENT OF SAMPLE ESTABLISHMENTS AS A PERCENTAGE OF THE EMPLOYMENT OF THE POPULATION OF ESTABLISHMENTS/") SELECTED YEARS, 1947-55 Year 1947W 1949W 1951 1953 1955

Manufacturing establishments

36.8 45.9 38.5 39.0 36.7

Non-manufacturing establishments 74.7W 79.4W 78.5W 77.9W 59. 1W

Total 43.9W 51.8W 46.5W 45.8W 41. 8() It was at the discretion of the EFS administrator which of the two procedures was used to project the EFS index for each industry at each target date. When an index projected by one procedure seemed to be unreasonable, in the judgment of the administrator, the index was recomputed, using the alternate procedure (but the same establishment forecasts).5 The results indicate that the administrator's judgment generally was good. These choices almost certainly reduced the errors of the EFS predictions, and thereby met the needs of the recipients of the EFS Report more adequately. However, from the point of view of this study, the fact that an element of subjective judgment entered into the sampling and forecast errors of the indexes is a complication. The purpose of the following chapters is to evaluate the feasibility of reducing the errors of future EFS predictions. Obviously, the subjective judgment of the administrator is not susceptible to such an assessment. The Derivation of Estimates The two obstacles to the precise measurement of the sampling and forecast errors of EFS indexes have been described. The first of these difficulties appears to be of little empirical importance. The second obstacle is more serious. Indeed, it precludes the precise determination of the extent to which the errors of EFS indexes can be attributed to the forecast and sampling problems respectively. The inability to measure the forecast and sampling error components of the EFS indexes is a distinct limitation in this study. Although the sampling and forecast errors of EFS indexes cannot be determined, statistics can be developed which are capable of serving many of the same purposes. EFS predictions can be recomputed in a manner which avoids both of the obstacles which have been described. The forecast and sampling errors of these revised EFS predictions can then be measured. As a computational convenience, and as a method of providing data in the form most suitable for this analysis, the following technique has been adopted for the measurement of the forecast and sampling errors of the revised predictions. It should be noted that this technique is comparable to the derivation of EFS predictions by the consistent use of the year-to-year procedure. (a) At each target date the sample data includes only those establishments which submitted both three- and six-month forecasts and their actual employment on the base (t—3 and t—2), and target dates (i + 1 and i+2). This circumvents the first difficulty discussed above. (e) The actual and forecast percentage changes in the employment of the sample establishments, as defined in (a), have been computed for each industry and target date in the period from April 1, 1953, to July 1, 1955.6 All changes have been computed over year-to-year time intervals. For industries and groups in which cumulative EFS predictions were derived, all sample changes have been weighted in accordance with the DBS employment of the components on the same target date in the

SOURCES OF PREDICTIVE ERRORS

47

previous year. This weighting procedure is comparable with that used in the derivation of EPS indexes by the year-to-year procedure. (c) The year-to-year changes in DBS employment indexes for each industry have been computed for the same target dates. (d) The difference, measured in percentage points, between the forecast change described in (2>) and the comparable change in the DBS indexes described in (c) is defined as the total error of the revised EPS prediction. (e) The difference, measured in percentage points, between the forecast and actual changes described in (b), compared over the same time interval, is defined as the forecast error of the revised EPS prediction. (/) The difference, measured in percentage points, between the actual change described in (¿>) and the DBS change described in (c), compared over the same time interval, is defined as the sampling error of the revised EPS prediction. Before proceeding to the analysis of the forecast and sampling problems, on the basis of the forecast and sampling errors derived by the procedure which has just been described, it would seem desirable to comment briefly on the significance which should be attached to the measurement of these errors from the revised, rather than the original, EPS predictions. Perhaps the comparison of the total errors of the revised and original EPS predictions indicates the importance of the distinction most clearly. The relevant data are provided in Table X. The mean (absolute) total errors of the original and revised EPS predictions are shown for each industry, covering the target dates in the period from April 1, 1953, to July 1, 1955. It is immediately apparent from the information supplied in the table that the mean errors of the revised predictions generally were larger, and for some industries much larger, than those of the original predictions. There appear to be three possible explanations for these disparities. The exclusion of the forecasts at each target date of those establishments which failed to report their actual employment on that date has increased the total error of the revised predictions. While a small portion of the differences between the two errors may be explained in this way, the evidence cited above strongly suggests that this factor is of little or no importance. The number of establishment forecasts excluded at each target date has been insignificant. There is a special explanation for the divergence between the total errors of the revised and original EPS predictions for the vehicles, aircraft, and railway rolling stock industries, and the two industry groups in which they are components. For these three industries, and these industries only, the administrator of the survey projected the indexes on the basis of data in addition to, or other than, the forecasts reported by establishments. The administrator was aware of, but unable to resolve, the severe sampling and other problems in these industries. The exceptional procedure was adopted as a method of providing reasonably reliable EPS indexes.7 All of the revised predictions were consistently derived by a technique which is comparable to the year-to-year procedure. This approach may overstate the "true" sampling errors for some industries (in the sense that estimates consistently derived by one procedure may be more reliable than estimates consistently derived by the alternate procedure).8 The EPS administrator, by the selection of the EPS index which seemed most "reasonable, " probably chose, indirectly, the optimum procedure with considerable frequency.9 Of these three explanations, it would appear that only the latter need be taken into

48

THE EMPLOYMENT FORECAST SURVEY

TABLE X MEAN (ABSOLUTE) ERRORS(") OF THE ORIGINAL AND REVISEDÍ*) EFS PREDICTIONS, MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 1955 (P.C. POINTS)

Industry Manufacturing Basic materials Chemicals Non-ferrous metals Non-metallic minerals Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood Consumer finished goods Clothing Electrical apparatus Food and beverages Furniture Vehicles Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding and repairing

Three-month forecasts Revised Original EFS EFS

1.72 1.37 1.78 1.95 3.49 1.95 4.47 1.82 - 3.07 3.64 4.45 1.65 4.56 3.35 1.79 1.91 4.08 1.65 13.07 4.47 2.50 2.26 6.57 3.92

2.64 1.81 4.62 3.59 4.25 2.09 5.58 2.58 5.52 4.95 5.71 1.98 6.89 2.83 1.85 3.76 7.16 5.12 10.70 14.40 1.99 2.80 18.93 7.88

Six-month forecasts Original Revised EFS EFS

2.61 2.37 1.89 3.32 2.28 1.91 7.96 1.86 3.15 6.17 5.82 2.70 5.28 5.17 1.42 3.12 8.60 2.98 12.08 5.81 3.10 2.81 9.53 5.12

2.82 2.23 3.77 3.63 3.78 1.59 8.86 2.58 5.29 8.37 6.15 2.88 7.36 3.47 1.86 3.26 10.13 5.31 10.81 19.74 3.04 5.22 17.37 6.53

SOURCE : The errors of the original EFS predictions were computed from the EFS indexes published in the Employment Forecast Survey Reports. Fourth Quarter, 1952—Second Quarter, 1955; and the DBS employment indexes given in Employment and Payrolls, supplemented by special tabulations provided by the Labour and Prices Division, DBS. The errors of the revised predictions were computed from the actual and forecast employment data of the sample establishments, as reported to the Economics and Research Branch of the Department of Labour. W The mean (absolute) differences, measured in percentage points, between the actual and predicted year-to-year percentage changes at target dates. (*) For a detailed description of the computational procedures by which the revised predictions were derived, see text.

account in the interpretation of the sampling and forecast error statistics. As a reminder of the fact that the sampling error which has been measured may overstate the "true" sampling error, this statistic is termed the "estimated sampling error," in the following discussion.

CHAPTER SIX

THE S A M P L I N G PROBLEM A CONCEPTUAL FRAMEWORK, within which the determinants of the errors of EFS predictions may be considered, was presented in chapter v. The sampling condition is one of the two principal determinants specified. This chapter attempts to examine the extent to which the sampling condition was met, to analyse the factors which have affected the realization of the sampling condition, and to assess the feasibility of resolving it. 1. A DESCRIPTION OF THE SAMPLING ERRORS It has been possible to measure the magnitude of the differences between the changes in the actual employment of certain sample establishments and the changes in the DBS employment indexes for the same industries, compared over the same time intervals. These differences, as discussed in the previous chapter, have been defined as estimated sampling errors. As an expository convenience, the initials ESE will be used to signify estimated sampling errors, on occasion. The data provided in Table XI summarize the estimated sampling errors by industry for the target dates in the period April 1, 1953, to July 1, 1955. The mean ESE (both absolute and signs into account) is shown for each industry and group, together with the frequency of ESE's less than 1.0 and greater than 5.0 percentage points. The extremely large, inter-industry variations in mean ESE can be seen from the statistics provided in the table. For some of the industries, such as fabricated iron and steel and paper products, the mean ESE is close to 1.0 percentage point. At the other extreme are the industries where there have been extremely large sampling errors. The railway rolling stock, aircraft, and vehicles industries fall into this category. The mean ESE for these industries ranges between eight and sixteen percentage points. The wide inter-industry variations in estimated sampling error cannot be attributed to a few extreme values. This is shown by the frequencies of sampling errors less than 1.0 and greater than 5.0 percentage points. The industries with the largest ESE's also had a high frequency of estimated sampling errors which were greater than 5.0 percentage points, and a low frequency of errors which were less than 1.0 percentage points. The converse was true of the industries with the smallest mean errors. The significance of sampling errors of this magnitude can readily be appreciated when the mean ESE for each industry is compared with the mean change in the corresponding DBS indexes. The ratio of the former to the latter is shown in Table XII. There were seven industries in which this ratio was greater than 1.0, which implies that the mean error was greater than the mean actual change. The food and beverage industry had the largest relative errors, when measured in this manner. The

50

THE EMPLOYMENT FORECAST SURVEY

TABLE XI ESTIMATED SAMPLING ERRORS^), MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 19550 Percentage of target dates with errors of

Mean (absolute) error (p.c. points)

Mean error (p.c. points)

Less than i.o percentage point

Manufacturing

1.49

—0.71

30

0

Basic materials Chemicals Non-ferrous metal Non-metallic minerals Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood products

1.64 4.85 3.24 4.83 1.14 1.87 2.31 4.13 4.56 5.13

+ 0.95 + 4.85 + 3.13 + 1.09 + 0.89 —0.81 + 0.01 + 2.88 —0.32 —1.94

30 10 30 10 60 20 10 10 30 30

0 50 40 50 0 0 0 30 30 40

Consumer finished goods Clothing Electrical apparatus and supplies Foods and Beverages Furniture Vehicles

1.45 6.94 4.21 2.41 4.81 8.41

+ 0.22 —1.37 + 2.99 —1.05 + 1.24 —2.28

60 10 0 30 0 0

0 80 20 20 50 70

3.91 6.03 10.93 1.08 3.01 16.09 5.61

+ 2.36 —1.45 + 9.62 —0.69 —0.12 + 5.81 —2.63

10 10 0 40 0 0 30

50 50 60 0 20 70 40

Industry

Producers finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding and repairing

5.0 percentage points or more

SOURCE: Table X. (a) Estimated sampling error is defined as the number of percentage points difference between the year-to-year change in the aggregate actual employment of the EFS sample establishments and the year-to-year percentage change in the corresponding DBS employment indexes. (6) The estimated sampling errors at ten target dates are taken into account.

mean year-to-year percentage change in the DBS employment indexes for this industry was 1.02 per cent for the changes considered at each of the target dates April 1, 1953, to July 1, 1955. The mean difference between the DBS changes and the corresponding changes in the actual employment of the sample was 2.41 percentage points. The mean sampling error was well over twice as large as the mean change in the DBS indexes. There can be little doubt that the sample provided unreliable estimates of the magnitudes of the changes in the DBS indexes, for many of the industries and groups considered. For only six of the twenty-four industries and groups were the mean errors less than one-half of the mean changes in the DBS indexes. Even for manufacturing and the three major groups, the mean errors were roughly one-half of the mean changes which the sample purported to estimate. Concern rests not solely with the sample estimates of the magnitudes of the changes in the DBS indexes. The reliability of the sample estimates of the direction of change are probably even more important for the present study. The data shown in Table

THE SAMPLING PROBLEM

51

TABLE XII THE RELATIONSHIP BETWEEN THE ACTUAL CHANGES IN DBS EMPLOYMENT INDEXES^) AND THE ESTIMATED EFS SAMPLING ERRORS, W MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 1955 Industry Manufacturing Basic materials Chemical products Non-ferrous metal products Non-metallic mineral products Paper products Primary iron and steel Products of petroleum and coal Rubber products Textile products Wood products

Mean(c) change DBS employment indexes (per cent)

Mean(c) estimated sampling error (p.c. points)

Ratio of mean ESE to mean change DBS employment indexes

3.52 3.18 2.64 4.63 2.07 3.15 10.81 1.91 7.14 8.56 4.08

1.49 1.64 4.85 3.24 4.83 1.14 1.87 2.31 4.13 4.56 5.13

0.42 0.52 1.83 0.70 2.33 0.36 0.17 1.21 0.58 0.53 1.26

3.68 6.19

1.45 6.94

0.39 1.12

Consumer finished goods Clothing Electrical apparatus and supplies (except heavy electrical machinery) Food and beverages Furniture, household machinery and apparatus Vehicles

8.36 1.02

4.21 2.41

0.51 2.36

5.68 10.35

4.81 8.41

0.84 0.81

Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial, office and business machinery Railway rolling stock Shipbuilding and repairing

6.87 17.14 16.39 5.17 3.50 11.12 10.74

3.91 6.03 10.93 1.08 3.01 16.09 5.61

0.57 0.35 0.67 0.21 0.86 1.44 0.52

SOURCE: As for Table X. (a) The year-to-year percentage changes in DBS employment indexes at the quarterly target dates. (è) Sampling errors estimated from revised EFS predictions. The estimated sampling error is defined as the number of percentage points difference between the year-to-year percentage changes in the aggregate actual employment of the EFS sample establishments and the year-to-year percentage changes in the appropriate DBS employment indexes at the same target date. (c) Mean absolute.

XIII describe the reliability of the directional estimates derived from the actual employment of the sample establishments.1 In the period for which data are available, there were 192 sample estimates of the direction of non-seasonal changes over six-month intervals, and 168 sample estimates of the changes over nine-month intervals, when all industries and groups are considered. Nearly 30 per cent of these estimates were incorrect. For twelve of the twentyfour industries and groups over one-third of the sample estimates of the direction of change over six-month intervals was incorrect. Similarly, for nine of the industries and groups over one-third of the directional estima tes of changes over nine-month intervals was incorrect. Only the directional predictions made for the Consumer Finished Goods group and the furniture industry were correct in every instance. When it is realized that there was only one turning point in the period, the directional estimates derived from the actual employment of the sample appear to have been even less reliable than the EFS directional predictions. It has been found that the sample estimates of the direction and magnitude of the

52

THE EMPLOYMENT FORECAST SURVEY

TABLE XIII INCORRECT ESTIMATES^) OF THE DIRECTION OF THE NON-SEASONAL CHANGES IN DBS EMPLOYMENT INDEXES DERIVED FROM THE CHANGES IN THE ACTUAL EMPLOYMENT OF THE SAMPLE ESTABLISHMENTS, MANUFACTURING INDUSTRIES APRIL 1, 1953—JULY 1, 1955

Industry Manufacturing Basic materials Chemicals Non-ferrous metals Non-metallic minerals Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood Consumer finished goods Clothing Electrical Food and beverages Furniture Vehicles . Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding TOTAL

Over sixmonth intervalsW Number M Per cent 1

4 4 3 0 3 1 4 4 3 4 0 3 2 5 0 1 1 1 1 1 3

4 2 55

12.5 50.0 50.0 37.5

0.0

37.5 12.5 50.0 50.0 37.5 50.0 0.0

37.5 25.0 62.5

0.0

12.5 12.5 12.5 12.5 12.5 37.5 50.0 25.0 28.6

Over ninemonth intervals W Number (rf) Per cent 0 2 4 3 2 1 0 5 3 1 4 0 4 1 4 1 0 0 3 1 0 3

4 0 46

0.0

28.6 57.1 42.9 28.6 14.3 0.0

71.4 42.9 14.3 57.1 0.0

57.1 14.3 57.1 14.3

0.0 0.0

42.9 14.3

0.0

42.9 57.1 0.0 27.4

SOURCE : As for Table X. M For a definition of an incorrect estimate of the direction of a non-seasonal change, see text. W The time intervals over which the non-seasonal changes are computed for EFS predictions based on the three- and six-month forecasts of establishments, respectively. See supra, P-32. M A total of eight estimates of the changes over six-month intervals were taken into account in the period April 1, 1953—July 1, 1955. () the magnitudes of the differences between the changes in the employment of the sample and nonsample establishments. The greater the magnitudes of the differences in the sample and non-sample changes, the larger must be the coverage of the sample if the direction of change estimated from the sample is to be the same as the direction of change of the employment of the population.

58

THE EMPLOYMENT FORECAST SURVEY

Because the present EFS sample is based on a cut-off sample design, and because this design has certain advantages, it would seem desirable to attempt to assess its adequacy within this framework. Ideally, such an assessment would be based on measurements of the differences between the changes in the employment of the sample and non-sample establishments. With this information it would be possible to determine the sample coverage that would be necessary to ensure7 correct sample estimates of the direction of changes in the employment of the population. Unfortunately, the data are not available from which such differences could be estimated. TABLE XIV RELATIONSHIP BETWEEN THE COVERAGE OF THE EPS SAMPLE AND THE RELIABILITY OF THE SAMPLE ESTIMATES, MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 1955 EFS sampling fractions

Industry

Per cent of employmerit in EFS sample

Per cent of establishments in EFS sample

Ratio of mean sampling error to mean actual change in DBS employment index

The number of incorrect estimates of the direction of nonseasonal changes Threemonth intervals

Sixmonth intervals

Manufacturing

39.0

5.9

0.42

1

0

Basic materials Chemicals Non-ferrous metals Non-metallic minerals Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood

(a) 32.5 59.5 36.6 51.0 90.2 61.4 68.8 40.8 20.2

(a) 6.8 6.4 8.9 17.0 33.3 29.5 19.3 7.7 3.0

0.52 1.83 0.70 2.33 0.36 0.17 1.21 0.58 0.53 1.26

4 4 3 0 3 1 4 4 3 4

2 4 3 2 1 0 5 3 1 4

Consumer finished goods Clothing Electrical apparatus Food and beverages Furniture Vehicles

(a) 9.3 54.2 34.4 (a) 71.5

(a) 1.5 9.6 8.1 (a) 8.7

0.39 1.12 0.51 2.36 0.84 0.81

0 3 2 5 0 1

0 4 1 4 1 0

Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding

(a) 72.1 69.5 39.1 (a) 18.9 59.9

(a) 15.6 7.9 7.5 (a) 17.5 10.9

0.57 0.36 0.67 0.21 0.86 1.44 0.52

1 1 1 1 3 4 2

0 3 1 0 3 4 0

SOURCE: Appendix D. Tables XII and XIII. (a) Not available.

However, some conception of the magnitudes involved can be gained by a comparison of the EFS sampling fractions (employment) for individual industries, and the reliability of the sample estimates of the changes (both direction and magnitude) in DBS indexes. The relevant statistics are summarized in Table XIV. These data show that the larger the proportion of the population of employees covered by the sample establishments the more reliable the sample estimates tended to be. At one extreme is the primary iron and steel industry. The sample drawn from the establishments in this industry employs about 90 per cent of the population of em-

THE SAMPLING PROBLEM

59

ployees. All of the sample estimates of the direction of the non-seasonal changes in the DBS employment indexes for this industry were correct with but one exception. Relative to the magnitudes of the actual changes, the sampling errors for this industry were smaller, on the average, than those of any other industry. The sampling fraction of the clothing industry is at the lower extreme, with only about 9 per cent of the total population of employees covered by the sample establishments. In this case there were three incorrect estimates of the direction of change, and the sampling errors were greater, on the average, than the changes in the DBS employment indexes which the sample purported to estimate. Although there is a strong indication that the sample estimates were more reliable, the larger the sampling fraction, the data provided in the table do not show decisively what the sampling fraction should have been to secure "sufficiently reliable" estimates. Several of the industry samples (for example, non-ferrous metal products, paper products, and rubber products) included approximately 50 to 60 per cent of the population of employees, but provided estimates of the direction of change which were unreliable. Where the sample included about 70 per cent of total employment, such as the vehicles and agricultural implements industries, the reliability of the estimates of the direction of change was higher, but the reliability of the estimates of the magnitude of change was lower, at least in the former industry. As described in chapter n, section 1, the United States Bureau of Employment Security (BES) uses a cut-off sample design in its survey of employers' intentions. The specifications for that survey require that the total employment of the sample respondents in each state should constitute a specified proportion of the total employment of the industry from which they are drawn. For manufacturing, each state sample is required to cover 65 per cent of total industry employment.8 From discussions with the staff of the BES,9 it would appear that this sampling fraction requirement may have been adopted largely on administrative grounds rather than from a detailed statistical appraisal of the sampling problem.10 The data given in Appendix B clearly show that, for a large number of industries, even the imposition of the BES sampling standard would have required substantial increases in the coverage of the EFS sample. For only five industries, of those under consideration, did the EFS sample include 65 per cent or more of the total employment of the industry. From data given in Appendix B, it is possible to make rough estimates of the numbers of additional employees and establishments the imposition of the BES standard would have entailed. As an example, manufacturing as a whole may be considered. To achieve the 65 per cent BES standard, the EFS sample would have been required to include about 765,000 workers.11 The sample accounted for about 460,000 workers. Ignoring the problem of establishment co-operation, and assuming that the ideal cut-off sample could have been implemented, the EFS sample would have had to consist of all manufacturing establishments employing 200 workers or more. This would have given a total EFS sample of 1,180 establishments. The actual sample included about 550 establishments in manufacturing. Thus, even under a minimum sampling requirement, and optimum response conditions, an increase in the EFS sample of at least 100 per cent would have been required. If the BES standard were applied to each of the individual industries within manufacturing for which EFS predictions have been derived, the necessary increase in sample size would have been much larger. It is estimated that to achieve the 65 per cent cut-off standard for the

60

THE EMPLOYMENT FORECAST SURVEY

clothing industry, for example, would have necessitated the inclusion of over three hundred establishments in the EPS sample ; the present sample includes about twenty- four. The foregoing evaluation has strongly suggested that the present sample is inadequate as a cut-off sample, with the possible exception of the samples drawn from a few industries with high establishment employment concentration. There can be little doubt that, to secure more reliable estimates for manufacturing as a whole, the EPS sample should be at least doubled in size. To secure reliable estimates for each of the covered industries, an even larger increase in the size of the sample probably would be required. The preceding analysis was restricted to the consideration of a cut-off sample design. In the following section an attempt is made to evaluate the EPS sampling problem in terms of a more elaborate design, which is, in part, a cut-off sample and, in part, a random sample. 3. THE RESOLUTION OF THE SAMPLING PROBLEM This section tentatively proposes an alternative sampling design which might be adopted in order to resolve the sampling problem. On the basis of this alternative sampling design, rough estimates are provided of the changes in the size and allocation of the present sample which might be required. In order to achieve a separation of the sampling and forecast problems, it is assumed that each establishment in the population makes completely accurate employment forecasts in a form, and at a time, which completely coincide with the EPS requirements. To avoid problems of establishment co-operation, it also is assumed that, upon request, each establishment would promptly submit its perfect employment forecast to the EPS.12 There are two basic approaches which can be made to any sampling problem. The required reliability of estimates can be established (acceptable sampling variation) ; then an attempt can be made to determine the size of sample which would be necessary to achieve this standard. Alternatively, an attempt can be made to estimate, for a given sample size, the expected reliability of the estimates which could be derived. Because the budget limit (and hence sample size limitation) of the EPS survey is not known, the former approach is adopted for the purpose of this study. It is assumed that the EPS estimates of the DBS indexes should not differ from those indexes by more than one per cent, two-thirds of the time.13 It is also assumed that this standard applies to manufacturing as a whole, and the three groups within manufacturing. It is extremely difficult to assemble the data needed to make precise estimates of the reliability of the EPS estimates which can be derived from any orthodox sampling design. This difficulty springs largely from the fact that EPS indexes are totals projected by means of ratio estimates. The estimation of the reliability of such totals requires data which are completely inaccessible. For example, it is necessary to know the coefficient of variation of the employment of the population of establishments at the base and target dates (the numerator and denominator of the ratio). It is also necessary to know the coefficient of correlation between the employment on the base and target dates for all population establishments. With some 10,000 establishments in the manufacturing population, and with possible differences in these values over time, the magnitude of the task becomes staggering. Fortunately, there is some information available which can serve as a guide in estimating these variables.

THE SAMPLING PROBLEM

61

The Dominion Bureau of Statistics has undertaken to derive, for government use only, preliminary estimates of certain employment indexes which are published in Employment and Payrolls. The preliminary estimates are secured from a sample of establishments selected from the population of establishments which report to Employment and Payrolls. Manufacturing employment is one of the series for which preliminary estimates are made. Estimates are not obtained for any of the components of manufacturing. The DBS estimating procedure is virtually identical with the EPS projection procedure. The DBS employment index (based on a population tabulation for the previous month) is projected one month by the ratio of the actual employment of the sample establishments in the current month to the actual employment of the same establishments in the preceding month. With the exception of the fact that the EFS survey involves projections of the DBS indexes over longer time intervals, and EFS sub-group estimates are derived, the DBS and EFS sampling problems are identical. The close similarity of the sampling problems, coupled with a knowledge that the DBS survey has been carefully designed and tested, make it possible to adopt the DBS design as an alternative to the present EFS sample. This section attempts to describe the size and allocation of the EFS sample which would be necessary to attain the required reliability of estimates should the EFS survey implement the DBS design. The DBS sample design calls for two strata.14 The first stratum includes with certainty all of the largest establishments ; it is, in a sense, a cut-off sample. The second stratum consists of a random sample of about 4.0 per cent of the balance of the population of establishments. On November 1, 1953, the date for which comparable data are available, the DBS manufacturing sample was composed of 518 establishments. This sample included about 52 per cent of the population of establishments which employed five hundred workers or more; about 4 per cent of those which employed 200-499 workers ; and roughly 4 per cent of the smaller establishments in the population. At the same date the EFS sample included 561 establishments but, as we have seen, the distribution of the EFS sample among the size groups was fundamentally different. To approximate the DBS sampling fractions, the EFS sample would have to reduce slightly the coverage in the five hundred and over, and the 100-199 employee size classes, substantially reduce the sampling fraction within the 200-499 class, and substantially increase the sampling fractions in the smaller size classes. Granted that the imposition of the DBS sample design would involve a substantial reallocation of the EFS sample among the size classes, it is still necessary to attempt to determine the approximate number of establishments the EFS sample should include. By taking advantage of the information assembled by the DBS, and with the aid of certain simplifying assumptions, it is possible to arrive at a rough, approximate estimate of the necessary EFS sample size. The relevant data, and the specific assumptions upon which the sample size estimates are based, can be found in Appendix K. This analysis suggests that to secure EFS estimates for manufacturing only, with a coefficient of variation of 1.0 per cent, the EFS sample probably should include about 1,360 establishments. This estimate may be compared with the estimated sample size required by the implementation of a cut-off sample, using the BES sample requirement. The latter estimate indicated that a sample of approximately 1,180 establishments would be required. Considering the crude methods which have been used, these estimates are surprisingly close.

62

THE EMPLOYMENT FORECAST SURVEY

The same data and assumptions can be used to derive very rough estimates of the EPS sample sizes which would be necessary to attain the same reliability of the EFS indexes for the three economic groups within manufacturing. The analysis suggests that the following sample sizes would be necessary for each of these groups. (The total is not exact because of rounding.) Required sample size (establishments)

Industry group

Basic materials Consumer finished goods Producer finished goods

1160 1140 740

TOTAL

3042 4. SUMMARY AND CONCLUSION

The EFS sampling problem has been examined from three points of view. First, the estimated sampling error statistics were used to describe the reliability of the sample estimates of the direction and magnitude of the non-seasonal changes in DBS employment indexes. Secondly, the factors which determine the adequacy of a cut-off sample, of which the present EFS sample is a crude example, were discussed in section 2 ; and an attempt was made to assess the adequacy of the present sample in terms of a cut-off sample design. Finally, an alternative sampling design was suggested, and the size and allocation of an appropriate EFS sample estimated. Although each of these investigations was not definitive because sufficient information is not available, together they suggest a decisive conclusion. The studies of the cut-off and stratified (partially random) samples which would be required to meet the needs of the EFS survey both indicated that the present sample must be most inadequate. Moreover, the sampling errors of the estimates derived from the present sample confirm this finding. To achieve estimates for manufacturing, and the three groups, which are reasonably reliable the previous analysis suggests that extremely large increases in the coverage of the EFS sample would be required. This conclusion, in and of itself, is favourable to the EFS survey. If the sampling problem has been onerous, and the sample estimates unreliable, it is possible that a substantial portion of the errors of the EFS predictions can be attributed to this source. Should this be the case, it may be possible to reduce the errors of the EFS predictions by improving the EFS sample. As suggested, there is no reason why the present sample cannot be improved without limit; given establishment co-operation and a sufficiently large budget. In the following chapter a comparable analysis of the EFS forecast problem is provided.

CHAPTER SEVEN

THE FORECAST P R O B L E M THE REDUCTION of sampling error, as the foregoing chapter indicated, is a complex problem. It is known, however, that all sampling problems are ultimately solvable, given the co-operation of respondents and sufficient resources (the census solution is always available, in extremis). The forecast problem is even more complex, or perhaps it would be more accurate to say, more unsatisfactory. There is no known theoretical framework within which the forecast problem can be approached. There are a multitude of possible reasons why forecasts may be in error, and a multitude of possible corrections or adjustments which might be applied in an effort to determine if their errors might be reduced. In this chapter the forecast errors of the revised EFS predictions are described, and an attempt is made both to determine the sources of these errors, and to evaluate whether they could have been corrected. It must be emphasized, however, that in this chapter the sample forecast errors are traced back to the establishments but not within the establishments. The evaluation of the possible corrections of previous forecasts, to which this chapter is restricted, endeavours to be suggestive rather than definitive. 1. DESCRIPTION AND ANALYSIS OF THE EFS SAMPLE FORECAST ERRORS Before an evaluation can be made of the feasibility of reducing the errors of future sample predictions, it is necessary to know the characteristics of the errors of previous sample forecasts. As defined previously (chapter v), sample forecast errors are the differences in percentage point between the year-to-year forecast and actual percentage changes in the employment of the sample establishments for a particular target date. Given the resolution of the sampling problem, it is the character of these differences which determines the character of the errors in the EFS predictions. The sample forecast errors (SFE) have been measured by industry and group, by length of forecast, and by target date for the period April 1, 1953, to July 1, 1955. These statistics are summarized in Table XV. For some of the industries, the mean (absolute) SFE are extremely large for both the three- and six-month forecasts; the agricultural implements, railway rolling stock, vehicles, and aircraft industries are among those with the largest errors. In each of these industries, the differences between the forecast and actual changes in the employment of the sample were more than 5.0 percentage points for at least half of the target dates considered. The basic materials group, and the chemicals, paper, and petroleum products industries within that group, had the smallest mean sample forecast errors. The mean SFE for these industries ranged between 0.59 and 1.95 percentage points (including both the three- and six-month forecasts) ; and nearly one-half of the errors were less than 1.0 percentage points for each industry. It is useful to compare the mean (absolute) sample forecast error with the mean

64

THE EMPLOYMENT FORECAST SURVEY

TABLE XV ESTIMATED FORECAST ERRORS^) OF THE EFS SAMPLE, MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 1955W Three-month forecasts

Industry

Mean (absolute) error (p.c. points)

Six-month forecasts

Percentage of target dates with errors of Less than i. o percentage point

5.0 percentage points or more

Mean (absolute) error (p.c. points)

Percentage of target dates with errors of Less than i. o percentage point

5-0 percentage points or more

Manufacturing

1.57

30

2.49

20

Basic materials Chemicals Non-ferrous metal Non-metallic mineral Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood

0.92 1.04 1.22 2.71 1.09 5.21 0.59 2.67 4.88 3.49

70 60 50 20 50 10 90 0 0 20

0 0 0 20 0 50 0 10 30 40

1.95 1.34 1.87 3.28 1.59 8.49 0.79 2.62 7.49 3.29

40 50 20 20 40 10 80 50 10 40

10 0 0 30 0 50 0 30 60 40

Consumer finished goods Clothing Electrical apparatus and supplies Food and beverages

1.85 3.33 2.68 1.68 3.54 8.72

40 0 30 30 30 20

1 20 20 0 30 50

3.63 4.24 4.84 1.82 6.15 17.49

20 10 20 60 0 0

20 50 40 10 40 80

2.61 12.44 6.38 1.56 1.36 8.18 4.50

20 0 30 40 10 20 0

10 80 50 0 0 60 30

3.73 12.66 11.43 2.41 1.85 11.25 4.26

0 0 0 20 40 10 20

20 60 60 0 0 70 30

Furniture

Vehicles Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding and repairing

SOURCE: The actual and forecast employment of the EFS sample establishments as reported quarterly to the Economics and Research Branch of the Department of Labour. (a) Forecast error is defined as the number of percentage points difference between the actual and forecast year-to-year percentage changes in the (aggregate) employment of the sample establishments. For a more detailed definition, and a description of the procedure by which the forecast errors were computed, see chapter iv. (¿0 The estimated forecast errors at ten target dates are taken into account.

(absolute) change in the employment of the sample over the same time intervals for each industry. The ratio of the former to the latter provides a rough measure of the relative errors of the industry forecasts. These ratios, classified by industry and length of forecast, are shown in Table XVI. These ratios indicate that: (a) For the three-month forecasts, there were fourteen of the twenty-four industries and groups for which the mean forecast errors were less than one-half of the mean actual change. There was only one industry for which the ratio was less than 0.30. (é) For the six-month forecasts, there were three industries with mean forecast errors less than one-half of the mean actual changes in actual employment. There were four industries for which the ratio was greater than 0.80. (c) With the exception of the six-month forecasts for the vehicles industry, the mean

65

THE FORECAST PROBLEM

TABLE XVI THE RELATIONSHIP BETWEEN THE ACTUAL CHANGES^) IN THE EMPLOYMENT OF THE EPS SAMPLE AND THE ESTIMATED EFS SAMPLE FORECAST ERRORS^) MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 1955

Industry

Manufacturing

Mean M actual change (per cent)

Three-month forecasts Mean(c) error (p.c. points)

Six-month forecasts

Ratio of mean error to mean actual change

Mean(c) error (p.c. points)

Ratio of mean error to mean actual change

4.40

1.57

0.35

2.49

0.57

Basic materials Chemical products Non-ferrous metal Non-metallic mineral Paper products Primary iron and steel Products of petroleum and coal Rubber products Textile products Wood products

3.22 2.56 2.40 3.80 2.10 10.65 0.95 4.59 10.78 5.00

0.92 1.04 1.22 2.71 1.09 5.21 0.59 2.67 4.88 3.49

0.28 0.41 0.51 0.71 0.51 0.49 0.62 0.58 0.45 0.70

1.95 1.34 1.87 3.28 1.59 8.49 0.79 2.67 7.49 3.29

0.61 0.52 0.78 0.86 0.76 0.80 0.83 0.57 0.69 0.66

Consumer finished goods Clothing Electrical apparatus Food and beverages Furniture Vehicles

5.06 7.25 8.54 1.93 9.70 15.96

1.85 3.33 2.68 1.68 3.54 8.72

0.37 0.46 0.32 0.87 0.36 0.55

3.63 4.24 4.84 1.82 6.15 17.49

0.72 0.58 0.57 0.94 0.63 1.10

Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding and repairing

8.61 17.19 21.37 4.71 3.99 22.60 15.12

2.61 12.44 6.38 1.56 1.36 8.15 4.50

0.31 0.72 0.30 0.33 0.34 0.36 0.30

3.73 12.66 11.43 2.41 1.85 11.25 4.26

0.43 0.74 0.53 0.51 0.46 0.50 0.28

SOURCE: Table XV. (a) The year-to-year percentage change in the aggregated actual employment of the EFS sample establishments at each target date. (¿) Forecast errors estimated from revised EFS predictions. Forecast error is defined as the number of percentage points difference between the actual and forecast year-to-year percentage changes in the aggregate employment of the sample establishments at the same target date. (c) Mean absolute.

forecast errors have, in every industry, been smaller than the mean actual changes in sample employment. Together these findings suggest that, for most industries and groups, the threemonth sample forecast errors have been, on the average, between one-third and onehalf of the magnitude of the corresponding actual changes in sample employment. The six-month sample forecast errors, on the other hand, have been between onehalf and three-quarters of the magnitude of the actual changes in sample employment. With such large errors, it is apparent that the forecasts of the sample establishments, in aggregate, have not provided highly reliable estimates of the magnitudes of future changes in their total employment. It should be noted, however, that the relative forecast errors (as measured by the ratios given in Table XVI) were conspicuously smaller than the relative sampling errors (as measured by the ratios given in Table XII). The percentages of incorrect sample forecasts of the direction of the non-seasonal

66

THE EMPLOYMENT FORECAST SURVEY

changes in the employment of the sample itself are given in Table XVII.1 With data available for only ten target dates, April 1, 1953, to July 1, 1955, it is possible to examine, for each industry and group, eight sample directional forecasts based on the three-month forecasts of establishments and seven sample directional forecasts based on the six-month forecasts of establishments. It is not necessary to emphasize that the number of predictions taken into account is unfortunately small, particularly for consideration of the reliability of directional predictions. The data given in Table XVII show that there were only four of the twenty-four TABLE XVII INCORRECT PREDICTIONS^) OF THE DIRECTION OF THE NON-SEASONAL CHANGES IN THE EMPLOYMENT OF THE SAMPLE, MANUFACTURING INDUSTRIES, APRIL 1, 1953—JULY 1, 1955 Over sixmonth intervals(') Industry Manufacturing Basic materials Chemical Non-ferrous metal Non-metallic mineral Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood Consumer finished goods Clothing Food and beverage Electrical apparatus Furniture Vehicles Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial, office and business machines Railway rolling stock Shipbuilding and repairing TOTAL

Over ninemonth intervals^)

Number M

Per cent

Number W

Per cent

2 1 1 2 2 2 3 4 2 2 1 0 1 1 2 1 3 1 3 0 2

25.0 12.5 12.5 25.0 25.0 25.0 37.5 50.0 25.0 25.0 12.5 0.0 12.5 12.5 25.0 12.5 37.5 12.5 37.5 0.0 25.0

4 2 1 5 3 3 5 4 2 2 2 3 2 2 1 3 7 4 3 3 4

57.1 28.6 14.3 71.4 42.9 42.9 71.4 57.1 28.6 28.6 28.6 42.9 28.6 28.6 14.3 42.9 100.0 57.1 42.9 42.9 57.1

0 0 2 38

0.0 0.0 25.0 19.8

3 4 1 73

42.9 57.1 14.3 43.5

SOURCE : As for Table XV. M For a definition of an incorrect prediction of the direction of a non-seasonal change in the employment of the sample, see text. (*) The time intervals over which the non-seasonal changes are computed for EPS predictions based on the three- and six-month forecasts of establishments, respectively. Seesupra, p.32. W A total of eight predictions of the changes over six-month intervals were taken into account in the period April 1, 1953—July 1, 1955. (rf) A total of seven predictions of the changes over nine-month intervals were taken into account in the period April 1, 1953—July 1, 1955.

THE FORECAST PROBLEM

67

CHART 7. The actual and forecast year-to-year percentage changes in the employment of the EFS sample, Manufacturing Industry Group, target dates April 1, 1953, to July 1, 1955. industries and groups for which all of the sample directional forecasts based on threemonth forecasts were correct in terms of direction. There were no industries or groups for which each of the seven directional forecasts based on the six-month forecasts of establishments were correct. Twenty per cent of the 192 directional forecasts based on three-month forecasts were incorrect, as were over 40 per cent of the 168 directional forecasts based on six-month forecasts. For half of the industries and

68

THE EMPLOYMENT FORECAST SURVEY

l

i

l

i

l

í

CHART 8. The actual and forecast year-to-year percentage changes in the employment of the EFS sample, Basic Materials Industry Group, target dates, April 1, 1953, to July 1, 1955.

groups, one quarter or more of both the three- and six-month directional predictions were incorrect. 2. ANALYSIS OF THE EFS SAMPLE FORECAST ERRORS There was one non-seasonal turning point in the period under review. Therefore, by the projection of the actual direction of change from the immediately preceding in-

THE FORECAST PROBLEM

69

CHART 9. The actual and forecast year-to-year percentage changes in the employment of the EFS sample, Consumer Finished Goods Industry Group, target dates April 1, 1953, to July 1, 1955.

tervals, it would have been possible to derive directional "forecasts" which, for most industries, would have been incorrect only once. Seen in this light, the sample directional forecasts for many industries probably were of little or no value. However, the discovery that there were large errors (in terms of both direction and magnitude) in the sample forecasts does not necessarily preclude their future improvement. As pointed out in chapter iv, it is possible that consistent errors in the EFS pre-

70

THE EMPLOYMENT FORECAST SURVEY

CHART 10. The actual and forecast year-to-year percentage changes in the employment of the EFS sample, Producer Finished Goods Industry Group, target dates April 1, 1953, to July 1, 1955.

dictions can be corrected. The same holds for the sample forecasts. Unfortunately, an analysis of the sample forecast errors discloses that they, like the errors of the EFS predictions, have not been consistent through time; nor have they been consistently related to the seasonal movements in the employment of the sample. To secure a fuller understanding of their characteristics, through time, recourse may be had once again to scatter diagrams. As before, the forecast changes are plotted along the hori-

THE FORECAST PROBLEM

71

zontal axes, and the corresponding actual changes are plotted along the vertical axes. Because of space limitations, only the data for manufacturing and the three groups are presented in graphic form. Charts 7 to 10 show that the sample forecast errors possessed the same cyclical characteristics as the errors of the EFS predictions discussed in chapter iv. When the actual employment of the sample was declining non-seasonally, the sample forecasts tended to underestimate the magnitudes of the declines. Similarly, when the actual employment of the sample was rising non-seasonally, the sample forecasts tended to underestimate the magnitudes of the gains.2 In plain language, the sample forecast changes lagged the corresponding actual non-seasonal changes. The data given in the charts strongly support the earlier contention that the sample forecast errors were not consistent, either in direction or magnitude, through time.3 Clearly no simple correction, such as the addition or subtraction of some constant, would have substantially increased the reliability of the sample forecast errors. Although there are other techniques, possibly the most obvious adjustment would have involved fitting a regression line (or lines) to the actual and forecast non-seasonal changes in the employment of the sample.4 Had there been relatively little variation about the fitted line, and had the parameters of the fitted line remained relatively stable through time, the actual changes in the employment of the sample could have been reliably estimated from the forecast changes by means of the regression equation. The plotted points shown in each of the charts strongly indicate that this procedure would not have been effective. There would not appear to have been any straight or curved regression line(s) that could have been fitted to the data in each chart such that the variation about it would have been substantially less than the variation about the 45-degree line. It is doubtful, therefore, that a significant reduction could have been made in the sample forecast errors by the adoption of a regression procedure. The principal obstacle to the correction or adjustment of the sample forecasts lies in the fact that the errors appear to have been related to cyclical changes in the employment of the sample. Any correction or adjustment of the sample forecasts would have required the reliable prediction of the cycle itself. For instance, the charts suggest that larger adjustments (movements from the plotted points to the 45-degree line) would have had to have been made for the forecast changes at target dates between cyclical turning points. To make such a correction it would have been necessary to know, at the time the establishments were making their forecasts for a particular target date, if the seasonally adjusted employment series (of the sample) would change direction prior to, or subsequent to, the current target date. Because the prediction of cyclical turning points is one of the prime purposes of the EFS, the adoption of this form of correction would have been entirely circular. The analysis of this chapter has strongly suggested that it is unlikely any simple correction could have been effectively applied to the aggregate sample forecasts. However, the data have covered only a short time period; and no consideration has been given to the individual establishment forecasts which comprise the sample forecast errors. In the next chapter an attempt is made to remedy these deficiencies.

CHAPTER EIGHT

THE P R E D I C T I V E V A L U E OF THE E S T A B L I S H M E N T FORECASTS

THE PREVIOUS CHAPTER indicated that the aggregated forecasts of the sample establishments had errors of the same kind as the EPS predictions which were derived from them. The purpose of this and the following chapter is to analyse the characteristics of the forecasts of the individual establishments included in the sample. Specifically it is necessary to consider (a) the adequacy of the forecasts submitted by the individual establishments; (¿>) the sources of their forecast errors; (c) whether the errors of the establishment forecasts could be reduced. The first question is examined in this chapter. It is possible to assess the adequacy of the forecasts submitted by individual establishments in the same way that the adequacy of the EPS predictions was appraised in chapter iv. By comparing the errors of the submitted forecasts with the errors that would have occurred had the actual employment of the establishment simply been extrapolated over the forecast interval it is possible to judge whether the establishment forecasts had any net value in the prediction of their own employment levels. Similarity, from a comparison of the frequency with which the establishment forecasts incorrectly predicted the direction of the non-seasonal changes in their own employment with the number of non-seasonal turning points they experienced over the same time period it is possible to determine whether the establishment's forecasts had any net value with respect to the prediction of the direction of changes in its own employment. Both of these assessments are made in this section for the 388 sample establishments which submitted complete data for the 24 target dates in the period October, 1952, to July, 1958, inclusive. There were three kinds of extrapolations used to evaluate the forecasts. In the assessment of the ability of each establishment to forecast the future level of its own employment two estimates were derived which were comparable with each of the forecasts submitted in the period under review. The first type of estimate, which we have called a "no change" estimate, was derived by the horizontal extrapolation, to the relevant target date, of the actual employment of the establishment at the time each forecast was submitted. In terms of symbols, the no change estimate used to evaluate each six-month forecast1 is of the form: zt+§= yt, where Zt+& is the estimate ofyt+6 and *¡+6 is the establishment's own forecast ofyt+&The second type of estimate used to evaluate the adequacy of the establishment forecasts as predictors of their own employment levels we have termed a "seasonal change" estimate. Estimates of this type were derived by projecting the latest actual employment of an establishment by the seasonal change which actually occurred in its employment over the same period in the previous year. In symbols the seasonal change estimate comparable with a six-month forecast, x¡+e, is of the form z't+§ =

PREDICTIVE VALUE OF ESTABLISHMENT FORECASTS

73

The seasonal indexes, which are denoted by the symbol S, were computed as the ratio of the actual employment of the establishment at each quarterly target date to the four quarter moving average of the actual employment of the establishment centred at the same date. The third type of estimate was used to evaluate the adequacy of the establishment's forecasts of the direction of non-seasonal changes in its own employment. In this case the sign of the non-seasonal change in the establishment's employment over the three-month interval preceding the date on which the forecast was submitted was taken as the estimate of the sign of the non-seasonal change in the establishment's employment over the six-month forecast interval. In symbols, the sign of

was taken as the estimate of the sign of

This esti-

mate of the direction of change is comparable with the prediction of the direction of non-seasonal change based on the forecast xt+e submitted by the establishment. This prediction is given by the sign of The forecasts submitted by a particular establishment have been defined as devoid of net value with respect to the prediction of the employment levels of that establishment when the average error of the no change estimate2 and/or the average error of the seasonal change estimate3 was less than the average forecast error of the establishment.4 The forecasts submitted by a particular establishment have been defined as devoid of net value with respect to the prediction of the non-seasonal changes in the employment of that establishment when the proportion of correct estimates of the direction of change was greater than the proportion of correct forecasts of the direction of change over all forecast intervals. Under the latter definition we are, in effect, claiming that the submitted forecasts of an establishment were of no predictive value if the frequency of incorrect predictions of direction was greater than the frequency of non-seasonal turning points. Forecast intervals are defined as following non-seasonal turning points when the sign of

is not the same as the sign of

By definition, the esti-

mates of the direction of non-seasonal change will incorrectly predict the direction of change over those intervals which follow turning points and correctly predict the direction of non-seasonal change over all other intervals. 1. FORECASTS OF EMPLOYMENT LEVELS BY INDIVIDUAL ESTABLISHMENTS Table XVIII shows the proportion of the 388 sample establishments which, over the period under review, submitted six-month forecasts which were without net predictive value with respect to the prediction of employment levels, by type of estimate. These data suggest that with respect to employment levels the no change estimates

74

THE EMPLOYMENT FORECAST SURVEY

TABLE XVIII PROPORTION OF SAMPLE ESTABLISHMENTS W WHICH SUBMITTED SIX-MONTH FORECASTS WITH No NET PREDICTIVE VALUE (W WITH RESPECT TO THE PREDICTION OF THEIR OWN EMPLOYMENT LEVELS, MANUFACTURING INDUSTRIES, OCTOBER, 1953—OCTOBER, 1957 Standard of evaluation

Industry

Manufacturing Basic materials Chemicals Non-ferrous metal Non-metallic mineral Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood Consumer finished goods Clothing Electrical apparatus and supplies Food and beverages Furniture Vehicles Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding and repairing Other manufacturing industries

Number of establishments

No change estimates (c)

Seasonal change estimates (d]

No change and/or seasonal change estimates

388 158 22 13 23 51 11 17 9 21 18 110 8 64 14

39.4 35.1 36.4 46.2 21.7 25.5 63.6 58.8 44.4 66.7 38.8 38.1 87.5 20.3 71.4 66.7 22.2 50.0 25.0 100.0 52.0 52.9 33.3 50.0 42.9

30.7 34.0 36.4 38.5 39.1 39.2 45.4 23.5 22.2 28.6 22.2 24.5 37.5 18.8 35.7 33.3 22.2 32.6 25.0 0.0 28.0 47.1 16.7 50.0 14.3

52.1 52.4 45.4 53.8 52.2 54.9 72.7 23.5 55.6 71.4 44.4 45.4 87.5 31.2 71.4 73.3 22.2 64.0 25.0 100.0 62.0 70.6 33.3 62.5 42.9

15 9

86

4 1

50 17 6 8 7

SOURCE: As for Table XV. M Includes all 388 manufacturing establishments which submitted complete forecast and actual data for the period October, 1952—July, 1958. Seventeen six-month forecasts submitted in this period were considered. (') An establishment's forecasts were defined as without net predictive value when the mean (without regard to sign) of the relative errors of the forecasts was greater than the mean (without regard to sign) relative errors of the comparable estimates. W No change estimates were based on the extrapolation of the latest actual employment of the establishment, at the time each forecast was submitted, to the target date. M Seasonal change estimates were based on the extrapolation of the latest actual employment of the establishment to the target date by the relative seasonal change which occurred over the same interval in the previous year.

provided a more stringent standard of net predictive value than did the seasonal change estimates. About 10 per cent of the establishments were judged to have net predictive value in this sense by the seasonal change estimates which were evaluated as devoid of net predictive value by the no change estimates. Taken separately, a majority of the establishments submitted six-month forecasts

PREDICTIVE VALUE OF ESTABLISHMENT FORECASTS

75

which, on the average, predicted their own employment levels better than either the no change or seasonal change estimates. However, a majority of the establishments could not surpass both the seasonal change and no change standards, except in the consumer finished goods group. In other words, had it been possible to know in advance whether to replace all of the forecasts of a particular establishment by the consistent use of one type of estimate or the other, the average absolute error of the forecasts of employment levels would have been reduced for over 50 per cent of the establishments. A similar examination of the three-month forecasts submitted by the same establishments over the same time period disclosed that over 60 per cent of them had submitted three-month forecasts which were without net value in predicting their own employment levels when judged by the no change and/or seasonal change estimates. The foregoing summary of the net predictive value of the forecasts in predicting employment levels is, of course, based on averages of the errors of the forecasts and the estimates ; and averages are notoriously unreliable because they often reflect extreme values. Two checks were made to ensure that these averages represented the true differences between the forecasts and the estimates. The first check involved an analysis of the frequency with which the estimates for each establishment would have given errors equal to or less than the error of the forecasts submitted by the establishments. It was discovered that only about 10 per cent of the establishments which were defined as without net predictive value with respect to employment levels submitted forecasts which most of the time had smaller errors than the comparable estimates. The same check when applied to the three-month forecasts showed that if the frequency with which the forecast errors had been greater than the errors of the estimates had been taken into account, 51.0 per cent of the establishments would have been judged to be without net predictive value with respect to employment levels, as compared with the 60.3 per cent mentioned above. A second check was made of the validity of the net predictive value (employment levels) definition. For each establishment the mean difference between the errors of the six-month forecasts and the errors of both types of estimates was tested in order to ensure that it was statistically significant. These tests disclosed that had net predictive value been defined in terms of significant differences (at the .05 level) the proportion of establishments judged to be without net predictive value (employment levels) would have been raised slightly. Together these two checks seem to confirm that our definition of net predictive value with respect to employment levels is meaningful in the vast majority of cases. We therefore conclude that roughly one-half of the establishments submitted sixmonth forecasts which had no value in predicting their own future employment levels as compared with the predictions which could have been made by simply extrapolating their actual employment. About two-thirds of the establishments submitted three-month forecasts with these characteristics. 2. PREDICTIONS OF THE DIRECTION OF NON-SEASONAL CHANGES BY INDIVIDUAL ESTABLISHMENTS Only 20 of the 388 establishments correctly forecast the direction of the non-seasonal changes in their own employment more frequently than would have been achieved had it been assumed that the direction of change which actually occurred over the

76

THE EMPLOYMENT FORECAST SURVEY

three-month interval preceding the date on which the forecast was submitted would continue over the six-month forecast interval. Put in another way, we can say that 348 of the 388 establishments failed to predict correctly the direction of change of TABLE XIX PROPORTION OF SAMPLE ESTABLISHMENTS M WHICH SUBMITTED SIX-MONTH FORECASTS WHICH FAILED TO CORRECTLY(*) PREDICT THE DIRECTION OF THE NON-SEASONAL CHANGES IN THEIR OWN EMPLOYMENT AT LEAST HALF OF THE TIME, MANUFACTURING INDUSTRIES, OCTOBER, 1953—OCTOBER, 1957 Predictions made for

Industry

Number of establishments

Manufacturing Basic materials Chemicals Non-ferrous metals Non-metallic minerals Paper Primary iron and steel Products of petroleum and coal Rubber Textiles Wood Consumer finished goods Clothing Electrical apparatus and supplies Food and beverages Furniture Vehicles Producer finished goods Agricultural implements Aircraft Fabricated iron and steel Industrial machinery Railway rolling stock Shipbuilding and repairing Other manufacturing industries

388 185 22 13 23 51 11 17 9 21 18 110 8 64 14 15 9 86 4 1 50 17 6 8 7

All intervals

Intervals (c) following turning points

Other (rf) intervals

All intervals and/or intervals following turning points

64.2 65.9 54.5 69.2 73.9 74.5 72.7 35.3 66.6 76.2 55.6 65.4 75.0 57.8 57.1 86.7 88.8 57.0 75.0 100.0 68.8 47.0 16.7 25.0 85.7

42.0 43.8 40.9 53.8 39.1 42.0 45.4 29.4 66.7 52.4 27.8 38.2 50.0 28.1 50.0 73.3 22.2 41.9 50.0 100.0 44.0 41.2 50.0 12.5 57.1

78.6 78.9 72.7 84.6 87.0 86.3 81.8 47.0 77.8 90.5 66.6 90.0 87.5 90.6 85.7 93.3 88.9 62.8 75.0 0.0 76.0 52.9 33.3 25.0 85.7

71.4 75.7 68.2 84.6 87.0 78.4 72.7 64.7 77.8 81.0 61.2 69.1 75.0 60.9 71.4 86.7 88.9 70.9 75.0 100.0 72.0 58.8 50.0 37.5 71.4

SOURCE: As for Table XV. («0 Based on the 388 manufacturing establishments which submitted complete forecast and actual data for the period October, 1952—July, 1958. Seventeen forecasts for each establishment were considered in this period. (') A forecast was defined as a correct prediction of the direction of change when the sign of the predicted non-seasonal change over the six-month forecast interval was the same as the sign of the actual non-seasonal change. The technique of seasonal adjustment is defined in the text. M When the sign of the actual non-seasonal change in the employment of the establishment over the six-month forecast interval was different from the sign of the actual non-seasonal change over the immediately preceding three-month interval the forecast interval was defined as "following a target date." (