New York – In the world of performance evaluation, we have generally accepted the concept of synthetic return, batting averages and sorting ability as the industry standards of how well a research firm’s recommendations behave. While we generally support the approach and goals of this analysis, there are other ways to assess forecasting ability, namely forecast error.
Forecast error evaluation is the standard mechanism to assess the trustworhiness of the results eminating from regression or econometric analyses. If a forecasting technique, be it qualitative or quantitative, is valuable, it should exhibit a low differential between the forecasts it produces and the future results that arise.
As such, a forecast that demonstrates a low variation in its errors is considered to be a good forecast.
In synthetic return analysis, we take each of a research firm’s recommendations and synthetically buy, sell or hold the firm’s recommended stocks. The firm that has the highest synthetic return wins the contest. While this is a common sense approach, it does not necessarily reflect good forecasting.
On the other hand, good forecasting does not necessarilly produce a profit. Since forecast error focuses on the closeness of the forecast to the actual, it is possible that the forecast and the actual could be very close, yet have opposite signs. This would be a bad result for a synthetic performance perspective.
Also, a firm that wins on synthetic returns, could have expected a stock to rise 5%, while it subsequently rose by 35%. The current systems would assess this as a superior result, even though it would be a bad forecast (i.e. the forecast is a vast distance from the result).
What we are really measuring using the synthetic returns is the ability of the firm to sort its coverage list into buy and sell buckets. If RPs could actually forecast stock prices, rather than simply allocate them to buy, sell, or hold classifications, we could be much more acurate in assessing the quality of those forecasts.
Meanwhile the institutional market has been shifting its focus from stock recommendations to lower level (primary) information, including supply chain analysis, channel checks, product pipeline analysis and market research. Although many of the firms in this space do not produce traditional stock recommendations, the ideas that are generated are clearly actionable.
Consequently, good performance is not necessarily synonymous with good forecasting, nor are either necessarily synonymous with good research.