There's More to Analysis Than Swingin for the Fence


New York – We have discussed the Wall Street Journal/Thomson Analyst review already this week, but we feel there are a few other points to make.  Of interest is the actual set up and measurement of the analysts’ returns. We view the process to be somewhat lacking, biased and incomplete.

First, as our Monday blog indicated, the analytical homerun hitters are generally rewarded by the way performance is calculated.  If the analysts makes one call that adds 200% over the period, his performance would be stellar, but we question whether it may have been simply lucky. With a greater amount of data, we could assess the probability of the calls being lucky.

Second, the analysis has no bogey. In other words, there is no way to know if the winner in a particular industry beat his own industry or not. Of course, chances are that he/she did, but this should be explicit rather than implicit via comparing the analysts’ returns with the returns of a passive buy and hold portfolio of the analyst’s stock over the same time frame.

Third, raw performance does not account for the analyst’s ability allocate stocks into the correct category. This can be estimated by looking at the percentage of stocks that the analyst got in the right bucket (batting average). For example, the analysts had and average return of 30%, but had only 3 of 30 stocks he called Buys that actually increased in value. He would then have a but batting average of 10%.

Fourth, the analysts are allowed to have the announcement effect by virtue of getting the previous day’s closing price when they make recommendation changes. This means that the analysts of the Bulge bracket firms can benefit from announcing the sell and watching it move south in response to their own call. Also the analyst can game the system by changing recommendations during the next day of trading to take advantage of any news that may have hit the markets that day and booking the change in price from the previous day’s close as part of their own performance.

Fifth, there is no analysis of the amount risk the analyst has taken on board to achieve the returns posted. This risk can be take the form of market risk and default risk. There are well documented ways to test for the risk/return of a portfolio, including the Sharpe Ratio and the Information Ratio. Additionally, one could calculate the semi-variance of the portfolio – i.e. the risk of the stock falling, given a buy recommendation, or the risk of a stock rising, given a sell recommendation.

The obvious benefits of the Wall Street Journal /Thomson analyst assessment is its longevity and the consistency with which it has been calculated over the years. However, the performance could be augmented by adding some of the missing elements mentioned above.

Comment by Scott Drysdale:
None of the available services measure sell side research on a basis comparable to a portfolio manager. That includes Starmine.

For example, assume a fund manager buys stock XYZ at $30. For the next two months, he owns the stock and finally sells it at $45. Not bad. Well, what’s his performance? Up 50%? Well, to the client who invested in the fund the day the stock was trading at $45, their price was $45 and their return was zero. In other words, the performance of a call is a function of the price of the stock during the entire time it was “rated” buy. An analyst who recommends a stock at $30 and keeps it rated a buy up to $60 and continues to keep it a buy down to $45 is a negative 25%. One might also say it is 50% minus 25%, or, 25%. But, in the world of a money manager, it is never 50%.

The issue then of measuring performance of sell side analysts w/o considring the realities of investing and economics is a bit like saying a car goes from zero to sixty in zero seconds flat. It’s absurd. At a minimum recommendations must be average returns, and better yet, returns weighted on a price per day/days traded basis. All the rest of the measures are just some of the reasons PMs really couldn’t care less about the “performance” of sell side calls. PMs are not impressed by stocks that are rated buy and relatively outperform their benchmark for 2 weeks and then underperform for 10 weeks regardless of the nominal returns from beginning to end of the holding period.

Coment by Scott:
As someone who is intimately familiar with the WSJ’s annual analyst review (in a former life, I was involved with it), I can assure you that “the consistency with which it has been calculated over the years” is a mirage. Two different firms using totally different data sets and methodologies have conducted the review over the years.

I would be surprised if any past results could be replicated today. In fact, I would bet good money that 3 months from now this year’s results could not be replicated using then existing historical data (i.e., 3 months from now, re-create the data set from original source data used for this year’s review and then re-calculate the results). I would also be willing to extend that bet to other vendors’ platforms (since, I believe, the underlying data comes from the same sources). This annual exercise has advertising value for the WSJ and Thomson, for sure, but probably not much else, in my opinion.

Comment by David Lichtblau:
These flaws are well known, which is why the sell side puts so little weight on the WSJ results.

StarMine has been measuring analyst performance using the most rigorous, fair, and objective performance metrics since 1999. That’s why so many of the world’s publishing analysts actually have their StarMine scores used in their reviews and compensation plans.

StarMine’s metrics include the appropriate bogey; they include the ability to sort stocks correctly; they ensure a minimum batting average; and they score analyst performance only starting AFTER the call was made (the most inexcusable flaw in the WSJ methodology.) Analysts can’t score highly on StarMine metrics through the luck of being assigned the hottest industry or by moving the stocks with their own calls.

Separately, StarMine also quantitatively measures the accuracy of earnings estimates and not just the results of buy/sell recommendations.

StarMine’s 6th annual public tally of the top analyst performances are available online.


About Author

Leave A Reply