New York – Our Friday post on performance measurement generated a response from Joe Gatto at StarMine. Mr. Gatto covers the mechanisms which StarMine has developed to deal with the concerns we raised regarding the effect of superstar analysts, transaction costs, liquidity, and volatility on stock-picking performance.
One thing we can be certain of with StarMine is that the statistical analysis is solid. That is, when they mention that a measured response is significant, it means statistically significant. For instance, the measured response of stock prices to superstar analyst recommendations (the “Blodget-Grubman effect”) is, according to Starmine, both significant (large) and statistically significant (proven to have relevant explanatory power).
We include some of Mr. Gatto’s comments on performance measurement below:
—————————————————————–
- The Blodget-Grubman effect [of an up/downgrade]: we have measured it to indeed be significant (average 5% relative to broader market!) within the day of the recommendation change. Interestingly, that price level change persists, and does not soon revert to the pre-rec-change price.
- That is why all StarMine rec performance metrics use the closing price on the day of the recommendation. We close out the previous position and open the new position with the closing price (for those recommendations made before the close). If it is made after the close, we use the close from the next day. The “axes” complain that we do not capture their impact. Portfolio Managers agree it is the right way to go since that price move is so fast, so few investors can profit from it.
- Volatility effect: both our single-stock recommendation score and our broker rankings (which rollup single stock scores) explicitly consider volatility. A “Buy” that returns 15% for a given time period will receive a lower score if it is on a high volatility stock than if it is a low volatility stock. This concept captures both “opportunity” (a Buy on a high-volume stock is more likely to get to a very high return relative to the benchmark than a low vol stock) as well as, to some degree, trading costs (volatility by illiquidity). You are right that synthetic “RETURN” necessarily misses this volatility effect, but that does not mean that all synthetic scoring misses this point. Batting average (a score, not a return) misses this volatility effect, but our single-stock recommendation SCORE handles volatility appropriately. We stress-test the algorithm to ensure that the “dart thrower” picking high volatility stocks will not, statistically, get a high or lower score, or higher or lower variance of score than his colleague throwing darts at low-vol stocks.
- While it is true that quant strategies have higher “signal turnover” than broker recommendations, most fundamental managers do not rely on quant signals (and StarMine, in its main offering, does not rate quant signals). On the other hand, real quant portfolio managers care only about the backtested performance (decile spreads, ICs, IRs, etc.) of quant strategy and not at all about broker recommendations.
- Your last paragraph states that PMs do not pay attention to performance of recommendations; rather they pay attention to themes or access to management. I’ll presume to interpret that sentence to mean that they don’t pay attention to recomendations; rather to access. That blends two different concepts. They pay for access to management (because that is scarce). They don’t pay for recommendations since they are considered widely available. Also they will claim (with much truth) that they do not “rely upon” analyst recommendations. But boy do they listen to them or at least consider them. That is why brokers have been unable to remove recommendations (which have caused them legal liability in retail circles) from their research reports (the recommendation is an important bottom line). And if broker recommendations are completely ignored, there wouldn’t be a “Grubman Blodget” effect. If stocks move 5% the morning of an upgrade, who is doing the trading if not portfolio managers?
Joe Gatto
Founder and CEO
StarMine
+1 (415) 874-8100