New York, NY – Over the past five years, one of the fastest growing segments of the investment research business has been Primary Research. The providers in this segment provide detailed research and / or capabilities that enable analysts to measure the demand for various products and services, or help them gain direct insight into how companies are performing based on feedback from participants within the supply chain or the industry in general.
One type of Primary Research that has seen significant interest from the buy-side has been custom survey work as investors have wanted to gain a unique information edge which will enable them to make better investment decisions. While we think conducting surveys should be an important part of any analyst’s toolbox, we also think that surveys are often misused, inappropriately designed, or implemented incorrectly – all factors that could limit the usefulness or accuracy of this research tool.
General Overview of Using Surveys
It is critical to start a discussion on the use of surveys as a form of valuable primary research with a suggestion that analysts should examine the real need for undertaking a survey. Some key questions that analysts should address when considering implementing a survey is “what” is he/she trying to learn, and “why” is survey work a better solution versus other research techniques that might be employed such as focus groups, customer visits, or choice modeling, among others. The research objectives and the character of the project, and of the desired data, determine if a survey is the best choice. This article discusses this topic in greater detail.
Surveys typically provide precise data on frequencies, averages, dollar amounts, etc., thus providing a sell-side or buy-side analyst with a specific framework for taking action. The very nature of survey work allows an analyst to determine the exact number of goods purchased by a respondent last week, but it might not help discovering in-depth details on why or how the purchases were made.
“Surveys are a confirmatory tool whose proper purpose is to limit, narrow, and specify; hence, this tool is largely incapable of expanding, broadening, and reconfiguring your understanding.” (Edward McQuarrie, The Market Research Toolbox: a Concise Guide for Beginners. Sage Publications, 2006, page 32)
Over the last 100 years, market researchers and securities analysts have found significant value in approaching a segment of a determined population – a group of individuals that share common characteristics -, in order to make inferences of interest to the researcher. Estimation procedures and hypotheses tests are some of the inferences that can be made from survey work. In opposition to a survey, a census attempts to identify and question all individuals in a specific population.
Errors Associated With Surveys
A critical aspect of survey work is the assumption that the segment addressed is representative of the population of concern to the analyst. Survey work is based on the probability sampling paradigm, which indicates that when selecting random samples, probability theory can be applied, making it possible to quantify the accuracy of the estimates.
In order to design and conduct a survey that adheres to the probability sampling paradigm, a set of aspects need to be taken into consideration. It is not hard to initiate and complete a faulty survey while believing, erroneously, that the results are trustworthy. Sampling survey theory indicates that the process can be faulty in one or more of the following areas:
- Definition of the project and its objectives;
- Definition of the capabilities of survey work in relation with the project objectives;
- Definition of the characteristics of the population of interest;
- Selection of the sample and its source;
- Selection of the survey methodology;
- Design of the questionnaire;
- Survey administration; and
- Analysis of the results.
All these phases are interrelated in a way that an error in one of them affects the others. In other words, if the objectives of the project have not been thoroughly defined, the questionnaire cannot be properly designed; if the questionnaire is poorly designed the results will not be reliable for analysis; and so on. A correct application of each one of these stages is crucial to ensure that the data gathered is truly relevant for the researcher’s purposes. This document elaborates further on each one of these stages.
A survey project can be affected by two kinds of errors: sampling and non-sampling errors. Sampling errors arise from scrutinizing a segment instead of the whole population. The concept “sampling error” refers to the variation implied in random sampling. On the other hand, non-sampling errors, can be caused by inaccuracies in the measuring, recording, analyzing, or responsiveness.
Statistics and other disciplines provide tools to estimate and control total error (including sampling and non-sampling errors) in survey work. There is a classic tradeoff between sampling and non-sampling errors. Because of their nature, sampling errors can be reduced by increasing the size of the sample. However, a larger sample will increase non-sample errors such as non-response rate or mistakes in recording.
Strengths and Weaknesses of Surveys
The following chart highlights the major pros and cons of using surveys.
Strengths/Weaknesses of Surveys
A cautionary remark is should be noted, especially after highlighting the strengths and weakness of using surveys and discussing the kinds of errors that may affect a survey project. Authors have warned against an over-reliance on surveys. Despite the various ways for identifying and controlling errors within a survey, the researcher must not forget what a survey is: a probability paradigm that has been tested as accurate in the majority of cases, but not in all. A prudent recommendation is quoted below:
“Think instead of surveys as yielding one more fallible data point, to be combined with other data that are fallible for different reasons, as input to a decision that ultimately remains your own but that is more likely to be successful because you gathered diverse kinds of data.” (Edward McQuarrie, The Market Research Toolbox: a Concise Guide for Beginners. Sage Publications, 2006, page 97)