Experimentation - Limitations of Experiments
Experimentation - Limitations of Experiments
Experimentation is a powerful tool in the search for unambiguous relationships that we hope may be used to make valid predictions about the effects of marketing decisions and to develop basic theories. The laboratory experiment is the preferred method because of its internal validity; however because of acute external validity problems in the laboratory setting, managers are reluctant to rely upon it. Unfortunately, the field experiment is beset by a number of problems whose net effect has been to limit the vast majority of marketing experiments to short-run comparisons across store; home placements of product variations, and so forth. Relatively few large-scale experiments with social programs, marketing programs, or advertising campaigns are conducted in any given year. What are the reasons?
Cost
Cost and time pressures are the first hurdle. Even "simple" in-store testa require additional efforts to gain cooperation; to properly place the display price, or promotion; to measure the uncontrolled variables; and then t: audit the resulting sales differences. The measurement costs alone are often substantial. When larger interventions, such as comparing alternative advertising themes in multiple geographic areas, are contemplated, management may be very wary that the benefits will exceed the costs. These costs are likely to be considerable if any amount of reinterviewing or special manipulation of advertising, product, or other controllable variables is required.
Another cost is the fact that management decisions are delayed by the research. For some experiments the impact of the experimental treatment can extend over a long time period, as much as a year or even longer Consider the Budweiser advertising experiments, which were run for a full year. The researcher is often placed in a difficult position. If the experiment-especially the after observation, is not allowed sufficient time, the validity of the results might suffer. On the other hand, if the experiment is too lengthy the resulting delays in making and implementing policy might be unaccept-ably long. One way to avoid the constraint of time pressure is to conduct an ongoing program of experimentation in anticipation of recurring decisions. Thus, some companies are building up data files of responses to marketing programs in a variety of contexts.
Security
Still another cost is that of security. A field experiment naturally involves exposing a marketing program in the marketplace, so it is difficult to hide from competitors who are in contact with their own field salesforce, store personnel, research suppliers, arid trade sources. For example, one consumer products company constructed product displays in several stores in one city for one eight-hour period on one Friday. At the end of the day all traces of the test had been removed. By Monday the major trade magazine and most competitors knew of the test and the details of the test product.
Implementation Problems
Implementation problems abound in the conduct of experiments. First, it may not be easy to gain cooperation within the organization. Proposals to experiment with varying the size and call frequency of a salesforce are resisted by regional managers, who do not want their market area to receive a reduction in sales effort. Administrators of social programs may resist efforts to assign people randomly to treatments. They want to decide the assignments according to who can benefit most from the service and which service is most suitable. A second problem to which experiments involving market areas are especially susceptible is contamination, because of an inability to confine the treatment to the designated experimental area. Buyers from one geographic area may visit an adjacent area or receive media messages that overflow from that area. It is seldom possible to partition geographic market areas so that the sales measurements and treatments exactly coincide. Experiments involving people as test units may become contaminated because people in the control group associate with people assigned to the program and learn what they have been doing. A third problem is that the variability in behavior across test units can be so large that it is difficult to detect experimental effects. Of course, some of the variability can be removed by adding blocking or by matching, but it will not be eliminated. The question is: Can it be reduced enough so that the experimental effects can be discerned?
The ultimate problem, however, is that there may be no person or geographic area available to serve as a control. This is the case with industrial markets composed of a few large buyers who communicate among each other or are geographically dispersed. Any effort to limit a new-product introduction to a subset of such a market would be unsuccessful. The same problem occurs in attempting to assess the effects of federal legislation, which goes into effect in all parts of the country at the same time. Also, practitioners may be unwilling to deny access to a social program because they believe that it is counter to their professional obligation.
Uncertain Persistency of Results
This is the final category of the problems that limit the acceptance and usage of field experiments. For an experimental result to be useful it must
hold long enough to be acted on to advantage. The two factors most damaging to an assumption of persistency are high rates of technological, economic, or social change in the market environment, and aggressive competitive behavior. During the experiment the competition may elect tc monitor the test independently and learn as much as possible—or take unusual action, such as a special consumer promotion, to confound the results. Similarly, when the test is expanded to a regional or national market the competitors may either do nothing or retaliate. This means that there are at least four combinations of circumstances, each with different implications for the nature of the causal relationship being studied:
c E No response Q g- Retaliation
POSSIBLE COMPETITIVE
RESPONSES
After Experiment
No response Retaliation
1 3
4 2
To the degree that the decision maker cannot assess, (1) the probability of each of the four possible events, (2) the magnitude of the retaliatory action if any, or (3) the number of direct competitors taking action, then the persistency—and hence the value—of the experimental results will be uncertain.
Experimentation is a powerful tool in the search for unambiguous relationships that we hope may be used to make valid predictions about the effects of marketing decisions and to develop basic theories. The laboratory experiment is the preferred method because of its internal validity; however because of acute external validity problems in the laboratory setting, managers are reluctant to rely upon it. Unfortunately, the field experiment is beset by a number of problems whose net effect has been to limit the vast majority of marketing experiments to short-run comparisons across store; home placements of product variations, and so forth. Relatively few large-scale experiments with social programs, marketing programs, or advertising campaigns are conducted in any given year. What are the reasons?
Cost
Cost and time pressures are the first hurdle. Even "simple" in-store testa require additional efforts to gain cooperation; to properly place the display price, or promotion; to measure the uncontrolled variables; and then t: audit the resulting sales differences. The measurement costs alone are often substantial. When larger interventions, such as comparing alternative advertising themes in multiple geographic areas, are contemplated, management may be very wary that the benefits will exceed the costs. These costs are likely to be considerable if any amount of reinterviewing or special manipulation of advertising, product, or other controllable variables is required.
Another cost is the fact that management decisions are delayed by the research. For some experiments the impact of the experimental treatment can extend over a long time period, as much as a year or even longer Consider the Budweiser advertising experiments, which were run for a full year. The researcher is often placed in a difficult position. If the experiment-especially the after observation, is not allowed sufficient time, the validity of the results might suffer. On the other hand, if the experiment is too lengthy the resulting delays in making and implementing policy might be unaccept-ably long. One way to avoid the constraint of time pressure is to conduct an ongoing program of experimentation in anticipation of recurring decisions. Thus, some companies are building up data files of responses to marketing programs in a variety of contexts.
Security
Still another cost is that of security. A field experiment naturally involves exposing a marketing program in the marketplace, so it is difficult to hide from competitors who are in contact with their own field salesforce, store personnel, research suppliers, arid trade sources. For example, one consumer products company constructed product displays in several stores in one city for one eight-hour period on one Friday. At the end of the day all traces of the test had been removed. By Monday the major trade magazine and most competitors knew of the test and the details of the test product.
Implementation Problems
Implementation problems abound in the conduct of experiments. First, it may not be easy to gain cooperation within the organization. Proposals to experiment with varying the size and call frequency of a salesforce are resisted by regional managers, who do not want their market area to receive a reduction in sales effort. Administrators of social programs may resist efforts to assign people randomly to treatments. They want to decide the assignments according to who can benefit most from the service and which service is most suitable. A second problem to which experiments involving market areas are especially susceptible is contamination, because of an inability to confine the treatment to the designated experimental area. Buyers from one geographic area may visit an adjacent area or receive media messages that overflow from that area. It is seldom possible to partition geographic market areas so that the sales measurements and treatments exactly coincide. Experiments involving people as test units may become contaminated because people in the control group associate with people assigned to the program and learn what they have been doing. A third problem is that the variability in behavior across test units can be so large that it is difficult to detect experimental effects. Of course, some of the variability can be removed by adding blocking or by matching, but it will not be eliminated. The question is: Can it be reduced enough so that the experimental effects can be discerned?
The ultimate problem, however, is that there may be no person or geographic area available to serve as a control. This is the case with industrial markets composed of a few large buyers who communicate among each other or are geographically dispersed. Any effort to limit a new-product introduction to a subset of such a market would be unsuccessful. The same problem occurs in attempting to assess the effects of federal legislation, which goes into effect in all parts of the country at the same time. Also, practitioners may be unwilling to deny access to a social program because they believe that it is counter to their professional obligation.
Uncertain Persistency of Results
This is the final category of the problems that limit the acceptance and usage of field experiments. For an experimental result to be useful it must
hold long enough to be acted on to advantage. The two factors most damaging to an assumption of persistency are high rates of technological, economic, or social change in the market environment, and aggressive competitive behavior. During the experiment the competition may elect tc monitor the test independently and learn as much as possible—or take unusual action, such as a special consumer promotion, to confound the results. Similarly, when the test is expanded to a regional or national market the competitors may either do nothing or retaliate. This means that there are at least four combinations of circumstances, each with different implications for the nature of the causal relationship being studied:
c E No response Q g- Retaliation
POSSIBLE COMPETITIVE
RESPONSES
After Experiment
No response Retaliation
1 3
4 2
To the degree that the decision maker cannot assess, (1) the probability of each of the four possible events, (2) the magnitude of the retaliatory action if any, or (3) the number of direct competitors taking action, then the persistency—and hence the value—of the experimental results will be uncertain.
Comments
Post a Comment