The Utah Summer Games marketing research project.

Roberts, Wayne A., Jr. ; Steed, Emmett

CASE DESCRIPTION

The primary subject matter of this case concerns the development, implementation, and analysis of a real market research project. Secondary issues examined include the link between research objectives and questionnaire development, sampling and non-sampling error, and practical problems and issues that affect marketing research projects. The case has a difficulty level of four. The case is designed to be taught in one to two class hours, and is expected to require 2 to 3 hours of outside preparation by students.

CASE SYNOPSIS

In 2004 the new director of the Utah Summer Games, an athletic event modeled after the Olympics that draws almost 7500 athletes, is concerned about the lack of any data other than anecdotes and annual registrations. No one was sure how satisfied athletes and their families are with the athletic events, the opening and closing ceremonies, and the products, services and environment of Cedar City. They also do not know how people learn about the events. The case depicts the planning, implementation, and some results of a marketing research project developed to measure satisfaction levels regarding the community and the opening ceremonies, and to assess what other activities participants do in conjunction with the games. Manageable in scope, the case illustrates marketing research steps, has some shortcomings for students to identify, and has enough results to permit them to reach some tentative conclusions. The case is simple enough to be used in a marketing principles course. Its value is probably greatest in a marketing research course, where it can also be used as an illustrative project in the beginning, and referred to throughout the course as sampling and non-sampling error, questionnaire development, and data analysis topics arise. It could also be used as a model for semester-long student projects.

INSTRUCTORS' NOTES

RECOMMENDATIONS FOR TEACHING APPROACHES

This case can be used to illustrate the steps involved in a typical marketing research project. The basic steps as presented in any marketing principles or marketing research text should be evident in this case. As such, it can be used in a basic marketing principles class, or in the introductory portion of a marketing research course. Further, the decisions made at each step of the process can be discussed and critiqued.

A nice feature of this case is that simple frequency distributions are sufficient to meet the research objectives as presented in the case. Students with a little more training in statistics can use data to generate confidence interval estimates and to test hypotheses regarding the equality of means. In a marketing research class the instructor can refer to the case during the latter part of the class to illustrate data analysis techniques.

DISCUSSION QUESTIONS AND ANSWERS

1. What other specific research objectives could have been pursued? Why do you think they weren't chosen?

The intent of this question is to get students to generate research questions that could have been chosen. There are really a very large number of possibilities, but some specific ones mentioned in the case are as follows:

a. Given that the whole reason for sponsoring the games was to increase enrollment at Southern Utah University (SUU), an obvious objective would be to assess the impact of the Utah Summer Games (USG) on SUU enrollment. The tough part is to figure out a way to measure this. Clearly, administering a questionnaire to USG participants or to SUU students would not be enough. If students bring this up discussing possible ways of measuring the impact can be treated, but designing such research goes beyond the scope of this particular case. A reason for not pursuing such a study is that USG officials are not responsible for SUU enrollments.

b. Determine how, where, and when participants learn about the Utah Summer Games. To what extent are word of mouth, publicity, advertising, and the Web reaching participants? Since the enrollments were strong, it was felt that pursuing this research objective would not be as useful as evaluating opening ceremonies.

c. Measure the extent that athletes around the state know about the USGs. Are potential participants being missed in the communications programs? This would require a different type of study. Rather than surveying USG participants, it would have required finding and surveying athletes around Utah.

d. What is the economic impact of the USGs on the local economy? It would be possible to sample participants about their expenditures, the number of days spent in Cedar City, etc. This would have been a project in and of itself, and would not have helped the USGs directly. Note that the questionnaire (and objective 4) skirts this issue. It might also be possible to estimate the impact in other ways, such as by reviewing sales tax receipts, surveying local businesses about their perceptions regarding USG participant expenditures, or by measuring the change of expenditure levels through records of businesses.

e. How satisfied are sponsors with the USGs? There is no indication that those associated with the USG know much about the sponsors' expectations, how satisfied they are, whether the price of sponsorship is too high or low, etc. They are provided a packet which has photos and other information that documents the exposure they get, but that is it. This study would be totally different than what was pursued, and could be done after the summer games. A variation on this would be to measure awareness of USG participants of USG sponsors. Measuring satisfaction with the USG opening ceremonies would help the director with regard to the most expensive single event in the summer games, and the other chosen objectives were believed to be achievable and useful.

2. The researchers did not do any real exploratory, or qualitative, research prior to designing their research or their questionnaire. What might they have missed by NOT conducting in-depth interviews and focus groups? What are the dangers of skipping this step?

The biggest danger of not doing any qualitative research is that the research might focus on issues that miss the mark from the standpoint of respondents. In essence, they may ask the wrong questions. For example, the questionnaire asked about the specific acts and events at the opening ceremonies. It might be that other issues are more important, such as the timing of the ceremonies. The opening ceremonies run until about 11:00 p.m., and the soccer fields are filled with participants early the next morning. This might be a major concern. Safety of young athletes might be another issue. If there are issues with the games themselves, such as the refereeing, the facilities, or the locations, no one will know.

3. The chosen method for collecting data was to drop off and pick up the questionnaires at competitive events following opening ceremonies. What other methods could have been used, and what are the pros and cons of each method? Was the sampling method appropriate for achieving the objectives of the study?

This question can be used to generate discussion of various alternative ways of sampling. Starting with the last question first, a big shortcoming with the method employed is that the director of the USG could not learn how satisfied all types of opening ceremony attendees were with the ceremonies. By surveying observers at the competitive events, those who did not have an interest in the competitive events would be largely unrepresented. Parents, who largely were the respondents, may have liked certain aspects because their kids were involved, e.g., the parade of athletes, or because they believed the events would be enjoyed by their children. Note that this is an issue with many of the possible methods discussed below.

Other methods that could have been employed, and a brief listing of their pros and cons, are as follows:

Method Pros Cons

Mail Respondents could take time Time
survey to respond

 Questionnaire could be longer and Nonresponse
 address more issues

 Sample selection could be more Cost
 controlled and broader

 Attendees with no USG interest Teams registered,
 could be contacted by using not individuals, in
 information collected through many cases, and
 drawing forms filled out at the athlete lists may
 opening ceremonies. not be complete
 enough

Email Same as above Availability of
 active, relevant
 email addresses an
 issue

 Cost minimal Not all people have
 active email
 addresses

 Team registration
 problem (discussed
 above)

 Time

 Nonresponse (?)

Telephone Response rate Timing issue (when
 would you conduct
 the telephone
 interview?)

 Questionnaire length Cost, at least
 compared to drop-off
 and pick-up

Drop-off Attendees with no interest in the Nonresponse--could
at opening USG could be sampled not send reminders
ceremonies
with a Questionnaire could be longer and USG participants
return address more issues that did not attend
envelope would be
 unrepresented.

 Response rate might
 be low, particularly
 among people from
 outside the area.

Other possibilities, with their pros and cons, may be generated by students. Since most marketing principles textbooks and marketing research textbooks have tables similar to the table above, these can be linked directly to this question.

4. Critique the questionnaire. The researchers did not do a pretest. What are the risks of skipping this crucial step, and how might it have helped in this particular case? The questionnaire has some strong points and some shortcomings. This activity provides students a chance to see if the questionnaire will work with all respondents, and get the information needed in order to meet the study's objectives. The questionnaire can be evaluated by answering the following questions:

1. Will respondents...

UNDERSTAND the question,
be ABLE to answer, and
be WILLING to answer?

2. Will the research team UNDERSTAND the response?

3. What questions relate to what study objectives, and do all questions have a purpose?

4. Are there any problems with formatting, instructions, or layout?

Using the proposed framework, an obvious problem concerns question 4, which asks respondents to indicate how much the minor in their care enjoyed the various activities. While they may not know (a source of error that must be lived with), another issue is that many adults had more than one minor in their care. How should they respond? Another obvious problem, which was discovered after 176 questionnaires had been collected, is that the "go to" instructions in question 4 inadvertently asked respondents to skip over question 5. The error was corrected before the rest of the questionnaires were distributed.

The layout is, to a certain extent, consistent with Dillman's Total Design Method (Dillman, 1978; Salant and Dillman, 1994), and can be used to show how to use his method, if desired. The questionnaire is precoded, although not in the best fashion, and this can be discussed, as well. Part of the layout, particularly at the end, is potentially confusing to respondents and could result in errors.

Many of the problems and potential problems with the questionnaire could have been corrected with an adequate pretest.

5. What additional analyses of the data would you recommend?

At the very least it would be desirable to have a frequency distribution of responses to all the questions.

It might be informative to do a cross tabulation analysis of satisfaction with the opening ceremony events against the response to question 3 (regarding the attendance by a minor under their care), to see if satisfaction with the ceremonies is independent of the attendance by a minor under the respondent's care. It may be that those that had no athletes in the parade of athletes may not have found it very interesting, and this may also apply to the Utah Jazz Bear or other events.

Another possible question to be explored is whether responses to question 5 (regarding raising or lowering opening ceremony prices and quality) depends upon whether they attended with a young athlete (question 3). This could be done using a chi-square test.

Still another question is whether satisfaction with lodging options and prices are dependent on lodging arrangements. Those staying home, or with friends or relatives, likely feel different about Cedar City lodging options than those who had to camp or rent a motel room.

Other possible analyses could be suggested by students, of course. A key point that can be raised concerns the usefulness of the questions asked: If responses to a question on the survey instrument is not linked to a research objective, and therefore will not be analyzed, then why ask the question?

6. Evaluate the cross-tabulations and the associated chi-square tests. What do the significance level numbers mean?

This question is intended to provide students an opportunity to interpret the results of a chi-square test. The null hypothesis in the first cross tabulation, depicted in table 20, is that satisfaction with activities between events is independent of lodging arrangements. The concern is that people who do not stay with relatives or friends or home while attending the USGs may have less to do to occupy their time between events. The significance level of .255 indicates that if the null hypothesis is true then the probability of obtaining a chi-square of 2.732 or greater, given the sample sizes, is equal to 25.5%. Therefore we cannot reject the null hypothesis. In other words, we cannot reject the null hypothesis that satisfaction with activities between events is independent of lodging arrangements. One way of interpreting this is that knowing the lodging choice of a participant would not influence the probability they are dissatisfied or satisfied with activities between events.

The null hypothesis associated with the second cross tabulation, depicted in table 21, is that overall satisfaction with the USGs is independent of lodging arrangements. In this instance the significance level is .099, which is interpreted as follows: If the null hypothesis is true, then the probability of obtaining a chi-square value of 4.627 or greater, given the sample sizes, is 9.9%. Using an alpha level of 10% we would reject the null hypothesis. Using an alpha level less than 9.9 would result in our accepting, or failing to reject, the null hypothesis. If we reject the null hypothesis we need to determine why it was rejected. It appears that those who stayed at home or with friends or relatives tended to be more dissatisfied. If the numbers were greater it would be possible to explore this in more detail. However, even in this 3X2 table there are cells with expected values less than 5, and this is a problem when interpreting chi-square values.

7. Construct the 95% confidence interval for the overall satisfaction level with the Utah Summer Games. Interpret this number. Is there any non-sampling source of error with which you would be concerned?

Table 23 provides the data necessary for constructing confidence intervals and conducting hypothesis tests, and can be used to the extent desired. Here the instructor can ask students whether the measurement scales used are ratio, interval, ordinal or nominal, and discuss the importance of this in conjunction with hypothesis testing and confidence interval testing.

The correct answer to this question is as follows:

4.31 +/- 1.96(.054) = 4.31 +/- .106, or 4.20 to 4.42

Interpreting this confidence interval, there is a 95% chance that the true mean is between 4.20 and 4.42. Note that this assumes that the sample is a simple random sample from the USG, which it isn't, or, alternatively, that the feelings of those who were sampled are the same as those who were not represented. Hence there IS non-sampling error. Among the many athletes that are not represented are baseball players, arm wrestlers, and distance runners. Even the satisfaction levels of soccer players at the conclusion of the tournament are not represented.

Another possible exercise is to have students conduct confidence intervals or hypothesis tests using proportions using the data in tables 2 through 19. For example, students may be asked to construct the 95% confidence interval of the true proportion of USG participants that are dissatisfied with lodging options in Cedar City, using the data from table 13. The mechanical answer is as follows:

.196 +/-1.96(.03) = .196 +/- .056, or 13.7% to 25.2%

Again, the assumptions behind this confidence interval can be discussed, as can the problems of non-sampling error.

8. What conclusions can be reached, and what recommendations would you make?

To reinforce the link between research objectives and research, it is probably a good idea to discuss the results with regard to each research objective:

Objective 1: Determine how satisfied attendees are with various aspects of the opening ceremonies.

Conclusions: It is clear that the fireworks display was a hit with both adults and young athletes, at least if parent/guardian perceptions are accurate. This is evident from the frequency distributions and the sample means. The least successful act was the Black Hawk Band, but it is hard to argue that it was less than well received. One question raised by the USG director and marketing director concerned what type of music would be best received. This is something that could be tackled in a subsequent study.

Note again that the people who were sampled did not likely include many who did not have an athlete attending the opening ceremonies, and therefore it is not possible to assess how attendees, in general, feel about the various acts.

Objective 2: Determine if attendees would like to have opening ceremony prices and quality maintained, raised, or lowered.

Conclusions: Based on table 12 it is clear that the opening ceremonies should not have prices and quality reduced. Only 8.8% wanted prices decreased, while 31.1% wanted prices and quality increased. Maintaining the price and quality level was the preferred option for 60.1% of the respondents. Here is another opportunity to discuss non-sampling error. Selection bias could cause problems here. It is possible that if prices and quality were decreased more general community members might attend. At this point the planners don't know. However, to the extent that the opening ceremonies should be designed to meet the needs and wants of participants and their families, this may not be an issue.

Objective 3: Determine satisfaction levels of USG participants with their Cedar City experience.

Conclusions: Based on the frequency distribution tables it appears that lodging options and prices are where there may be a problem. Note that 19.6% of those that responded to this question were dissatisfied with lodging options, and 20.2% were dissatisfied with lodging prices. With regard to dissatisfaction with lodging options it is not clear what has led to respondent dissatisfaction. It could be the difficulty of obtaining a reservation, with the quality of the facilities, or something else. This might be a fruitful area for someone to pursue.

Following lodging, the next area of potential concern, based on frequency distributions, concerns satisfaction levels with activities between events. Dissatisfaction was registered by 11.1% of the respondents, and only 7.6% were very satisfied. It is worth pointing out that when sample means are compared, the sample mean for activities between events is the lowest in table 23. A question that can be raised in regard to this is, Are the differences in means statistically significant?

Results to the question regarding overall satisfaction with the summer games suggest that the games are successful: More than 92% are satisfied or very satisfied, and only 3.7% are dissatisfied or very dissatisfied. It is worth mentioning that as with the responses to all the questions asked, we do not really have a standard for comparison. Note that the sample mean to the overall question is not overly high when compared to other means in table 23. The fact that fewer people are very satisfied as compared to satisfied, and that more respondents are very dissatisfied than simply dissatisfied implies that there may be some areas for improvement. Tackling this issue would seem to require some in-depth and focus group interviews.

Objective 4: Determine what other activities visitors participated in while at the USGs.

Conclusions: Table 22 reveals that not many people tour the SUU campus, although this may be a function of the ambiguity of question 6. Many respondents may have not indicated they toured the campus because they did not have an officially guided tour: "walking around the campus" may not have been interpreted by many respondents as a tour. The big activities were shopping and visiting with friends and relatives in the local area.

With regard to recommendations these will likely vary considerably. However, the results do suggest the opportunity to gain useful knowledge through follow-up research.

REFERENCES

Dillman, Don A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: John Wiley & Sons.

Salant, Prescilla and Don A. Dillman (1994). How to Conduct Your Own Survey. New York: John Wiley & Sons.

Wayne A. Roberts, Jr., Southern Utah University Emmett Steed, Southern Utah University