SPORTS PARTICIPATION RESEARCH: NOT YET A SCIENCE

By Harvey Lauer

To paraphrase the famous comment on democracy, survey research is the very worst way to measure sports participation — but it's the best one I've seen yet!

Sports Participation, when compared with the many bland topics of conventional marketing research (taste tests, shopping diaries, retail point-of-sale tabulations, etc.), is perceived as more interesting and entertaining to researchers and study populations alike; but while sports participation does have a certain appeal, its sexier subject matter confers no advantages on the research process. The rules and principles of sampling and question writing are governed by the same orthodoxies that rule more prosaic branches of research, and if pressed to name the first law of questionnaire design, ordinary researchers and sports researchers should both respond with the familiar imperative: "Ask people questions they can answer".

SPORTS PARTICIPATION — THE METRIC OF CHOICE

Dollar sales volume — as in a $61 billion soft drink market, $183 billion prescription drug sector or $8.6 billion cell phone market — is a popular way of depicting the size of a product. In some categories, markets are more vividly portrayed by participation behavior. 85 million gardening households, 159,000 professional photographers, 32 million crossword puzzle aficionados or 50 million frequent fitness participants are handy statistics — for some purposes, more valuable than fragmented, indecipherable purchase data, lacking context. When products are consumed on a frequent basis, or by nearly everyone, it often makes sense to think about the size of a market or industry volumetrically — by product units rather than people. Hence, annual U.S. consumption of 1.5 billion gallons of orange juice is arguably more utilitarian than say, a near-unanimity of 250 million orange juice drinkers.

In the sporting goods industry, markets are usually described in terms of participation behavior. While the number of Baseball bats, gloves, and helmets are indispensable statistics, one hears more often about the population of 11 million Baseball players — not about the equally important measurement of product purchases. On the other hand, the U.S. athletic shoe market is not readily captured by sports participants, because they represent only a minority of buyers; as I popularized twenty years ago, 80% of all athletic/sports shoes are never sweated in. But this is another matter.

In sports participation, the convention is to describe a population in terms of 12 million Beach Volleyball players, not as a volumetric behavioral estimate of what might be called "tonnage" — e.g. 271 million aggregate national participation days or Volleyball experiences per year. Tonnage is often a more precise measure of sports participation, but too awkward for practical usage. Marketing professionals have difficulty operationalizing "2.6 billion aggregate Running days in 2002"; cognitively, 36 million Runners is a far more riveting and actionable metric, translating easily into demographics, products and markets.

WHO IS A "PLAYER"?

To discover the approximate number of Baseball players, Gymnasts or Stationary Cyclists in the U.S., we need to define "player" or participant; we then conduct a survey of the entire population. It is not sufficient to ask whether or not respondents "play Baseball", regard themselves as "Billiards players", or claim to be "Surfers"; at best, subjective self-descriptions are loose and imprecise. A second convention is to define a sport/activity participant (e.g. Treadmill user, Yoga practitioner or Lacrosse player, etc.) as someone who participated in the activity at least once in the last 12 months. Aside from the relative handful of respondents who may have participated in a sport only once or twice about a year ago, this question is neither daunting nor freighted with ambiguity. If all else goes well, such straightforward queries (even requiring 12-month recall) produce viable measurements of sports populations. Most of us know if we Bowled at least once in the past year, or did not; if we Ran at least once, or did not, etc.

THE "TELESCOPING" PROBLEM

Still, minor distortions are inescapable, even for simple measurements of sports participation. A small number of respondents succumb to "telescoping" — incorrectly recalling for example, that a single, isolated Bowling night 15 months ago happened within the "last 12 months". Or perhaps the person does remember the exact date of the "non-qualifying" Bowling experience... but that it occurred outside the 12-month time-frame is utterly beside the point; he or she simply wants credit for it!

Occasional sports participants are more susceptible to telescoping, and therefore are more likely to artificially inflate the measurement of sports populations. Sports that are played or participated in less often (e.g. Skiing, Scuba Diving, etc.) are at higher risk for this type of distortion than say, Running or Walking — sports/activities that are indulged in more frequently. This is because occasional or infrequent participants — the "at-risk" elements — constitute a larger proportion of "smaller" sports, and a smaller percentage of "larger" sport populations.

In 2002 for example, a majority of the 3.3 million projected Scuba Divers engaged in that activity fewer than four times, suggesting a high potential for inflated measurement — if the dates of some occasional experiences (which occurred more than a year ago) are incorrectly reported. While rare events are more salient and easier to date — especially when they happen on a winter vacation, or during some other particularly memorable occasion — there is little doubt that certain of these distinctive experiences are deliberately misplaced in time (included in the 12-month recall period).

From this point of view, telescoping is less vexing in higher-frequency sports and activities. Only 5% of our 35.9 million Runners participated fewer than four times; 63% ran more than 25 times. It would seem therefore, that whether or not one has run "at least once" in the last year is a less challenging cognitive task, (and also one that invites less dissembling); most alleged "Runners" are truly Runners.

Memorable sports participation events may be even more susceptible to the telescoping of time — even though the amateur athlete can accurately date such occasions. Very often, a marathon run 18 months ago will, in the mind of the respondent, have occurred within "the last 12 months". This is not memory distortion at a preconscious level, but far more often, a simple conscious desire to receive credit for a prodigious athletic feat…and precisely when it happened is totally irrelevant. Once again truth becomes a casualty, and unsuspecting consumer researchers who measure "marathons" or "triathlons" will be astonished at inexplicably large numbers...

ANTIDOTES TO TELESCOPING: SPURIOUS OR UNAVAILABLE

Critics of sports participation research point to the difficulty, if not futility of 12-month recall; it is absurd they say, to think that a person might accurately recall the number of times he or she has Skied, Golfed, or visited a Health Club in the last 12 months. A solution, they argue, is a three-part question: number of days per week; number of weeks per month; number of months per year. Here, computerized multiplication upgrades the respondent's scribbled margin notes, ostensibly providing a more "accurate" estimate. But in the end, this "refinement" confers no material advantage: the result will be an inflated number — similar to the respondent's answer to a single open-ended question about the "last 12 months".

In a noble but misguided effort to side-step long recall periods, undergraduates in sports management courses (and all too many professional researchers) innocently ask: "how many times per week do you participate in"…. This approach discounts seasonal, sporadic, or occasional participation, and assumes the respondent is a 52-week year-round participant who faithfully adheres to this weekly regimen. Needless to say, such data are worthless.

On occasion, when abundant resources have permitted more frequent surveys and compacted research periods (e.g. monthly, weekly or even daily surveys) members of the general research community innocently believed they had finally outwitted 12-month recall. By simply adding the daily, weekly or monthly results of these more frequent surveys with shorter recall periods, they blissfully discovered the Holy Grail of accurate, annual sports participation projections.

This is a staggering failure of logic. We cannot merely sum 12 individual months, 52 individual weeks, or 365 individual days of independently surveyed sports participation and arrive at an annual grand total; any such aggregation will always exceed the true number. Twelve monthly surveys of Basketball players cannot be added for an estimate of how many people played the sport in the past 12 months — without double-counting! The simplest illustration: If there were two days in a year and three people in the U.S. (one of whom had biked on both days, another on one day, and the third not at all), there would obviously be two cyclists in this miniature nation — a result obtainable only through an annual survey asking about any cycling activity throughout the entire (two-day) year. A survey conducted for Day 1 (showing two cyclists) added to an independent survey of Day 2 (showing one cyclist) will fallaciously yield three cyclists — not the true "population" of two cyclists.

It would be permissible to add volumes of products — such as the total pairs of athletic shoes purchased or total gallons of orange juice consumed. But we cannot add shoe buyers, orange juice drinkers, or Runners.

FREQUENCY OF PARTICIPATION — OVERSTATED, BUT USABLE

Telescoping, to varying degrees, will result in the artificial elevation of sports populations — how many people participate in a sport; but for the bigger sports, these overstatements are not significant. Far more serious exaggerations occur in self-reports of participation frequency — how often people play or engage in a sport/activity, usually in days (times) per year. Memory is quite fallible, and in most behavioral measurements — certainly in the case of sports participation — human error favors the higher number. Unless we have the luxury of a daily consumer diary, the recall problem — be it innocent memory distortion or deliberate dissimulation — cannot be avoided.

Respondents are quite consistent in overestimates of total sports participation days, much higher than "objectively" derived measures from lift-ticket sales at Ski resorts, card-swipes at Health Clubs, or register receipts of daily fees at public Golf courses. On the other hand, these more "objective" nose counts — for a multitude of reasons — are never useful in projecting the size of a sports population. For example, the entire universe of Ski areas might not be captured (or known); informal, off-site participation (e.g. in Paintball, Ice-Skating, Snowboarding, etc.) may be huge; gratis, non-paying seniors may go uncounted; Bowlers may own, not rent shoes; people may hunt or fish without licenses, etc. Virtually any market or product-based data compilation will fall short of Divinely ordained sports participation.

In 2004, the crude and inexact method of consumer recall is the only practical methodology with which to gauge aggregate sports participation behavior on a national scale. Bloated results cannot be adjudicated by weighting procedures, and the only form of damage control is an awareness that "average number of participation days" (not total number of participants) are likely overstated 20% - 50%, across-the-board, regardless of research method. In very long self-administered surveys — where, for a very different reason, the total numbers of participants are underestimated — there may be a serendipitous cancellation effect, accidentally producing a slightly more "accurate" projection of sports participation.

For aggregate sports participation (persons multiplied by number of days), the golden mean lies somewhere between the extremes of survey research and various forms of industry or product-based reckoning.

THE EFFECT OF TELESCOPING: A UNIQUE QUANTIFICATION

The National Human Activity Pattern Survey (NHAPS), conducted by the Survey Research Center at the University of Maryland from October 1992 - September 1994, provides a rare opportunity to gauge the extent of telescoping. NHAPS interviewed 7,515 adults on a daily basis, employing a 24-hour recall methodology. Respondents were asked to enumerate each and every activity (e.g. sleeping, eating, driving, exercise, etc.) performed on the previous day. The research objective was to quantify total energy expenditure by Americans, with activity categories taken from a "Compendium of Physical Activities", developed at the University of Minnesota. The data were then assiduously coded and converted into aggregate units of energy. This landmark research has not been replicated, but its 24-hour recall methodology remains a unique standard for assessing the effects of telescoping in sports participation research.

Unfortunately, the NHAPS study delineated only a handful of specific sports/activities, such as Basketball, Bowling, Tennis, etc. The vast majority of sports and activities were lost to either compound measurements (e.g. Fishing/Hunting), or miscellaneous aggregate categories, such as "Other heavy sports/exercise". In the end, only 5 meaningful comparisons could be made with ASD participation data from approximately the same period.

If we accept NHAPS as the gold standard, redemption for 12-month recall is at hand only for Golf. Over a two-year period, 63 of the 7,515 adult respondents reported playing Golf on the previous day. This microscopic daily average Golf participation (.0084), when multiplied by the 1993 adult population of 185.4 million, yields an average of 1,557,360 Golfers playing on any single day. This figure, multiplied by 365 days per year, results in an aggregate 568,305,000 total player/days, the only comfort for an independent 12-month recall comparison: approximating 582,806,000 aggregate Golf days were projected by ASD for 1993. This unusual and inexplicably accurate Golf recall has been noticed elsewhere, and the only plausible explanation is tongue-and-cheek: Golfers are more likely than any other sports participants to repress painful memories — thereby reporting fewer outings, and as a corollary, attenuating the telescoping problem!

The five available comparisons between ASD and NHAPS data were the following:

RESPONDENT RECOGNITION: THE NEED TO BE "INCLUDED"

It is axiomatic that people want to be recognized for athletic achievements. This is true not only for high-school swim records, college football rosters, or later-in-life 5K race times, but for mundane sports/fitness participation of no particular moment, except to the respondent who, quite understandably, is determined to inscribe these achievements in the annals of sports research. In a survey of Outdoors sports participation, a well-known research firm momentarily rocked the sporting goods industry with a "discovery" of over 50 million "Trail Runners"! This absurdity was the result of a fatal omission: along with the standard list of Outdoors sports and activities such as Hiking, Camping and Canoeing, the hapless questionnaire writer dutifully included "Trail Running" — but not "Running"! Ninety percent of the resultant "Trail Runners" were mere frustrated Road Runners who simply needed recognition for participation, choosing to sublimate pent up mental energy through the only available outlet in the survey instrument — Trail Running. By contrast, American Sports Data, Inc. projected a far more reasonable number (only 5.6 million Trail Runners for 2002), and of course, many, many, more Road Runners.

MAN-MADE OBSTACLES: LONG QUESTION BATTERIES

Were there really 55 million Bowlers in 2003, as proclaimed by American Sports Data, Inc.? The answer — quite apart from the universal issue of random but predictable fluctuations due to sample size — is that consumer mail panel research (like quantum physics, which tolerates the ambiguity of an electron — or anything else — in two places at the same time) allows both more and less than 55 million Bowlers.

Due to simple respondent fatigue, or more dangerously, a waning interest in panel membership, long lists of sports/activities ensure lower response rates — but only for sports appearing at the ends of long batteries. Even if we reverse the order of presentation for half the sample, the problem is attenuated, but not solved completely; any self-administered questionnaire battery in excess of 25-30 line items will understate whatever is being measured. But there is a counterweight — the telescoping effect will artificially add sports participants, restoring some of the deficit.

On the other hand, statistics derived from single questions are somewhat more immune to respondent fatigue or disinterest. For example, the ASD projection of Health Club membership (based on a single stand-alone question), aligns closely with other (triangulated) approaches; and this confluence of independent findings provides much comfort, if not spiritual fulfillment. Shorter batteries (or individual questions) generally produce higher sports participation estimates.

THE MOST TREACHEROUS PITFALL: BORED, DISINTERESTED PANELISTS

This problem is endemic only to consumer mail panel methodology, and is treated at length elsewhere. As we know, nothing is free, and the tremendous cost savings of consumer mail panel research (elimination of the interviewer) exacts a huge tradeoff — unreliable response patterns from panel members who have outlived their usefulness.

Known in the trade as "Eager Beavers", newly minted panelists faithfully execute their duties with care and precision, generally producing higher rates of sports participation (or any other reported behavior) than longer-tenured respondents. Fresh recruits tend on the whole, to be diligent and conscientious in filling out questionnaires; but whether or not Eager Beavers reflect the ultimate "true" reality may never be known — it is conceivable that these bright-eyed novices actually over-report sports participation behavior.

Indeed, recently recruited panel members — according to experimental research conducted by ASD in 1997 — reflected sports participation rates 22% higher than those of more seasoned panelists. Even less comforting was the perfect negative correlation between tenure and incidence of sports participation: with each successive year of panel membership, respondents became less attentive and fastidious to questionnaire completion — a trend paralleled by eroding sports participation rates, known as the "Lazy Dog" effect.

Great care must be taken to ensure that each year, the final sample contains roughly similar proportions of panel members in various stages of the membership cycle. This can be accomplished via sample weighting, where panel membership tenure is treated as an additional variable in the sample balancing program.

The failure to account for changing tenure composition in consumer mail panels can be fatal — producing unaccountable fluctuations in year-to-year tracking results.

RESPONDENT COOPERATION: SMARTER QUESTIONS, BETTER ANSWERS

A recent government-sponsored study offered the "illumination" that 82% of Americans aged 16+ (175 million people) "walked" at least once in the past year! This profoundly useless statistic is undoubtedly correct, but has approximately the same zero value as a national estimate of "exercisers" or the number of people "active in sports and recreation".

When questionnaires clearly define a sport or activity, people reward such detailed painstaking nomenclature with more relevant, actionable results. For example, a three-pronged measurement of Court, Grass and Beach Volleyball is far more sensible than a one-dimensional estimate of Volleyball. Over the years, the American Sports Data, Inc. SUPERSTUDY® of Sports Participation has made similar refinements to the line-item definitions of Tackle Football, Wrestling, Mountain Climbing, Bicycling, Swimming and many others. An original measurement of "Football" may have yielded 21 million players, and then, cinched by a more rigorous definition of "Tackle Football", narrowed to 10 million. With the ultimate refinement of "protective equipment", the number was halved to a more realistic estimate of only 5 million.

RESPONSE DISTORTION: A BRIEF SUMMARY

The telescoping of time is usually a form of preconscious memory distortion. Sometimes it is deliberate and conscious; people simply want credit for activity performed more than a year ago — or in rare cases, for activities never performed at all. Another type of response distortion is driven by simple ego-need: the tendency to present ourselves in the most flattering light and to inflate our achievements…in this case to exaggerate the number of participation days we report. People also need to feel "included" in a survey — especially when one of their sports is missing from the questionnaire. On occasion, they may actually forget some instances of participation, but this is rare. More commonly, research subjects, especially in mail surveys, may — for reasons unrelated to memory capacity but very much associated with lackadaisical attitudes toward the interview or a widening alienation from panel membership — simply fail to report participation in various sports and activities.

BOTTOM LINE: WHAT CAN WE TAKE TO THE BANK?

Without the cooperation of the Census Bureau, the divination of sports participation (or any national statistic) becomes exceedingly difficult, because research — subject not only to the vagaries of random sampling error — must negotiate other minefields. In general, five problems afflict sports participation research:

• Panel Tenure (Suppression of Sports Participation)

• Questionnaire Length (Suppression of Sports Participation)

• Telescoping (Elevation of Sports Participation)

• Forgetting (Suppression of Sports Participation) — Rare

• Poorly Defined Sports Categories (Neutral, but of limited Analytical Value)

Burnt out panel members will artificially depress sports participation projections, as will unduly long question batteries. Telescoping has the opposite effect, but it almost certainly does not entirely compensate for losses exacted by bored respondents or oppressive questionnaires. An educated guess is that with proper weighting procedures and reasonable questionnaire length, the results of a mail panel study will not differ greatly from similar surveys conducted with other methodologies.

We will never know the precise count of Wakeboarders or Pilates practitioners, but basically sound (if crude) methodologies are able to capture the general magnitude of a sport population. More important, consistent year-to-year methodologies can provide very accurate trending.

Due to other sources of measurement error, we know that even a Census count of Skateboarders or Mountain Bikers could not be engraved in stone. But through a "triangulation" of measurements from various other consumer surveys, magazine readership, industry product market estimates, obligatory governmental tabulations and other external sources, we know that "imperfect" survey research can produce national projections of high face validity.

The case for confirmation is strong. The 2002 ASD projection of 51.4 million Fishermen exceeds, but is perfectly consonant with the roughly 28 million Fishing licenses sold each year in the U.S. The estimate of 36 million Health Club members across the U.S. (18 million of whom are "Commercial" Health Club members) is gratifyingly close to a nationwide Yellow Pages compilation by InfoUSA. For over 15 years, ASD Health Club membership trends derived from consumer research has accurately tracked "known" Health Club expansion, as gauged by this independent service. The ASD finding of 4.1 million frequent Soccer players is an excellent proxy for the combined memberships of USYSA, AYSO and SAY Soccer. About 15 million Hunting licenses are issued each year by the 50 states…a number which dovetails neatly with 16 million Hunters projected by ASD (licenses are more obligatory in Hunting than Fishing). And although not a confirmation of validity, for Outdoors sports participation, the periodic U.S. Fish and Wildlife survey produces results very similar to those provided by ASD. To the best of our determination, the Tennis industry sells roughly 4 million tennis racquets every year — a not unreasonable contrast with the ASD estimate of 4.0 million frequent Tennis players in 2002.

THE IMPERFECT SCIENCE: SOLUTIONS FROM A PERFECT FUTURE

By the year 2050, technological progress will have taken us far beyond our quaint reliance on fallible human memory and perception. Unobtrusive monitoring devices — gleaning data from bodily implants, or perhaps even less obtrusive retinal scanning devices — will free respondents from the primitive need to remember "how many days per year"…Investigators of sports participation behavior will avoid the minefield of consumer research altogether, because hard physiological data will have finally trumped soft social science.