Applied Marketing Research A  JOURNAL FOR PRACTITIONERS
Third Quarter 1990 € A Journal of the Marketing Research Association € Volume 30, No. 3 Interview With Susan Pogash On the Problem of Professional Respondents

EDITOR’S NOTE

N.O.R.C., where I learned the survey research craft, maintained a nationwide cadre of interviewers instructed not to interview anyone who had previously participated in a study.

When I was a young adult, virginity was still the norm. Again, it was equally true in interpersonal relationships as it was in data collection. This note is not a commentary about changes in the nation’s social mores. It is a comment on, and a statement of concern regarding that vanished breed, the virgin respondent.

Virgin respondents are indeed on the endangered species list. Underlying this reality is the apparently unquenchable thirst for information needed by marketers, politics and all other manner of folk who make decisions based on the professed perceptions, attitudes and behaviors of the consuming public. The demand for this information is ubiquitous. The situation is complicated by the reality that the data users are so numerous, while the consumer population is limited. There are only 240 million Americans, and not all of them fit into the sample frame of the majority of our studies. Most data collection occurs in a select number of markets – New York, Southern California, Chicago, Atlanta, Philadelphia, the San Francisco Bay Area, Denver, Dallas and three or four others. And this is as it should be. 1990 Census data will show that most consumers are living in 50 counties, the counties containing our major metropolitan areas. These individuals not only make up the mass of the market, they are also the innovators and early adopters of products and services.

Given the limited geography of these 50 counties, there is a limited population on which to draw. The results are inevitable. Individuals will have had previous experience as respondents. Worse, since we now pay our respondents for their services, there has arisen a new group in the underground economy – the professional respondent. These individuals must be discouraged. Members of the industry who have been able to determine how to stop this deception must come forward and share.

While virgins are a vanishing breed and we must adjust to that reality, it is nowhere written that we have to accommodate our businesses to ladies (and gentlemen) of the evening who only seek out participation for its monetary rewards, and in so doing, disrupt and destroy the organized inquiry that is marketing research.

This issue of Applied Marketing Research is a first attempt to consider this dilemma. The One-On-One with Susan Pogash focuses on the issue and many of the articles address it, too. Future issues will also offer insights and address the challenge. Your comments and observations are solicited.

Marshall Ottenfeld
Editor

AMR

One-On-One

Frequent Repeat Respondents

AMR Interviews Susan Pogash, Susan Pogash Marketing Research

AMR: With this One-On-One, we welcome Susan Pogash back to the pages of Applied Marketing Research. So tell our readers how you became a marketing research professional.

POGASH: I graduated from Skidmore College with a degree in Psychology and a minor in Political Science. My career from then until now has always been research and marketing. I was with the Washington, D.C., office of the Peace Corps in the late ’60s doing research. I subsequently worked for think tanks in Washington and Santa Barbara, designing studies, writing proposals and implementing projects.

After doing graduate work in Business Administration at George Washington University, I became a research manager of Landor Associates in San Francisco. The firm is engaged in package design and corporate identity consulting. I have also been in account management with Bozell & Jacobs, an advertising agency, and product management at Swift & Company.

I started my own firm in 1982. My firm specializes in new product/concept development and evaluation, image research, packaging and marketing problem solving. In addition to pursuing my own firm’s business, I am active in the Qualitative Research Consultants Association and have chaired the organization’s convention for the past two years.

AMR: Earlier this year, Susan, you chaired a meeting of the Chicago QRCA group that discussed the professional respondent problem currently facing the industry. The purpose of this One-On-One is to obtain your perspective on the problem and to provide our readers with insights into the moderator’s point-of-view regarding the problem’s solutions.

Let’s begin our conversation with a definition of terms. What is the problem we are facing and how are you defining the individual we will be discussing?

POGASH: We are talking about the problem of the frequent repeating respondent who is a cheater. The issue here is not that we are recruiting past participants, or that we should be recruiting more virgin respondents. Our objective is not to limit ourselves to virgins. We are far more reasonable and realistic.

Depending on the study, client and researchers may choose to include individuals who have participated frequently in focus groups. This is their prerogative.

The operative descriptor here is that these past participants are individuals who knowingly lie about themselves, in some fashion, in order that they can gain admission to a focus group.

They may lie about the time since they last participated in a focus group. For example, when the recruiter asks if they have participated in a focus group, or any other form of marketing research within the past 12 months, they may answer “no,” when in fact they have participated in one or even several studies within that time. Or they may claim false brand use, age grouping, or any other screening criteria they think will result in their being invited to participate.

AMR: Why IS this such an important problem?

POGASH: Increasingly, cheater respondents are showing up in focus groups with great regularity. Their appearance used to be a rarity. Today, they must be ferreted out of our studies, and that takes ingenuity.

The problem is serious because they should not be there. Their presence in the group is a breach of contract between the field service and the moderator, as well as between the moderator and the client.

These individuals should not be in our groups because their presence can contaminate the group and skew the results. These individuals are familiar with the group process, and they can use this knowledge deceptively, mischievously, or for their own amusement. Whatever their intent, they are tampering with the data and compromising the findings, findings which are used to decide key marketing research issues. This puts the integrity of the research process in question.

Suppose there are two cheater respondents in a group of 10 people and they give misinformation to gain admittance. Further, as members of the group, they continue to answer incorrectly. The group has not only lost its value to the client, it is a hazard to marketing considerations. It is not only that the wrong decisions could be made, but the decision could have significant fiduciary ramifications.

Consider also that these individuals are leader types. They could use their familiarity with the group process to artificially sway fellow respondents’ attitudes. Conversely, consider that the individuals are non-responsive. They join the group, are mute for two hours and collect their money and leave. They have done as much damage to the group as the leader types.

AMR: How long has this problem been going on?

POGASH: Moderators have been aware of this problem for a very long time. It was always assumed that the problem resided with the recruiters and field facilities. It was assumed that they were taking the easy way out and meeting quotas in groups by completing them with non-qualified individuals. These unethical recruiters were motivated, it was believed, by being paid on a show rate.

As long as that was the belief, control of the problem was simple. Facilities have reputations. Some are considered scrupulous and others are considered unscrupulous. A moderator would simply shy away from facilities that were unscrupulous or used recruiters who were. And the word spread. Scrupulous facilities became known, as did those to stay away from. Moderators could choose whom to use and whom to stay away from. This choice ensured the integrity of the work being conducted.

The assumption of fault, however, has changed. Today, it is not only recruiters who can be dishonest. There are dishonest respondents. Respondent dishonesty puts an entirely new cast on the problem.

AMR: How so?

POGASH: Today, there are individuals who actually are supplementing their incomes through participation in marketing research studies. Most of these individuals attempt this participation by being dishonest. Most frequently, the recruiters and the facilities they serve are not even aware of the respondents’ deceptions. This year, situations have gotten more problematic. Reports have spread, and there are experiences documented, about so-called professional respondents who make their living as focus group respondents.

Further, there have been reports of networking among such individuals, giving each other the names and dates of upcoming sessions, complete with the screening requirements so that the facility or recruiter can be called and the “pros” signed up.

There have also been reports of the selling of lists of moderators and facilities. Also, information has been published and circulated about how to make money as a respondent by doing study after study.

Moderators know this is happening because my colleagues and I are receiving waves of calls from individuals asking to be respondents. So we know that lists are circulating.

What this means is that the blame for recruiting problems can no longer be placed at the door of a few, isolated field facilities or recruiters. The problem of such individuals appearing in groups is no longer resolved by boycotting specific facilities.

AMR: How aware of the problem is the qualitative research community? 

POGASH: Fortunately, there is a direct, positive relationship between the problem’s growth and awareness of the problem. Few, if any moderators are walking around with their heads in sand. On a formal plane, QRCA’s Field Committee is addressing issues of field performance and field-moderator relations. In addition, some local QRCA chapters have met to discuss the problem and its resolution. And, of course, it is a topic of conversation when any two moderators meet.

AMR: Who allowed this situation to happen? What led to the situation?

POGASH: That is not an easy question to answer. Nor is there only one answer. Part of the reason the situation exists has to do with the nature and structure of the qualitative research business. Moderators, field facilities and recruiters all share in the situation being a reality. Equally, we must all contribute to the solution.

The field facilities share because of the nature of their business. Any firm is reluctant to turn away business, particularly if they have a room available. As a result, they will take actions designed to fill that room. Often, there are five to ten days available to recruit two, four, six, or eight focus groups. In some cases, it may mean subcontracting recruiting to an outside party they do not control. In other cases, they may bend the requirements for admission to a group themselves. In either situation, quality has been sacrificed for expedience and the work suffers. Without quality control, the professionals are not screened out.

Moderators, too, have some responsibility. First of all, we would like to think that there are no cheaters in the groups we conduct. But the reality is, there are. How many of us tell our clients, during the bidding process, that the study will cost more and take longer if we want to be completely sure of screening out frequent repeater respondents? Making such a statement could put us at a distinct competitive disadvantage. This is a business in which we pride ourselves on our quick turnaround and timely responsiveness to out clients’ needs.

Clients, too, have a part. Clients want what they want, when they want it. Our corporate research clients are under the gun to make the research happen quickly for their marketing managers. Focus group answers are needed immediately. Management is waiting for the findings to move along with advertising, product development, and other key marketing decisions and activities.

Together these conditions leave the window open to the problem of cheater respondents.

AMR: Please elaborate on the reasons underlying the problem.

POGASH: Marketing research in general, and qualitative research in particular, operates in a crunch mode much of the time. What happens is that:

• Brand managers are under pressure from their management;

• In turn, they apply pressure on the research departments;

• Research departments place pressure on the moderators;

• Pressure is then placed on the facilities’ managements;

• They place pressure on the recruiters.

As a result, at present, recruiting is done most often from name banks. Name banks are composed of people who have previously participated or who have expressed an interest in participating. They are easier to recruit, but the opportunity for cheating is also great.

AMR: Why is having frequent repeat respondents in a group considered such a debilitating problem? After all, if you have a pair of “pros” in a group they will help the moderator get the “ball rolling.”

POGASH: Simply put, the integrity of the group and its response set far outweighs any initial camaraderie. It is the moderator’s job to establish rapport and convert rapport into interactive conversation.

Put another way, when you have someone in a group who is a frequent repeater, you have a “ringer.” You have a professional playing an amateur’s game. To accept a frequent repeater is to change the rules, including those for recruiting and group process. If the recruiting and process are changed, the outcome will change. Group dynamics will most probably occur, however, the dynamics will be different. A group member, the frequent repeater, will be an actor playing a role. However, the role played will not be of an amateur group discussion participant. Rather, it will be the role of the professional. The role could be played out as a passive or an active group member. The frequent repeater could attempt to become the socio-emotional leader or even usurp the moderator’s role as task leader. Whatever the role, it is game playing. It is a role decided prior to participation and therefore will be artificial and inappropriate. Most importantly, it will bias the contributions of others and contaminate the output.

Because the frequent repeater is knowledgeable about the process and can be suspected of influencing the process, he must also be suspected regarding content. Given that the individual knows the course of the activity, what is to preclude him or her from being less than serious in their responses and even responding in a dishonest manner? The individual who has deceived the recruiter in the screening process has no investment in telling the truth in the focus group. Rather, the investment is in hiding out, or looking good... just to make a buck.

The key is that these frequent repeaters are, by definition, dishonest and nothing they do or say can be trusted.

AMR: How can a moderator tell if there is a frequent repeater in the group? What are the signs that set one apart?

POGASH: Sometimes it is just a feeling. You feel uneasy about an individual. Other times, it is a mix of behaviors or attitudes. Frequent repeaters tend to be jaded about the subject matter and the process. Or, they are overly cocky or even feisty.

They may arrive late, thereby proscribing their participation in rapport-setting. Sometimes they are angry or belligerent toward the moderator.

Frequently, they use sophisticated marketing or advertising jargon. Equally as often, they do not participate unless directly questioned.

The behavior has a pattern. It is never one thing, but rather a constellation of behaviors.

Infrequently but on occasion, the behavior is more blatant. The same face appears on consecutive nights (and it’s not because facilities have a high proclivity for recruiting identical twins). The individual uses one name in introduction and another name later in the evening (and it wasn’t because people in the focus groups inordinately focus on the names of friends).

In-group solutions are few. Individuals under suspicion can either be removed or silenced. Each approach is not without risk, and the action is a disruption of group process and has a negative effect on group morale. At the same time it undermines the morale of the moderator and the client. Each is left to question the legitimacy of the entire group, the validity of the study and the value of the focus group process itself.

AMR: Are there solutions?

POGASH: Solutions do exist. The solutions that will be effective will take time and they will require investment. They will also require everyone paying close attention to detail. They may even require alterations in research design.

Everyone must share in the responsibility. The problem has gotten too big, too pervasive for anyone affected to sit back and say, “Let the moderators (recruiters/facilities) fix it.”

AMR: Is there any resistance to augmenting a solution?

POGASH: Yes. I have personally witnessed resistance on the part of both moderators and field facilities. Some moderators and facilities are reluctant to inconvenience the respondent in any way... even if it means a cleaner sample.

This attitude perpetuates and even nurtures the problem. It encourages individuals to be “professionals.” Moderators and facilities should project a different attitude. The attitude projected should indicate that respondents are accountable for treating us fairly.

The nature of the relationship between the facilities and moderators must be understood for what it is – a business relationship. In business, people are held accountable for their actions. And there are consequences if they are not accountable. Further, solutions to problems are accomplished on a pragmatic basis rather that an emotional one. Respondents are paid for their time. In agreeing to participate they enter into a contract. They are NOT doing us a favor. For us to view the relationship as a set of reciprocal favors is crazy.

AMR: What solutions to you suggest? How can the moderators, recruiters and facilities act in concert to alleviate this problem?

POGASH: The solutions are on many levels. The first level is to employ the informal moderator network as a clearing house identifying facilities that have less than acceptable standards. These are the facilities that let dishonest or frequent repeating respondents into the group. Usually, such behavior occurs because the recruiter is unable to complete the quota. I believe every recruiter knows people that they can use to fill in if they have a need... and some use them. If moderators share the names of facilities that have less than acceptable standards; and facilities share the names of recruiters that have less than acceptable standards; and recruiters share the names of frequent repeaters, the problem will be significantly diminished.

A potent solution is to require identification with a photo I.D. Almost all adults have a driver’s license and these documents have photos. This approach would work if the moderators would require respondents to bring a photo I.D. to the session and it were used to screen group members on-site. Using a photo I.D., name, address, birthdate and face can be checked against the information on the screening form. If the respondent fails to bring the document or the screening information is not verified by the I.D., the respondent would be sent home (without being paid a gratuity). If this approach is used, the respondent must be notified of that fact as part of the screening.

Handled properly in a business-like manner, there is no suggestion of mistrust. Most importantly, if all moderators and facilities insisted that photo I.D.s be standard practice, this approach would flush the repeater respondents out of the system. If only done by a few, the result would be that the repeater respondents would move to more tolerant facilities. 

SIGMA is another safeguard. The Sigma Validation system is affordable to moderators whether the cost is passed on or not. The problem is that few moderators are aware of the service. Sigma allows a moderator to cross-check the activity of respondents across other studies conducted in a specified time and geographic framework.

There are other elements to the solution. Some of the most effective are:

1. Screeners should be worded more carefully. Using crafted wording, frequent responders can be discovered. As an example, B. Angell & Associates asks: “Is there anyone in your household who has never participated in a focus group?”

2. Another aid would be for moderators and facilities to insist on a minimum lead time to recruit a study. Recruiting takes time. Clients should be alerted to that time frame and the quality compromises that may occur if that time period is not respected.

3. Moderators should use only facilities having computerized respondent banks. These systems allow the facility to cross-check respondents by name, telephone number and address. That way, Bob Jones can be cross-checked against R.J. Jones and 123-4567 can be checked to determine how many names it is registered to.

4. Clients must be brought into the solution equation. They must be alerted that the problem exists and the degree to which the pressures of time influences group quality.

Back