Quality control and judgment calls: Would you count this survey complete?

Filling quotas with high-quality completes while providing quick turnaround is the aspiration of every researcher. Unfortunately, researchers are sometimes forced to choose between the two.

Back in college when I worked as a telephone interviewer, I was calling on a B2B study that was slow-going – to put it mildly. It was a struggle getting past the gatekeepers and reaching C-level executives who were actually at their desks and willing to spend 30 minutes on the phone during the workday. So when, at 4 p.m. on a Friday afternoon a cheerful office assistant told me that the person I was trying to reach would gladly speak with me, I was jazzed.

Until he answered the phone.

“It’s Miller Time! What can I do you for?”

Stifling my laughter I explained the purpose of my call and began the interview. After the first few preliminary questions it became obvious that the respondent was extremely inebriated. (Apparently at this company happy hour starts at your desk!) Never having encountered this before and since he was still answering the questions, I continued. However, the interview unraveled about halfway through when he put me on speakerphone and began conferring with his office assistant and another unidentified man in the room about what he should say in response to each question. They all seemed pretty amused with themselves.

They stuck with me until the end and I thanked them for their time and hung up the phone. When I reached the screen at the end of the questionnaire to categorize the result (e.g., complete, disqualified, wrong number, etc.), I was stumped. Should this survey count?

I walked over to my supervisor and explained the bizarre interview. The supervisor called over the project manager, who asked me a few questions and said, “If he finished it, it’s complete.”

Researchers are expected to catch the cheaters, repeaters, straightliners, robots and whatever else is out there compromising data quality, but what about making straight judgment calls? Is one respondent’s feedback more valid than another’s? Or is a complete a complete, no questions asked? When deciding between filling a quota and controlling for quality, on which side do you err? Would you have let this survey in?

This entry was posted in Business-To-Business Research, Interviewing, Market Research Humor, Quantitative Research, Telephone Interviewing, The Business of Research. Bookmark the permalink.

5 Responses to Quality control and judgment calls: Would you count this survey complete?

  1. This interview should not count, even though it is complete. You should have told the respondent that interviews are personal and not a panel. The rule is that most Quantitative interviews are one on one and not one on two or three. Another thing, the respondent must qualify your screening questionnaires.If the respondent you were talking to cannot provide the answers you need and has to rely on another person, then he is not qualified in the first place. You thank and close, once you notice another intruder-respondents has joined the interview

  2. Trudy Fula says:

    The interview should have been thrown out. If the interviewer was able to tell that the respondent was
    inebriated, the interview should have ended with, that’s all the questions I have, and thank them for their time, no matter how much of the question had been asked. It was an error on the interviewers part to continue.

  3. Jenine Carey says:

    This interview should not have counted, nor have been completed. The interviewer should have kindly & professionally excused themselves when the interview first began to go awry. I’m not certain what other firms do, but we often replace extremely poor interviews with an additional recruit. We pride ourselves in quality research.

  4. Ron Sellers says:

    Have to agree with the others – this is not a valid interview. When I’m looking at quant data, I sometimes struggle with a respondent who seems odd (based on the responses), such as someone claiming to earn $40k a year and drive a Rolls. I’ve learned that unless there’s obvious cheating, not to discount oddball data (e.g. you can buy a 70s Rolls for a pretty low price). But in this case, the interviewer could tell that the respondent was not taking it seriously. No different than a bunch of college guys gathered around a respondent, saying, “Dude, tell him you’re a chick!” Interview should not have even been completed – politely tell the person that’s all the questions you have, thank him for his time, and move on.

  5. John Clay says:

    I agree with the others – it should not count- but I’ve also seen many supervisors make the same call and in worse circumstances. When data collection facilities get paid by the complete, the pressure is intense to get completes! All CATI vendors say they provide quality interviewers but think about research calls you get.

    How often is the interviewer nearly unintelligible? I’d say “very often”. How often does the questionnaire skip basic security questions (like do you work for a market research company?)? Often! How often are you forced to do a questionnaire that has nothing to do with you? Often! (I’ve had to do an extensive questionnaire about Tetris several times in the past few years and I’m middle-aged and haven’t played something like that in 30 plus years!). How often is the interview really a guise to sell? Too often!

    At least this respondent represented the overly-social inebriated-at-work segment of your population frame! :)!