Filling quotas with high-quality completes while providing quick turnaround is the aspiration of every researcher. Unfortunately, researchers are sometimes forced to choose between the two.
Back in college when I worked as a telephone interviewer, I was calling on a B2B study that was slow-going – to put it mildly. It was a struggle getting past the gatekeepers and reaching C-level executives who were actually at their desks and willing to spend 30 minutes on the phone during the workday. So when, at 4 p.m. on a Friday afternoon a cheerful office assistant told me that the person I was trying to reach would gladly speak with me, I was jazzed.
Until he answered the phone.
“It’s Miller Time! What can I do you for?”
Stifling my laughter I explained the purpose of my call and began the interview. After the first few preliminary questions it became obvious that the respondent was extremely inebriated. (Apparently at this company happy hour starts at your desk!) Never having encountered this before and since he was still answering the questions, I continued. However, the interview unraveled about halfway through when he put me on speakerphone and began conferring with his office assistant and another unidentified man in the room about what he should say in response to each question. They all seemed pretty amused with themselves.
They stuck with me until the end and I thanked them for their time and hung up the phone. When I reached the screen at the end of the questionnaire to categorize the result (e.g., complete, disqualified, wrong number, etc.), I was stumped. Should this survey count?
I walked over to my supervisor and explained the bizarre interview. The supervisor called over the project manager, who asked me a few questions and said, “If he finished it, it’s complete.”
Researchers are expected to catch the cheaters, repeaters, straightliners, robots and whatever else is out there compromising data quality, but what about making straight judgment calls? Is one respondent’s feedback more valid than another’s? Or is a complete a complete, no questions asked? When deciding between filling a quota and controlling for quality, on which side do you err? Would you have let this survey in?