skip navigation

CHS Research News
Vol 18, Issue 4, Fall 2006

Feature Article

CHS Survey Program innovates to improve data quality

By Katie Saunders

A CHS study made news recently with its report that in a random sample of more than 3,400 women members of Group Health, 44 percent said they had experienced some form of intimate partner violence in their adult lifetime. Perhaps overlooked in the wake of this sobering fact is the significant challenge involved in gathering such sensitive information from over 3,000 women. This task fell to the CHS Survey Research Program.

The domestic violence study is just one example of the many studies the Survey Program has fielded over the 18 years since its founding in 1988. Currently under the direction of manager Cheryl Wiese, MA, the program offers a range of services including:

• consulting on contact materials
•  piloting instruments
•  mailing advance letters
•  determining eligibility
•  interviewing subjects
•  preparing datasets for analysis
•  entering data
•  scanning surveys
•  facilitating focus groups

Some Survey Program customers turn over all aspects of subject recruitment and data collection to the program; others pick and choose among the services, perhaps opting to collect the data themselves, but have the Survey Program perform data entry.

Just as the Survey Program’s services have expanded since its inception, so have its clients, which include CHS investigators, Group Health departments, the University of Washington, and other entities. In fact, the Survey Program has become a go-to resource for the HMO Research Network (HMORN), a group of 15 HMO-based research programs, including CHS, that collaborates on a broad range of large studies.

Says CHS Executive Director Eric Larson, "I was really pleased and proud when a scientist from NHGRI (National Human Genome Research Institute) specified that they wanted the survey work on an HMORN-related project to be accomplished by our Center's survey department. This endorses an important achievement: Day in and day out the quality of our survey work is first rate."

Larson hit the nail on the head—data quality is the centerpiece of the Survey Program. Says Survey Program manager Cheryl Wiese, "My goal is to increase the quality of the data collected by the Center, whether a project team is doing it itself or whether they're using survey. I see it as my job and my mission that we help raise the quality of data overall."

Top

Experience, skills, and the right technology

And how does the Survey Program ensure quality? First and foremost is the program's skilled staff. Core staff include Project Managers Julia Anderson and Paula Sandler, data editors Melanie Mamiya and Ellen Schartz, programmer John Parker, phone room manager Dottie Oliver, and interviewer supervisor Joetta Mattson. Wiese feels fortunate to have inherited such a talented staff when she joined the Survey Program three years ago. With cumulative years at CHS approaching 80 years, the core staff has pretty much seen it all. Consider that they have worked on studies involving domestic violence, puberty, depression, safe sex, pediatric asthma, back pain, diabetes, and many more.

In addition to its core staff, the Survey Program employs from 12 to 40 interviewers, depending on the number and size of studies in the field. Being on the "front lines," interviewers are critical to the Survey Program’s commitment to high quality data. Before every study, they receive 8 to 12 hours of classroom training followed by a certification process that requires conducting a mistake-free practice interview. Once they are in the field, "lead" interviewers monitor 10-15 percent of completed interviews to ensure that data quality remains high.

Interview data quality is also enhanced by technology—in this case Computer Assisted Telephone Interviewing (CATI). This software allows interviewers to directly enter subject responses into a computer, eliminating the need for costly data entry that's required for paper-administered surveys. Further, CATI is programmed to automatically handle skip patterns and check for data that seems far from normal ranges—steps that enhance interviewer accuracy, especially when the surveys are long and complex. Pointing out that some members of the HMO Research Network still rely on paper surveys, Wiese says, "It's become second nature for us to have an interview programmed [in CATI] and for the interviewers to flow through it and read what's on the screen."

While CATI has been around for a while, the field of survey research continues to push the envelope in terms of new technologies and methodological experiments. Recognizing the importance of staying on top of these innovations, in the last two years Wiese arranged for survey luminaries to visit the Center and share their expertise. One is Washington State University’s Don Dillman, author of the "bible" in the field, Mail and Telephone Surveys: the Total Design Method. The other is University of Michigan’s Mick Couper, an expert in Web-based surveys and non-response.

Top

Conducting research on survey methodology

Following the example of these experts, the Survey Program and Wiese, who describes herself as "passionate about survey research," have conducted their own methodological experiments to improve data quality. For example, in a recent study of the relationship between depression and obesity, they compared response rates achieved when respondents received a $5 "pre-incentive" gift card (prior to agreeing to be in the study) with rates obtained with a $5 "post-incentive" gift card—a thank you for having enrolled in the study. The obvious advantage to post-incentives is that only actual participants are paid. However, the experiment revealed that fewer advance letters would have been necessary if all had received the “pre-incentive” only. This increased efficiency in subject recruitment would have translated into cost savings, even taking into account people who received the money up-front and did not enroll in the study.

As an aside, Wiese explains that a pre-incentive in the form of a $2 bill is far more effective than a $5 gift card pre-incentive. She explains that pre-incentives are not intended to compensate a subject for study participation but rather work as attention grabbers that encourage potential subjects to thoroughly read the letter. A $2 bill is more attention getting than a $5 gift card—and more noteworthy than two $1 bills.

In another opportunistic experiment, the Survey Program analyzed the impact of mixing data collection modes. That is, the study design called for subjects to return a mailed survey; if they did not respond, interviewers attempted to conduct the interview by phone. Wiese and colleagues found that mixing modes had an adverse impact on data quality because subjects responded differently to written vs. oral questions. Wiese recommends sticking to a single mode of data collection; if this is not possible, she advises keeping the modes as similar as possible (e.g., print and Web surveys are more alike than print and telephone surveys).

Another Survey Program data quality procedure is to match qualities of the interviewers to those of the population being surveyed based on research showing that people answer more candidly if they feel an affinity with the interviewer. For example, southerners have been shown to be more willing to talk to an interviewer with a southern accent. As part of the HMORN Eat for Life study, the Survey Program interviewed African Americans members of the Henry Ford health plan in Detroit and Kaiser Permanente in Atlanta about their consumption of fruits and vegetables. Because the interview delved deeply into issues of ethnic identity, the Survey Program used African American interviewers. Similarly, the program employs female interviewers in studies of women's health issues such as mammography or hormone replacement therapy.

Maximizing response rates, and thus the representativeness of the data, is integral to data quality. Wiese says the biggest hurdle to achieving high response rates is an increasing difficulty in reaching people; once contact is made, people tend to participate. To optimize reach, Survey Program staff place up to 14 calls to local subjects at varying times between 9 a.m. and 9 p.m. seven days a week. In fact, the phone room is "often abuzz" as early as 6 a.m. in order to reach people on the East coast who request morning calls. The strategy appears to work. Wiese says she is thrilled with the response rates of 60–70 percent achieved on Survey Program studies, especially compared to the rates of 35–40 percent she witnessed in her "former life" conducting surveys among the general population while working at the University of Nebraska Bureau of Sociological Research.

Wiese sees the Survey Program as a resource to ensure data quality—even when the program is not performing the data collection itself. Specifically, the program could provide contact protocols, training expertise, and training manuals to help standardize data collection across multiple sites. Wiese’s goal aligns with the mission of the Coordinated Clinical Studies Network, an HMORN effort led by CHS Executive Director Larson to facilitate the conduct of multi-site clinical trials by standardizing processes such as human subjects applications, information systems, and surveys.

Top

More multi-site studies on the horizon

Asked about future goals for the Survey Program, Wiese mentions continued involvement with HMORN studies, either as a resource for data quality or as a data collector. “Having these experiences and having them come off as positive is such a good thing for survey and for the whole idea of doing multi-site studies,” she says. Currently in the offing is a genome study which involves asking members of Henry Ford Health Systems about their perceptions of the source of disease—e.g., to what extent do genes vs. behaviors play a role.

Wiese also sees more Survey Program involvement with Group Health in the future. The program recently interviewed a random sample of 800 Group Health members for Project WIN. The purpose was to assess the health status and rates of “risky” behaviors (e.g., smoking, excessive drinking) of Cooperative members. Because the survey was based on questions from the Center for Disease Control’s (CDC) Behavioral Risk Factor Surveillance Survey (BRFSS), we now have data to compare to the population of Washington State as well as nationally.

And, of course, the Survey Program will continue to offer its services to Center investigators and staff. "I'd like for the first thought of investigators to be 'how can we involve the Survey Program to improve our research,' " Wiese says. It appears that she and the Survey Program are well on the way to achieving that goal.

Top