New data-collection tools that elicit consistent responses on health issues will be ready this fall.

“On how many of the past seven days did you eat breakfast?” If I were to ask you that question, how would you answer? More importantly, how would you interpret the question? Over the past few months, Success Measures, a social enterprise at NeighborWorks America specializing in community development evaluation, has been testing seemingly straightforward questions like the one above to better understand how to ask people about their status, attitudes and behaviors related to health; about their access to and use of health care services; and about their housing conditions related to health. What we have learned is that a seemingly straightforward question like the one above is anything but.

[figure][/figure]

In 2014, in response to a new initiative at NeighborWorks and interest from community development practitioners, we began to develop a comprehensive set of data-collection tools to help organizations further understand the health-related outcomes of their community development work. These community development organizations had begun to focus on both the relevance of their existing work to long-term health outcomes, and the potential for increasing impact by incorporating health into their work. Based on the bodies of research about social determinants of health and health equity, we assembled a group of experts in related fields and developed a framework that included outcome measures in three main categories:

1. Individual and household.
2. Home, work, and the physical environment.
3. Organizational measures to better understand partnership and collaboration.

In June, after significant research, input, and resources from our working group and other stakeholders, our team completed drafts of 47 data-collection tools to cover the topics identified in the framework. Working with a research consultant, we then developed a field test methodology to test the tools for a variety of factors that affect validity and/or reliability, including cultural relevance, bias, inconsistent question interpretation, and inability to respond.

To conduct the field test of our draft tools, we recruited nine community-based organizations that are already pioneering health-related programming in the context of community development. The sites represent a range of demographics including age, race, and ethnicity, and were located in both urban and rural communities across the country. Field testing has included focus groups, cognitive interviewing, debriefing of individual written responses, and retesting the tools for reliability. All of those methods allowed us to talk with community residents and better understand how they think about health; what other aspects of their lives they connect it to; and how to frame questions so that they are engaging and thought-provoking but also straightforward and easy to understand cross-culturally. As of today, we are in the final phase of field-testing and will release the tools in late fall.

So far, our learning curve has been steep; the people we have interviewed have given us a great deal of insight into identifying clearer and more appropriate ways of asking things practitioners are hoping to find out. Questions we thought would be straightforward, like “on how many of the past seven days did you cook your own dinner?”, turn out to reflect a bias for a nuclear family structure and do not account for the sharing of home-cooked meals with households or nearby family members. We discovered that while a respondent’s answer might be “three nights a week,” they might in fact be eating home-cooked meals on other nights outside the home. We eventually shifted the question to: “Thinking about the past seven days, on how many days did you eat a home-cooked dinner?”, which has garnered consistent responses.

When we posed questions about managing long-term illness or disease, respondents told us to specifically include autoimmune conditions like rheumatoid arthritis, fibromyalgia, and lupus in addition to the traditional research-based questions about diabetes and high blood pressure. Respondents in multiple locations felt that autoimmune conditions were prevalent in their communities but, until recently, had gone undiagnosed or unnoticed, although they affect their ability to work and travel.

Returning to the original example, “On how many of the past seven days did you eat breakfast?”, some respondents considered eating breakfast foods at other times of the day the equivalent of “eating breakfast” and others were not counting morning meals with non-breakfast foods. We have found that asking, “Thinking about the past seven days, on how many days did you eat breakfast first thing in the morning?”, has yielded consistent answers across respondents and will help practitioners understand what they are really looking to know: if the individuals and households they serve are getting that important first meal of the day.

The few examples mentioned here highlight only some of the assumptions we, as practitioners and researchers, make about behavior and health. As we unpack these nuances within the framework of survey questions for an evaluation, we continue to refine them and intend to share them in written form at the end of the field test, in the hope that they add to the field and influence practices and programs.

If you would like to receive updates and an announcement when the tools are released, please send an email to jmulcahy@nw.org with the subject “Health Tools.”

Jessica Mulcahy, senior manager of evaluation and research at Success Measures, is a cultural anthropologist with significant experience in mixed-method evaluation. She directs the development of Success Measures’ shared-indicator projects and leads the enterprise’s evaluation research efforts. Jessica also manages custom consulting evaluation projects for national and regional funders, intermediaries, and funding collaboratives. For more information please contact Jessica Mulcahy at jmulcahy@nw.org or 617-585-5075. 

Success Measures, a specialized outcome-evaluation resource for the community development field and philanthropy, is based at NeighborWorks America, a national nonprofit affordable housing and intermediary organization. Operating as a fee-for-service social enterprise, Success Measures offers an integrated set of evaluation, capacity building, advisory, and technology services for community-based organizations, intermediaries, funders, and funder collaboratives seeking to learn from and demonstrate the results of their programs and investments in communities and metropolitan regions. For more information: www.successmeasures.org

Homepage photo/Ed Yourdon