Findings

Breadth and Strength of the Outcomes of the CRC projects
As described in our previous evaluation report (Transforming Research, 2005 ), the CRC awards facilitate meaningful inclusion of those most affected by breast cancer in the creation, implementation, and reporting of research on the disease. Each CRC project achieved impact in numerous areas, including health education/service programs, policy advocacy, the increasing of scientific and community knowledge, high visibility within the community, and significant distribution of results. Collaboration was cited by most participants as adding important value to most of the steps in the research project.

The conclusions, as reported in our previous evaluation report (Transforming Research, 2005), are:

  1. Community-based participatory research is an effective way to stimulate research in under-represented populations.
  2. The CRC research projects funded by the CBCRP all resulted in some level of positive outcomes in all four of the dimensions expected of community-based participatory research projects: impact on knowledge, programs, and policies; impact on the quality of the research; impact on community agencies and members; and impact on academic researchers.
  3. CRC awards were most effective at:
    • Improving the quality of the research (especially recruitment/retention and methods development);
    • Providing benefits to participating community-based organizations;
    • Addressing questions important to the community.
  4. CRC awards were least effective at:
    • Providing funding for community members to participate in the study;
    • Leveraging additional funding for the research teams;
    • Impacting the analysis of the data;
    • Publishing peer-reviewed journal articles;
    • Increasing academic partners’ knowledge of the community.

(Note: Some teams have anecdotally reported greater success in these, as well as other dimensions, since the evaluation was completed).

Figure 1 shows the scores in the four outcome areas considered for all seven teams included in the study. Teams C, D, and E scored highest for the combined totals of all four outcome areas.

Figure 1: Outcomes of CBPR

Project results, peer-reviewed journal articles, and other materials developed through these CRC projects can be see in Appendices A, B, and C.

CRC Projects and the Princip les of CBPR
This study evaluated the question, “Have the CRC projects been implemented according to the principles of community-based participatory research?” by considering three primary areas: the levels of collaboration throughout the study period, the group dynamics in conducting the study, and the levels of involvement of the broader community and the participating community-based organization. Table 4 shows the average partnership characteristics score, within each of these three primary areas, for all teams, in order from high (3) to low (0).

Table 4: Partnership Characteristics by Primary Area

Each of the elements in the table above is explored in depth below.

Levels of Collaboration
Participants were asked to discuss:

Origin of Research Question: In five of the seven teams, the research question and the activities to create the partnership clearly came from the community or community organization, who sought out the academic partner to participate in the study. The academic partner in one team had seen the request for proposals from the CBCRP and approached the community partner. Another team was formed when a community member asked a health provider about breast cancer screening information that the provider did not know. The provider called a meeting of providers, researchers, and community advocates to discuss the question and from that meeting, the research team was formed. As one participant commented:

This was truly, and I’m very proud of it, truly a community initiated project.

Developing the Research Question: Teams developed research questions in different ways ranging from the academic partner developing the research question with input from the community principal investigator, to the academic partner acting as a consultant to a team of community members who then did “a tremendous amount of research to see what was going on in terms of epidemiology studies for breast cancer.” The latter team met with experts, read articles, discussed the findings, considered the financial and research limitations, and decided what to study. Another team worked collaboratively throughout this step “at the kitchen table” in the academic partner’s home. One member of this team described the process as “creative and synergistic.” Another community partner commented,

We really focused largely on what were the important questions for the women with breast cancer that we were working with. So we worked collaboratively to look at what questions we were concerned about, and then our academic partners would really help us look at some of the literature and where the literature was going in terms of areas of interest.

Developing Research Methods: Only two teams included the community partners equally in developing the research methods for their studies: the team mentioned above, which worked on each step at a member’s kitchen table, and a team who utilized, at least in the beginning, their academic partner in a consultative role. In the other five teams, the academic partner drafted a methodology on his/her own and then solicited feedback from the community partner(s). One community partner commented:

We were all involved in [developing research methods] pretty much. There were some portions of it that were just decided without the community partners. But we were 60 percent involved.

Conducting the Project: Three teams conducted the study in a fully collaborative style with both partners sharing equally in the work effort, the ongoing discussions and modifications, and facilitating the projects’ conduct and completion. For two teams, the community organizations conducted the interventions, but did not experience a “feeling” of collaboration in the research. These groups described the process as more solo work than joint work with the researcher. One team “split the state” between the community and researcher principal investigators, with each doing their own work in different regions of California and coming together to discuss how it was going. The last team decided the community principal investigator shouldn’t be involved in the actual study because she had developed the intervention and they were concerned about the appearance of researcher bias. One academic partner commented:

I was involved to the extent of making sure everyone felt comfortable with what they were doing. I trained some of the surveyors, and then watched, observed the intervention. But mostly conducting the study itself was probably the community health educators, the community outreach workers. They were the ones who actually implemented it because of the language [of the non-English-speaking community].

Analyzing Data: For most teams, collaboratively analyzing the data was difficult. Only one team conducted the analysis of data in a truly collaborative manner—again, at a kitchen table. A statistician entered the data and returned the data runs. The team then sat around the kitchen table and talked about what they thought the statistics meant and what was happening in the community at different times—for example, community-specific publicity—that might have affected the results.

Several researchers commented on the idea that “data was data” when analyzing quantitative data. They realized that the community could give different interpretations or could provide helpful perspectives on what the “potential inferences could be” but were not convinced that community collaboration on data analysis was particularly useful or important. One community partner remarked that this differentiation of role best utilized the skills and experience of the academic partner. Another participant commented:

[It was a team effort] because the data doesn’t mean anything unless you put the community into it and the cultural variations to explain some of the things we were seeing.

Disemination: All seven teams in this evaluation reported participating in presentations of their study results to either community or scientific audiences, with four teams presenting data at both community and scientific gatherings. Two teams did not present their data back to the community.

One team discussed various ways they wanted to get the information from the study back to the community, as well as to scientific audiences. Each partner participated in all of the community forums, and each took the lead in writing articles, including peerreviewed journal articles, with input from the others. They jointly developed a PowerPoint presentation of their study and results so that any team member could give the same talk. They worked together to ensure that the community members of the research team were comfortable with the science and that the academic partner was fully educated about the community. Two partners commented:

We decided that we needed to do community forums. So I went to all the community forums and then writing up journal articles, we’ve been involved in that. But we usually would have one person take lead on the journal article, then we all add and review and critique and edit. And then just making sure people get the information and then going to the different conferences and presenting. So I’ve been a part of that. But we all kind of take turns, we make sure everyone has a turn, as well as everyone is represented.

[The academic partner] from the beginning was always saying that the community owns the data, that’s why you have to go back and do a community forum, we have to do this. And then the health educator [said], we’ll lose our reputation and trust if we don’t go back.

Balance of Power: Participants were asked to discuss the balance of power within their team. Four teams described their team as evenly balanced while the other three teams indicated that the academic partner was more in control. In two of the less balanced teams the community partners indicated that the lack of balance was acceptable to them. The lack of balance in one team was a source of contention. Team members described this power imbalance as the academic partner “carrying the ball,” functioning as “point guard,” “carrying more weight in the whole research project,” and “controlling the study.” Participants on two other teams commented:

The balance of participation was as expected, and on both sides of the equation, the community and the academic, because roles and responsibilities were very clearly defined.

Pretty much the whole gamut of activities that are involved in a research project, our community partners have always been involved as equals, but providing the community perspective.

Group Dynamics
Participants were asked to discuss their experience of conducting the project together. Specifically, they were asked about:

Communication: The level of communication was fairly high among all the groups. Some teams met in person monthly and spoke on the phone and emailed sometimes daily. Other teams communicated much less frequently, mostly by phone and/or email. One team went from weekly to monthly communication during one part of the study. While frequency and type of communication were not based on the geographic proximity of the partners, the teams that maintained frequent face-to-face communication resided and worked within the same area. One participant commented:

We were communicating on an ongoing basis because we had activities throughout the community. And that’s one thing that I guess generated such a positive relationship.

Relationships/Experience of Project: The level of satisfaction with working in the team varied, but was fairly high for all but one team. Two teams expressed effusive admiration for all members of the team. Four of the teams stated that they had very positive, “friendly and collegial” relationships, and that they felt listened to, respected, and understood. Four teams shared examples of discomfort with members of the team or with their partner, with three of those teams having conflict among members of the team. One team’s conflict resulted in continued strained relationships even after the study was complete. Two teams ended the study with the belief that they would not be willing to work together again, especially with specific members of the team. The other five teams indicated an absolute willingness to work together with one team member stating, “Yes, in a heart beat” and another saying, “Yes, with clearer parameters.” Other comments on team relationships included:

The very act of our collaboration became a living example and inspired others to get involved. And this collaboration has always been open-ended, anybody can come. In fact, people are always welcomed. And so it’s an evolving collaboration.

It was never, when we sat at the table, this is the researchers, we’re the community. It was more like everyone getting together, and hey, what are we going to do about this. And the only main difference between us was that [the academic partners] knew the statistical stuff that we didn’t know. But they never ever made us feel like we didn’t know anything. In fact, they made us feel like we were the experts and that they were learning.

I think I’ve learned how to be more collaborative, to insist that the community be involved when there’s assessments going on, when there’s distribution of funds, that the people who are affected need to be involved in the decision making. In that sense it’s probably had the greatest impact on me.

It all seems like some sort of miracle that it all gets done, given the distances, the different disciplines of people involved, the different types of institutions. It was really very interesting. And the readiness of women to contribute was always very touching to me and still is.

Turnover: Five of the projects experienced significant challenges due to turnover of project personnel or senior staff from the community principal investigator’s organization. Three of those projects had turnover in their community principal investigator between the pilot grant and the full grant, with two of those projects having turnover because the community principal investigators died from breast cancer. Two of those three projects also had a change in the executive director in addition to the community principal investigator. Turnover had both an emotional and an organizational impact on the projects. As two participants commented:

This was hard for us because we all, none of us wanted to see this happen. But [the community principal investigator] really experienced a decline in function that was hard to face and admit for all of us.

Because we’d worked together, or many of us had worked together on the pilot and we’d had a year working together putting the project, the application together, somehow I think it worked. I think we made it... I think it was okay. I don’t think it changed anything in a major way.

Conflict: Three of the seven teams reported somewhat to significant challenges on their projects. Two of those teams had significant conflicts erupt between team members that could not be resolved through mediation or intervention by the CBCRP. One team’s conflict was due to a perceived uneven balance of power between the communitybased organization and the academic partner. The other conflict was due to a lack of understanding between the academic partner and the community organization about ownership of the data and future uses of the data (including who had the power to authorize other researchers to access the data). Reflecting on conflicts between a communitybased organization and an academic partner, one partner said about community-based participatory research, “I think it’s more complicated than people think.”

Community principal investigators left their agencies at or toward the end of two of the projects. This caused conflict because the community and researcher principal investigators assumed they would take the project with them and it would no longer be associated with the community organization. In one of those situations, the community organization fought to keep the study. In the other situation, the CBCRP informed the community principal investigator and community organization that the study had to stay with the community organization. On this team, both community principal investigators had doctoral degrees, raising the possibility that each had a more academic orientation to their role in research, and assumed they could take the study with them, as is common in academia. Another participant commented about conflict:

Sometimes it becomes difficult to try and get work done. Academics are driven by certain kinds of expectations and community members, their expectations are different. So you definitely are going to have challenges. But the important thing is to be able to try and not lose sight of the bigger picture, to find answers to the challenges.

Written Agrements: Most teams did not have written partnership agreements for the two areas inquired about in the interview: ownership of data and handling disagreements. There was confusion among several of the teams about what was written down and what wasn’t. Two teams talked about “guiding principles” that included a definition of collaboration and the need to create a safe environment in which to raise concerns. On three teams, the academic partners thought they owned the data, but the community organizations thought it was shared. Both members of one team assumed the academic partner owned the data, but it was never discussed. A member of another research team commented:

We had a discussion. We now have a written statement about that and policy on it, but at the time we didn’t. Because we worked so well together and we all worked in a group, it was understood that the community owned the data.

Benefits vs Costs: Even with the difficulties stated above, all but one respondent said that the benefits of the collaboration outweighed the costs. Respondents expressed sentiments such as: “Oh, absolutely,” “Without a doubt, on every level,” “I was glad to do it, it was sort of a little bit of altruism there.” One academic partner spoke of the importance of researchers of color to participate in community-based participatory research, but said that academia, in general, does not value community involvement, so the costs to advancement are great. Other research team members commented:

The basic drawback by doing community-based projects, especially research, is you spend so much time working in the field, working on the research, that it slows you down, in terms of being able to keep pace with your grant writing commitments and publication commitments that basically can help your academic career. So if you are doing community-based participatory research, you basically have to be cognizant of the fact that it will slow your career down.

I’ve been doing this for twelve years. If I felt that the benefits did not outweigh the costs, I would have stopped doing community-based participatory research.

I think researchers learned that it takes a lot more time, because you have to do a lot of relationship building.

Involvement of the Community Organization and Broader Community
Participants were asked to discuss the level of involvement of community members from outside the community-based organization and the level of involvement of the organization’s staff and board members.

Broader Community Involvement: Community members outside the community-based organization were significantly involved in two projects as members of the research teams, as interviewers, or through an active community advisory board. Two teams had virtually no involvement of outside community members. One of these had many community members on the research team, but they were all on the communitybased organization’s board of directors or staff. The other team held focus groups to help design the survey tool. However, community members taking part in focus groups were really participants in the study, because they had no power to make decisions or influence the study in any way other than providing comments for the researcher to interpret. Two research teams utilized community leaders to help recruit participants for their study. Members of other research teams stated:

At the community level the members of our local support group were very involved in putting together the details of the project, and acting as advisors. For instance, we took the intervention instruments and gave them to a pilot group. How long did it take? Was it understandable? And so on.

I think the other impact on those women who were involved in it from the beginning was that they were empowered to use their ideas and thoughts and to talk with other women, and be more forthright about having breast cancer and be more sharing about it.

Community-Based Organi zation Involvement: One team’s communitybased organization board and staff were deeply involved throughout the study, with numerous presentations and requests for input and guidance. Four teams included some community-based organization staff and volunteers in the research team or as representatives of the research project out in the community. Two teams involved the community-based organization staff and/or board at a minimal level, such as asking them for names of people to contact to help with recruitment. Members of two teams gave contrasting comments:

I [the community principal investigator] think probably the executive director was the only other person [in the organization] who was involved. And it was mostly moral support that I got from her regarding doing the work.

We reviewed everything with all three organizations and all their staff, all the staff that were appropriate, who were either involved in breast cancer, either worked in the…community, the directors of all the agencies, and we included the other organizations, not just the funded but community organizations… getting the president and their board involved in saying is this appropriate, is it okay if we go out. So we asked permission of everyone about doing the study, going on outreach, letting them know when it was going to occur, giving them feedback, having them review all our tools, and then also in all the community forums, helping us plan the community forums.

Overall Partnership Characteristics Scores
A graph showing the combined point totals for each team can be seen in Figure 2. Three teams (C, D, and E) had partnership characteristics that more closely resembled characteristics of “authentic partnerships” as identified in the community-based participatory research literature. As one member of one of these teams said, “The collaboration drove the research.”

Figure 2: Combined Partnership Characteristics

Relationship Between Partnership Characteristics and Outcomes
When partnership characteristics were compared to outcomes for each team, the three teams ranked highest for partnership characteristics also ranked highest for outcomes (see Figure 3 below). The four teams ranked lowest on partnership characteristics also ranked lowest on outcomes. The correlation between partnership characteristic scores and outcome scores indicates that there may be some association between implementing community research collaboration projects according to generally accepted collaboration criteria and achieving positive outcomes with those projects.

Figure 3: CBPR Characteristics Compared to Outcomes

Table 5 lists the differences in partnership characteristic scores between the three teams who scored highest and the four teams who scored lowest for outcomes from their research projects. The three teams who scored highest for outcomes scored significantly higher on six key partnership characteristics than the four teams who scored lowest for outcomes. The six key partnership characteristics are:

These characteristics might be important for encouraging all team members to feel like full partners and take responsibility for project completion, as well as ensuring positive outcomes with their study results.

The four teams who scored lowest for outcomes had their lowest partnership characteristics scores in:

Regardless of their scores on outcomes, most teams scored well on several partnership characteristics. Most teams maintained good relations with their partners, participated in some level of dissemination, and all managed to overcome most of the study challenges. However, while all teams on average scored well in those areas, the high outcome teams, on average, still scored higher.

All teams had comparably low partnership scores for lack of written agreements and lack of sharing data analysis. Given the generally positive relationships the teams maintained, the lack of written agreements were only challenging for a couple of the teams. And sharing data analysis has been identified throughout the country as a challenge in community-based participatory research. Table 5 shows the average partnership characteristics score comparing the high outcome teams with the low outcome teams, and the difference in the average score, in order from the highest possible score of 3 to the lowest possible score of 0.

Table 5: Partnership Characteristics Differences Between High- and Low-Outcome Teams

One research team member noted that the process of doing the research taught the team about collaboration:

This project actually brought out a lot of other ideas around what is truly community participatory research for us, and what does it mean to have a true partnership. And I think because [the academic partner] is such, just incredible about what a true partnership is, for us it helped us so that when we do our dissemination and whenever we go out to the community, we talk about this is community participatory research, this is what you need to have in place, and this is what a true partnership is. This project is the one that generated all that. So we didn’t have anything in place during the project, but it made us think of all the things that groups would need in place. And so we’re advising others on that.