Greater Boston Evaluation Network News

RFP for a strategic planning consultant

Fri, 26 Sep 2025 13:46:23 GMT

GBEN is launching our next round of strategic planning. We are recruiting a paid consultant to help us facilitate conversations with members and document decisions. Please consider applying! Proposals are due 5pm October 9, 2025

Request for Proposals

**9/26 update: Since we're not able to hold conversations about the RFP before the close date, we are adding a Q&A period. Please send any questions to gben@greaterbostoneval.org by Thurs 10/2. We will disseminate responses to everyone who asks by Fri 10/3. Thanks!

Our Ongoing Commitment to DEI

Fri, 21 Mar 2025 16:19:26 GMT

Everything in our country and our world is rapidly changing. The commitment of the Greater Boston Evaluation Network (GBEN) to diversity, equity, and inclusion (DEI) is not.

Our organizational mission is to bring together, inform, and support our community of evaluators, broadly defined, in Greater Boston. Our membership works in all sectors and many are directly or indirectly impacted by the recent attacks on any and all policies, projects, and language designed to create a more just and equitable society. Professionally and personally, the current climate is an assault on what we value and aspire to be.

In keeping with this commitment, GBEN will:

Reaffirm previous statements supporting equity efforts made in June 2020 and May 2022.
Continue to use, and encourage others to use, the GBEN DEI Guiding Questions resource in decision making and review processes that affect our organization.
Center Diversity, Equity, and Inclusion in our next round of strategic planning, as we did in 2021 (GBEN DEI Strategic Plan).
Continue to offer our monthly Equitable Evaluation Discussion Group for members to ground their practice in DEI, resources, and community with one another.

Evolve our DEI Committee to be a monthly convening for connection, solidarity, and problem-solving among GBEN members who value DEI in their work and workplace broadly.

This continuing commitment by GBEN is driven by our belief that diversity, equity, and inclusion across our profession are foundational to fair, authentic, and actionable evaluation work. As an organization, our commitment to diversity, equity, and inclusion helps ensure that all voices and perspectives among our members are heard and respected.

Advancing fairness in our society requires all of us to make an active commitment to diversity, equity, and inclusion in our work and to stand up to attacks against it. GBEN stands ready to do its part and invites you to join us in this effort. Most of our committees meet monthly and are open for members to join. Please reach out to us at gben@greaterbostoneval.org if you'd like to chat with someone about ways to get involved.

GBEN Statement on Racist, Colonialist, and Misogynist Violence - May 2022

Tue, 31 May 2022 12:55:02 GMT

To be transparent, the Executive Committee has discussed whether to release a statement based on current events - and if so, how to acknowledge the breadth of what is happening and assure folks that we see the pain they may be going through, while not retraumatizing people in the process. We are trying our best with this message to support our full community.

If you would like to know more about why we feel moved to reach out to our members now and you feel in a safe space to read about upsetting events, please read below our signatures.

What is bringing us hope

In parallel with tragedies in our world, there is ongoing resistance and efforts to achieve the full inclusion of women, BIPOC, and LGBTQ people. Presently, we find inspiration and hope with the following people/efforts, among many more:

The leadership of students and other young people in organizing for their own safety
The Black Lives Matter movement
The ¡Milwaukee Evaluation! network, which will be holding their 5th annual conference on social justice and evaluation this fall
The May 13 Group, an invitation and a movement spearheaded by Dr. Vidhya Shanker to “repair, reverse, redress, and regenerate from the pattern of epistemic violence initiated by The May 12 Group’s racialized circulation of capital within evaluation’s political economy and convene an ecosystem structured instead around collectivism and solidarity, which nurtures epistemic healing and wholeness.”
Deb Haaland, the first Indigenous Secretary of the Interior, under whose leadership a recent report on boarding schools was released
Frank Waln, a Lakota rapper and speaker who shared his story during a plenary session of the American Evaluation Association 2021 conference

What we are doing

As a network of people working in and around evaluation, we know we must acknowledge these contexts to work towards full inclusion of our members with these identities. We ultimately strive to achieve equitable and just outcomes for the people and communities with whom we work. Therefore, it is necessary for us to mark these moments, extend our wishes for healing, and commit to continued and expanded action.

For our members and friends targeted or directly affected by recent assaults and impending political changes, GBEN commits to standing with you in healing and justice. Please reach out to greaterbostoneval@gmail.com if you would like support during this time, or if you have ideas for how GBEN can better support our community. Our committees are actively discussing ways to create more inclusive and supportive community-building throughout GBEN, and we welcome your ideas. We also welcome your feedback on this messaging.
For the allies among us, we wish you the courage and humility to show up for women, BIPOC, and LGBTQ people in the ways that will be of greatest service. Here are some resources and ways to get involved:
The GBEN Executive Committee is currently finalizing our Strategic Plan for the next three years. Our strategic goals include growing a more diverse membership, offering more programming that contributes to a more just and equitable field, increasing community-building and inclusion particularly for historically excluded folks, and embedding more DEI-focus principles and practices in GBEN structures and Committee work. Please look for more details soon.

Best,

The Greater Boston Evaluation Network Executive Committee

Min Ma
Matan BenYishay
Elizabeth Brown
Trang Hickman
Susan Putnins
Noe Medina
Bryan Hall
Trish Dao-Tran
Chantal Hoff
Dana Benjamin-Allen
Kelly Washburn

Please stop here if you do not wish to read about current day assaults and tragedies

What prompted us to send this message

The Executive Committee of the Greater Boston Evaluation Network (GBEN) condemns the recent assaults on women, BIPOC, and school-children. We initially drafted this statement prior to the murder of school-children and teachers in Uvalde, TX on May 24. While some aspects of that tragic, unacceptable assault are different from some of the earlier assaults we mention below, we also view it as a perpetuation of toxic masculinity and privileging the comfort of (white) men over the rights to existence and bodily autonomy of women, BIPOC, and LTGBQ people.

On May 14, an avowed white supremacist carried out an act of terror on the Black community of Buffalo, NY, explicitly choosing a gathering place for East Buffalo’s Black population. Earlier the same week, a shooter targeted a Korean salon in Dallas, TX, continuing an appalling trend of violence against Asian American people and women in particular. These physical assaults follow the news of the Supreme Court appearing to prepare to overturn Roe v. Wade and the national rights of bodily autonomy for women and people who can get pregnant. This assault is also overlaid on the history of systemic racism in this country, which ensures that BIPOC and low-income people will be the worst affected. In addition, last week saw the long-overdue release of a report by the Department of the Interior about the racism, colonialism, and violence carried out by over 400 forced boarding schools for Indigenous children, which led to over 500 deaths. Lastly, all of this has taken place against the backdrop of ongoing efforts to demonize and exclude LGBTQ people, especially transgender children.

These assaults are part of a past and present of white supremacist and misogynist violence and attempted control. In his draft opinion, Justice Samuel Alito argued that the right to bodily autonomy is “not deeply rooted” in our nation’s traditions. We find this sentiment abhorrent; however, these recent events have also demonstrated yet again that racism, colonialism, and patriarchy are far too deeply rooted in this nation’s traditions.

Our message above is intended to support our community while we all grapple with these unfolding events. We hope that including the details here may help some people feel seen in their pain, and help others learn more about the context their colleagues may be facing.

Check Out the New GBEN One-Page Flyer

Wed, 28 Jul 2021 15:53:49 GMT

The Membership Committee is excited to share a new flyer that details some basic information about GBEN and the perks that come with a GBEN membership. Click the image below to access a PDF version of the flyer. Please feel free share widely with your professional networks, evaluation students or classmates, and other colleagues who are interested in evaluation.

April GBEN Member Spotlight - Cassandra Tavaras

Fri, 30 Apr 2021 13:00:00 GMT

Our members are AWESOME! Member Spotlight is a quarterly blog and member newsletter feature created to make space to learn about each other and the breadth of talent existing in the Greater Boston Area.

This month's Member Spotlight is Cassandra Tavaras.

Name: Cassandra Tavaras

Social Media Handle: https://www.linkedin.com/in/cassandra-tavaras-ma, https://www.instagram.com/cvchica25/

Occupation & Field of Work: Project Manager at MGH Center for Community Health Improvement, focusing on youth development program evaluation.

How did you get started? I had always been interested in research and evaluation while in graduate school and held positions as a research assistant prior to becoming a Massachusetts Promise Fellow at CCHI. I was fortunate enough to obtain a position on the evaluation team at CCHI after my year of service and have been doing this work for over five years.

Why do you continue to work in evaluation? I enjoy the strategy and problem solving that comes with being an evaluator, and evaluation allows me to tap into my curious/analytic side. I also really enjoy having the opportunity to work with different stakeholders and community partners.

What do you like about what you do? Finding ways to help my colleagues/community partners understand and use the data/results from evaluations. It's always rewarding to be a part of conversations where it is clear that people have learned something new from the data and find ways to take action/make positive changes.

What aspect of your work are you most proud of? Leading qualitative data collection and analysis for the 2019 Community Health Needs Assessment. I really love working with qualitative data so having the chance to lead this piece of the work was a great learning opportunity.

Of all your accomplishments outside of work of which are you most proud? I teach Zumba part-time and I love it! I have had the opportunity to teach at events and have been hired to teach private classes and create choreography for a wedding! It's a lot of fun and something I am very passionate about. I love that I get to share it with so many different people.

What are you currently reading, listening to and/or binge-watching? "Just As I Am: A Memoir" by Cicely Tyson

Interested in being spotlit? E-mail Dana@backofhtenapkinconsulting.com. We look forward to celebrating your efforts.

March 2021 GBEN Roundtable Summary - Critical Race Theory

Tue, 30 Mar 2021 16:46:17 GMT

On March 18th, Kimberly A. Truong, Ph.D. (she/her/hers/they/them/theirs) from XEM Consulting Services presented an overview of Critical Race Theory (CRT) to 33 GBEN members. Kim briefly discussed the history of CRT, which began in the 1970s, and touched upon many of the people who have contributed to CRT throughout the years. Along with the history of CRT, Kim wove in what is happening currently in the country using a CRT lens. One point mentioned during the presentation was understanding the potential intersectionality of issues or topics and it isn’t “either/or”. Before breaking into small groups, two case studies were presented to begin discussions on how us as evaluators could use CRT within our work. The three small groups, led by GBEN DEI committee members Ben Faust, Patience Misner, and Susan Putnins, discussed three of the CRT tenets (Permanence of Racism, Experiential Knowledge, and Committed to Social Justice) and how it applies to evaluation and how we can use CRT to inform our own work.

A non-exhaustive list of people and readings/publications for more learning:

People

W. E. B. Du Bois
Paulo Freire
Kimberlé Crenshaw
Daniel Solórzano
Zeus Leonardo
Derrick Bell
For those in the education field: Dr. Dena Simmons and her work with LiberatED shares resources
bell hooks

Readings

GBEN seeks Committee co-Chairs!

Danelle Marable — Sat, 27 Mar 2021 16:27:09 GMT

Membership Committee:

Are you an advocate for evaluation? Do you live in the Greater Boston area? Are you a strategic thinker who wants to find more ways to get involved with the work of GBEN? Consider joining the GBEN Executive Committee! We are currently looking to fill the role of Membership Co-Chair(s). As Membership Co-Chair you will lead the charge for recruitment and retention of GBEN members and work alongside the Executive Committee and Board to advance the work of GBEN. This is an excellent leadership opportunity and avenue to continue to build and elevate the membership experience. Please email Elizabeth Brown at brown_ew@yahoo.com or Danelle Marable at dmarable79@gmail.com by Friday April 9th.

Programming Committee:
Do you have ideas on roundtables and presentations that will benefit GBEN members? Looking for ways to get more involved in GBEN? Then nominate yourself for the Programming Co-chair position! You'll work with the current Programming Co-chair, Kelly Washburn, and the 4 committee members to research and plan different events to provide professional opportunities to GBEN members. As Co-chair, you'll also be part of the Executive Committee. For more information on the responsibilities and time commitment, check out the Co-chair description located HERE. If interested, please email Elizabeth Brown at brown_ew@yahoo.com or Kelly Washburn at Kelly.Washburn02@gmail.com by Friday April 9th.

New Feature! GBEN Member Spotlight - Eric Williamson

Fri, 29 Jan 2021 16:17:43 GMT

Our members are AWESOME! Member Spotlight is a new, quarterly blog and member newsletter feature created to make space to learn about each other and the breadth of talent existing in the Greater Boston Area.

This month's Member Spotlight is Eric Williamson.

Name: Eric Williamson

Social Media Handle: https://www.linkedin.com/in/ewilliamson3/

How do you engage with GBEN? I am on the Programming and Diversity Equity and Inclusion Committees

Occupation & Field of Work: Educational Evaluation and Research Ph.D. Candidate at Boston College

How did you get started? Discovering evaluation was a happy mistake for me - I enrolled in BC's Educational Evaluation and Research program in 2016 because as a classroom teacher, I had used educational research to inform my teaching, and I wanted to learn how to contribute to this body of research. As part of the curriculum, I took two courses on evaluation, and the rest is history! I completed the M.Ed. program in 2018 and continued on in the Ph.D. track, becoming a candidate in Fall of 2020.

Why do you continue to work in evaluation? My strongest professional motivation is helping to ensure that all children in America have equal access to a safe, successful, and fulfilling life. I have found evaluation to be one of the best levers we have to work toward this goal, helping programs dedicated to equity deliver on their promises.

What do you like about what you do? As a people person, I very much enjoy the balance between working with program stakeholders, and (as someone who would like to think of themselves as a critical thinker--jury's still out on that one!) brainstorming ways to drive programs towards more equitable outcomes.

Of what aspect of your work to date are you most proud? Just this past November, the first evaluation that I assisted on was completed, and I got to experience the evaluation process from design to reporting. A small feat, but one that I am proud of!

Of all your accomplishments outside of work of which are you most proud? As a nerd of all things science-fiction, I'm proud that I had a short story published in a small science-fiction magazine back in 2015. I also had the opportunity back in 2019 to visit the recipient of bone marrow I had donated a few years prior, an act that is probably the thing I am most proud of in my life so far.

What are you currently reading, listening to and/or binge-watching? Currently listening to the audiobook of Volume One of President Obama's memoir, A Promised Land. Currently reading China Mieville's The City & The City, and Kim Stanley Robinson's The Ministry for the Future. Binge-watching the Office, as always!

Interested in being spotlit? E-mail Dana@backofhtenapkinconsulting.com. We look forward to celebrating your efforts.

Let's Hear it for the Visuals!

Mon, 19 Oct 2020 19:27:47 GMT

Hi! I’m Kelly Washburn, Project Manager of Evaluation and Strategic Support at MGH’s Center for Community Health Improvement (CCHI) and the GBEN Programming Chair. This blog post is based on my presentation I did for GBEN’s September roundtable. At CCHI, I act as an internal evaluator focusing on four community coalitions supported by MGH. When I started at CCHI over 4 years ago, I was inspired to create infographic/visual reports to highlight the coalitions’ work after reviewing the reports they are required to submit to the Attorney General and grant funders. Those required reports typically come with a template and for the coalitions, never highlight all the different areas of work they do each year.

These infographic reports are meant to complement the other reporting they have to do. As I started to move from text-heavy reports to visuals, the following tips helped me.

State the purpose: The purpose of these coalitions’ infographics was to highlight all their work during the fiscal year in an easy to understand format.
Know the intended audience: My audience was coalition staff and current and potential members. It’s important to know that this format may not be appropriate for all audiences. For example, I would not create an infographic with senior leadership at the hospital as the intent will be different for them.
Know how much knowledge and buy-in is currently there with program staff: I developed relationships with the program staff through team meetings and other evaluation activities and those conversations around reporting really showed their need in creating something more visual.
Know what data will be included and how to collect it: As an internal evaluator, I was able to devote time to attending their coalition meetings and internal team meetings, so I was able to continuously collect data on the work they were doing. I also created a basic Word document that had all their process outcomes and asked staff to fill in the gaps. That method has worked well for them, however, I’m starting to rethink the tracking process as I update their evaluation plans.
Consider the budget and time constraints: It might not be possible to get a paid subscription to programs such as Piktochart or Canva, but there is a lot that can be done with their free subscriptions or even PowerPoint. I’ve used both Piktochart and PowerPoint to create visuals. I lean towards PowerPoint when I need to include more charts and tables.

The coalitions have really appreciated the infographics and love sharing them with their audiences, along with easily pulling the data points for other reports. To see some examples of what my colleagues and I have created, check out the Charlestown Coalition’s Data Report page on their website.

For myself, I am thinking about how to change or add to this foundation including, showing trend data (when applicable) and ensuring the accessibility of the reports. For resources discussed at the presentation by myself and other GBEN members, go to the presentation for the resources slide.

Call for Nominations for GBEN Vice President - Due October 7, 2020

Wed, 09 Sep 2020 13:58:58 GMT

GBEN elections are coming up and we are reaching out with a call for nominations for the Vice-President position. The next 3-year term for Vice-President starts January 1, 2021. This is a great leadership opportunity to demonstrate your commitment to the value of GBEN and help to shape its future! You may nominate yourself or a committed GBEN colleague. The deadline for nominations is Wednesday, October 7, 2020.

Position Descriptions

GBEN is governed by an Executive Committee, which serves as the board of directors, and consists of a President, a Vice-President, a Treasurer, a Clerk, and chairs from each of the committees. The President, Vice-President, Treasurer, and Clerk are elected by the membership.Any GBEN members may serve on the GBEN Board.

The Executive Committee meets monthly tooversee all GBEN activities and operations, including overseeing all subcommittees. The Committee is also responsible for setting dues and approving a budget for each year. The Executive Committee oversees elections, fills vacancies, holds special elections, and removes Committee members as outlined in GBEN’s by-laws.

Vice-President Position Description: (10-15 hours per month)

Oversee strategic planning for GBEN;
Oversee operations of GBEN to ensure that plans are executed and tasks are accomplished;
Assist the President in conducting the business of GBEN and preside in the President’s absence including if the President’s position becomes vacant;
Preside over all Executive Committee meetings of GBEN;
Chair the Governance Committee;
Assist in planning and organizing the Annual Meeting.

Qualifications and Time Commitment

Membership with GBEN and AEA
Some leadership or management experience
Minimum of 5 years experience with evaluation-related work
Capacity to commit 10-15 hours per month
Some Board experience preferred
Strong organizational skills

Submission Process

Each nomination submission should include:

Name, Title, Affiliation, Email, Phone
Resume or CV
A brief statement answering the following questions:
- Why are you interested in becoming Vice-President of GBEN?
- What are your qualifications for Vice President?
- What is your vision for GBEN?

Submit COMPLETED nominations to GBEN via email (greaterbostoneval@gmail.com) by Wednesday, October 7, 2020.

Questions?

If you have questions about nominations process, please contact Danelle Marable, DMARABLE@mghihp.edu.

Diversity, Equity, and Inclusion Survey Results

Mon, 31 Aug 2020 13:26:14 GMT

In June 2020, the Diversity, Equity, and Inclusion (DEI) Committee conducted a brief survey with GBEN members and friends in order to understand the demographics of evaluators affiliated with GBEN as well as their needs, usage, interest, and capacity pertaining to equitable evaluations.

Methodology: A 13-item survey was administered to all current GBEN members and friends (n = 324) via Google Forms. Participants were given three weeks to complete the survey. Participants who completed the survey were entered into a raffle with a chance to win one of three $25 gift certificates, one of two GBEN tote bags, or a one-year GBEN membership.

Response: The survey was opened by 170 of the 324 people who were invited to participate. We received 51 responses, of which 45 were from GBEN due paying members. Specifically, among GBEN due paying members, 32% (45/140) completed the survey. Here is a summary of the main findings.

Demographics: Nearly 8 in 10 respondents identified as female, 14% as male, and 8% did not respond. No one identified as transgender and non-binary. The majority (81%) identified as heterosexual, 10% as gay/lesbian, 8% as bisexual, 4% as queer, 2% as fluid, and 16% did not respond. Most identified as White (76%) followed by Black/African American (12%), LatinX (2%), Asian (2%), and 8% did not respond.
Equity in GBEN: Respondents overwhelmingly responded “Never” (96%) feeling excluded at GBEN events due to your identity (ex: race/ethnicity/gender/sexual orientation). Approximately 4 in 10 respondents think that an equity-focused lens has been infused too little into the content of the roundtables, while 26% believe equity has been adequately infused.
Equity Training: In the past three years, participants attended a variety of equity trainings. Attending in-person workshops or online webinars were among the more popular answers, 73% and 35%, respectively. On average, respondents reported using three self-assessment methods when asked, How have you assessed your own cultural awareness or biases? with Reflection, Implicit Bias Test, and Workshops being the most selected.
Engaging community stakeholders and managing conflict on projects involving racism or oppression: Most respondents felt somewhat prepared (61%) to engage community stakeholders around equity. Only 8% reported feeling not at all prepared. Additionally, participants were asked, How prepared they were to respond to managing conflict when discussing racism, discrimination, and oppression?. The largest proportion of respondents indicated feeling Somewhat prepared (47%) and 10% responded Not at all prepared.
Review the PowerPoint Presentation to learn What respondents would like to see, learn, or contribute to DEI.

Next Steps: The results were shared with an equity consultant in August 2020. The consultant will conduct additional assessments that will aid GBEN in developing a DEI focused 3-year strategic plan, which will be shared with GBEN members in early 2021.

RFP for DEI Visioning and Planning Consultant

Wed, 10 Jun 2020 01:32:36 GMT

GBEN is excited to launch a visioning and strategic planning process around diversity, equity, and inclusion (DEI). We seek a consultant to facilitate the process in partnership with GBEN's DEI Committee.

For more details and application guidelines, see the Request for Proposals.

Questions should be submitted by 10 AM ET on Friday, June 19, 2020. Proposals are due by 5 PM ET on Thursday, July 2, 2020.

Greater Boston Evaluation Network statement on white supremacy, police brutality and COVID-19

Mon, 08 Jun 2020 14:46:14 GMT

The Greater Boston Evaluation Network (GBEN) unequivocally condemns police brutality and white supremacy, which has both taken the lives of Black Americans and continues to subject People of Color to police violence at this moment. We encourage jurisdictions to adopt evidence-based solutions to reduce police violence against Black Americans as quickly as possible.

In addition, we are appalled at the health inequities laid bare by COVID-19 in Massachusetts and beyond, which have similarly stolen the breath from communities of color and immigrant communities. Black Americans have been dying from COVID-19 at rates 2.4 times as high as that of White Americans. Out of the ten communities in Massachusetts with the highest COVID-19 infection rates, nine far exceed the statewide proportion of people of color.

The police violence and these health inequities are a direct manifestation of a legacy of racism that Black Americans have experienced since slavery began in 1619 in the United States. Though much has been accomplished in dismantling discrimination in our country, much more needs to be done. We commit to the fight for equity, and in particular to leveraging the truth-speaking power of evaluation.

What we are doing and how you can help

While evaluation would not have saved George Floyd’s life or that of countless others, we are committed to looking within ourselves and our organization to do better and do what we can to dismantle discrimination, racism, and white supremacy. Here is where we are starting:

Earlier this year, GBEN formed a standing Diversity, Equity, and Inclusion (DEI) Committee. This committee is working towards a plan of action to improve the state of DEI within GBEN and our members. Please consider volunteering with us (email info@greaterbostoneval.org if interested).
More than ever, as part of our planning, the DEI Committee needs to hear from GBEN members about your thoughts, knowledge, and experience around equitable evaluation. We have launched a short survey and invite all GBEN members to participate. Please fill out the survey, so that we can design the best plan for our organization and members.
In partnership with and through the generous support of the Barr Foundation, GBEN will shortly be issuing a Request for Proposals for a consultant to help us with our DEI planning. Please look out for this RFP and help circulate it to your networks.

If there are other ways for us to help, please do not hesitate to be in touch.

Resources:

List of anti-racism resources

Equitable evaluation resources:

Can Social Justice Live in a House of Structural Racism? A Question for the Field of Evaluation

Being responsive: The first assessment of Culturally Responsive Evaluation in Wisconsin: Findings from the 2017 survey

More on the GBEN site (requires member login)

Sincerely,

The GBEN Executive Committee

Danelle Marable, President

Matan BenYishay, Vice President

Eileen Marra, Treasurer

Elizabeth Brown, Clerk

Kelly Washburn, Programming Co-chair

Min Ma, Programming Co-chair and DEI Co-chair

Calpurnyia Roberts, DEI Co-chair

Bryan Hall, Communications Chair

Annie Summers, Membership Chair

February 2020 GBEN Roundtable Summary: Building Evidence

Thu, 27 Feb 2020 20:29:53 GMT

On February 12, 2020, Chuck Carter, Senior Evidence Director at Project Evident, hosted a roundtable workshop and discussion on building evidence with 22 GBEN members. The title of the workshop was entitled “The New Normal: Practitioners at the Center of Evidence Building.”

Mr. Carter started the discussion by stating that Project Evident firmly believes that practitioners should be at the center of building evidence. When led by practitioners, evidence building will result in better outcomes for communities. The next generation of evidence must be:

Be Practitioner-Led: Practitioners must be empowered to move from the caboose to the engine to drive their own continuous evidence-building agendas.
Embrace Research and Development: R&D must become standard practice in the social sector to enable timely and relevant learning and innovation.
Elevate Community Voice: The needs, beliefs, values, and voices of the community must inform and drive the work of practitioners and the field.

Mr. Carter also stated is it critical that we intentionally build evidence with a diversity, equity and inclusion lens to continually understand what works, for whom, and under what circumstances. Project Evident has developed an Evidence Matrix, a structured process designed to effectively promote and intentionally integrate a diversity, equity and inclusion framework into the evidence building activities of an organization or program. The DEI Evidence Matrix helps organizations evaluate their current evidence building practices using a DEI lens, prioritize learning questions and identify next steps that drive towards equity focused outcomes.

Mr. Carter also provided 3 examples of what it might look like for practitioners to be at the center of evidence building.

A non-profit engaging youth and young adults with lots of new data, a new program model, a developing data and evidence infrastructure, but also where staff are not focused on or prioritizing the collection or use of data.
An urban housing authority with an established data team, clear evaluation questions, and the capacity for conducting experimental studies.
A pediatric primary care program with a model that has shown positive impact, internal research and evaluation capacity, and has questions about effectiveness and scaling in new areas.

The group discussion centered on how one can create a learning culture in the organization and engage practitioners.

The workshop concluded with some questions for practitioners to consider:

Are you using program data for decisions on how to improve the model delivery and participant outcomes?
Do you have a learning agenda that evidence can inform?
Do you have a theory of change? Is it reviewed on a regular basis?
Is senior leadership bought in?
Do you understand how data is collected, stored, shared, reviewed?
Are underrepresented and vulnerable groups a part of the data collection practices?
Are you collecting data that will highlight underrepresented and vulnerable groups?
Are there regular team performance meetings, learning goals, coaching for staff, and ongoing staff recognition?

Mr. Carter’s presentation slides can be found in the member roundtable resources section.

October GBEN Roundtable Summary: What Evaluators Should Know about IRBs

Mon, 25 Nov 2019 16:22:49 GMT

On October 29, 2019, Hila Bernstein, from the Cambridge Public Health Department, facilitated a conversation about Institutional Review Boards (IRBs), how to determine whether your work needs IRB review, and best practices for evaluators when working with IRB staff to navigate the process. Nearly 20 GBEN members participated in the discussion.

During the roundtable, Hila provided a description of IRBs, how an IRB reviews evaluation studies, consideration for evaluators in working with IRBs, recommendations, then facilitated a Q&A and discussion. Below are a few key summary points from the presentation.

What is an IRB?

An IRB is a committee or group of individuals who review certain risks associated with research studies and, if warranted, provide approval for the study to go forward. IRBs review all components of a research study including protocols, methods, materials, and tools used in the study. A key IRB responsibility is ensure a study meets the following requirements:

Risks to subjects are minimized.
Risks to subjects are reasonable in relation to anticipated benefits.
Selection of subjects is equitable.
Informed consent will be sought from each prospective subject or legally authorized representative.
Informed consent will be appropriately documented.
When appropriate, the research plan makes adequate provision for monitoring the data collected to ensure safety of subjects.
When appropriate, there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data.

Identifying an IRB to Review Your Work

Depending on the funding for your study and the type of organization or company you work for, the requirements and options for identifying an IRB to review your work can be different. Research that is federally funded must be reviewed by an IRB. Internal evaluators may already have an established IRB in their organization; if not, that organization could use another institution’s IRB by obtaining a federal wide assurance (FWA) number. In certain situations, an evaluator may need IRB review at their own institution (for example, large institutions and research centers association with hospitals and universities). Independent evaluators or those without an institutional IRB may need to contract with an external IRB entity.

Is it Research?

An important question to consider before even reaching out to an IRB is considering if your work is even considered to be “research” at all. This may seem like a simple or unnecessary question to consider, but many institutions have complex, specific definitions for research, and those definitions may be different across institutions and/or disciplines. Many institutional definitions of research focus on the use or involvement of human subjects in the research. If a project is determined by an IRB to not be research, it may not require IRB review at all.

Common Modes of IRB Review

There are four common modes of IRB review:

Exempt: Exempt from IRB review. Experienced reviewer must make the determination that a project is exempt (this usually requires submitting materials for review)
Limited Review: Limited IRB review is required for research that meets specific expedited or exempt categories. This is new as of January 2019.
Expedited: Protocol is reviewed by one IRB member, done on a rolling basis. Studies must fit into one of the 7 expedited categories.
Full Board: Greater than minimal risk or doesn’t fit into an expedited category. These are the projects that go to a fully convened IRB.

Considerations for Evaluators

Hila offered a few helpful considerations for evaluators.

Evaluation that’s part of a research study is reviewed together with that research study (cannot separate components of a research study) while evaluation of practice would be reviewed even if practice is not subject to review.
If a project is not research or not human subjects research, it is still good practice to carefully consider consent content and process for collecting consent.
Evaluators should look through the required elements of consent and think about what information you would like to have available if you were a participant. Recruitment processes, data protection, privacy and confidentiality are also helpful to think about.
Take advantage of all of the resources that your IRB offers (may include guidance documents, meetings, educational sessions) and don’t be afraid to ask questions or request examples.
Start early! Obtaining a determination or going through the review process will take time, there may be questions (about the proposed activities or evaluation more generally), and the details of the project may evolve.

Hila’s presentation slides can be found here (members only).

September GBEN Roundtable Summary: Hiring for an Evaluation Position

Thu, 26 Sep 2019 16:51:33 GMT

On September 10, 2019, Bryan Hall, Senior Director, Evaluation at BellXcel, discussed his experience hiring evaluation professionals at his organization, building a three-person evaluation department, and the process and challenges he encountered along the way. Over 20 GBEN members participated in-person or virtually.

Mr. Hall discussed the pre-hiring process and what hiring managers should consider before making a job posting public, the hiring process and what to consider when creating a job posting, the applicant and resume review process, and the key characteristics of strong evaluation professionals. Below are a few key summary points from Bryan’s presentation.

The Hiring Process Begins Long Before a Job Posting is Public

The pre-hiring process is critical, and can make or break a hiring initiative. Before developing and making public a job posting, it’s important to consider a host of factors.

Hiring managers should identify the key processes and stakeholders that will be part of the entire hiring initiative. For example, what role will your organization’s Human Resources department plan in the hiring process? Who will participate in the interview process and what scheduling accommodations will be needed?
It’s important to know your full budget for hiring, beyond (but including) the salary range for the position. Do you have budget to fly a candidate in for an interview? Will you have budget to train staff once hired?
Don’t underestimate the amount of time needed to complete the hiring process. Some hiring processes can take upwards of a year to complete. The interview process alone can sometimes take two-to-three months. Bryan noted that a recent hiring process for an Evaluation Manager position took 6-8 months before leading to a hire.
Think seriously about the workload of the position you are hiring for. What will their day-to-day, month-to-month work life look like? It’s important to consider whether you even need a full-time employee at all or if part-time, seasonal, temporary, or consultant staff would be a better fit for your needs.

The job posting is important for candidates and the hiring manager

Once you are ready to formally start the hiring process, it’s important to develop a job posting that you will make public to interested candidates. Similar to the pre-hiring process, it’s important that the job posting strongly reflect your organization’s needs. A poorly designed job posting can delay your hiring process or attract candidates that may not be the best fit for your needs. A few considerations for the job posting:

The job posting is not the same as the job description you hand a new employee once they start. Avoid making a job posting an exhaustive list of responsibilities, but instead try to capture the high level job responsibilities and requirements you are looking for.
A job posting should include key information that allows an interested candidate to decide if they are a good fit for the job. Key items to include are: brief position description, key responsibilities and expectations, key hiring attributes and requirements, a brief description of your organization and work, brief summary of benefits, the process to apply, and salary range (ideally, but not always possible).
Seriously consider what attributes a strong candidate must have in order to be considered for the position, and what are simply nice to have. Most employees learn on the job and receive significant training once they start. Consider which attributes are flexible and which are non-negotiable for a candidate.
Consider the power of transferable and general skill domains, versus hiring for a specific skill set. For example, if your organization uses Salesforce as a data system, you don’t necessarily need to hire a candidate with Salesforce experience. Instead, consider – and advertise for in the job posting - a candidate with strong “technology proficiency” in other systems who can be trained on how to use Salesforce.
Don’t let perfect be the enemy of great. There is no such thing as a perfect candidate. Of the “must haves” and “nice to have” attributes in your job posting, consider 3-5 that are most important to you, and which others you can be flexible on.

There are many online resources for job postings

Your job posting should ideally be hosted on your company’s website and/or LinkedIn account. In addition, the evaluation world has a few key job websites for job postings including the American Evaluation Association (AEA) website and evaluationjobs.org. In addition, consider general job websites like LinkedIn, indeed.com, and idealist.org (for non-profits). Job sites like Monster and Career Builder may not be that useful for evaluation positions. Lastly, consider discipline-specific professional associations, as most offer the ability to post job openings. For example, if you work in the field of public health, consider listing the job through the American Public Health Association (APHA) website.

Interview Questions Should Fill in the Gaps of the Resume

A resume and cover letter (if provided) should tell you 80-90% about a candidate. The purpose of the interview is to fill in the rest. Therefore, focus your interview on attributes about the candidate that may not be expressed via the resume such as passion for the work, soft skills, and problem-solving skills. Example questions that Bryan has used in past interviews include:

Why are you interested in this position? Why did you apply?
Tell me about a recent job experience and relevance to this job?
A key job responsibility is ____. Tell me about a time you did ____ ? Are you comfortable/do you enjoy doing ______ ?
Tell me about your work personality? How do you work with others and/or independently? What are your needs as an employee?
Tell me about a time you faced a conflict/challenge/problem – how did you approach and resolve it?

Mr. Hall’s full presentation slides can be found here (members only).

Call for Nominations for GBEN Treasurer and Clerk - Due October 7

Thu, 26 Sep 2019 16:41:05 GMT

The next 2-year terms for GBEN Treasurer and Clerk start January 1, 2020. This is an opportunity to show your commitment to the value of GBEN and help to shape its future! You may nominate yourself or a committed GBEN colleague. The deadline for nominations is Monday, October 7, 2019.

Position Descriptions

The Executive Committee meets monthly to over see all GBEN activities and operations, including overseeing all subcommittees. The Committee is also responsible for setting dues and approving a budget for each year. The Executive Committee oversees elections, fills vacancies, holds special elections, and removes Committee members as outlined in GBEN’s by-laws.

Clerk Position Description:

Record the proceedings of GBEN;
Keep the records of Bylaws and subsequent amendments;
Handle all the general correspondence of GBEN, as directed by the President and Vice-President;
Support creation of agendas for GBEN meetings;
Work with the Treasurer to submit annual IRS filing for 501c(3) status and attend to any other administrative and annual reporting work associated with 501c(3) status.

Treasurer Position Description:

Collect dues and any other funds to be received by GBEN;
Document all financial transactions related to GBEN;
Report monthly financial updates to the President and Vice-President and the Executive Committee;
Report at general membership meetings and prepare an annual/fiscal year report;
Transact the general business of GBEN in the interim between meetings;
Disburse funds and pay bills in accordance with the provision of the Bylaws or policies of the Executive Committee;
Work with the Clerk to submit annual IRS filing for 501c(3) status and attend to any other administrative and annual reporting work associated with 501c(3) status.
The outgoing officers shall deliver to their successors all books and materials of their respective offices by January 15th.

Qualifications and Time Commitment:

Membership with GBEN and AEA
Some leadership or management experience
Minimum of 3 years experience with evaluation-related work
Capacity to commit 10-15 hours per month
Some Board experience helpful
Strong organizational skills helpful

Submission Process:

Each nomination submission should include:

Name, Title, Affiliation, Email, Phone
Resume or CV
A brief statement answering the following questions:
- Why are you interested in becoming Clerk or Treasurer of GBEN?
- What are your qualifications for Clerk or Treasurer?
- What is your vision for GBEN?

Submit COMPLETED applications to GBEN via email (greaterbostoneval@gmail.com) by Monday October 7, 2019 or earlier, if possible.

Questions?

If you have questions about nominations process, please contact Danelle Marable, DMARABLE@mghihp.edu.

Question of Practice: What is the next big skill you want to learn?

Wed, 31 Jul 2019 14:14:07 GMT

Big data. Data science. Predictive analytics. Social network analysis. The field of evaluation is expanding to new frontiers, becoming a transdisciplinary practice.

Based on your work experience and interests in the field of evaluation, what is the next area(s) that you want to learn more about and integrate into your evaluation practice?

June GBEN Roundtable Summary: Social Network Analysis

Wed, 26 Jun 2019 19:24:03 GMT

On Tuesday, June 18^th, GBEN hosted its second roundtable on the topic of social network analysis. Over a dozen GBEN members and guests participated. The roundtable discussion was led by Kelly Washburn, MPH, from Massachusetts General Hospital’s Center for Community Health Improvement. Kelly is also one of GBEN’s Programming Committee co-chairs.

Social network analysis (SNA) is the mapping and measuring of relationships and flows between people, groups, organizations, computers or other information/knowledge processing entities.” (Valdis Krebs, 2002). SNA can show the performance of the network as a whole and its ability to achieve its key goals, characteristics of the network that are not immediately obvious, such as the existence of a smaller sub-network operating within the network, the relationships between prominent people of interest whose position may provide the greatest influence over the rest of the network, and how directly and quickly information flows between people in different parts of the network.

Kelly walked through a small social network analysis she conducted to walk participants through the different steps needed, challenges, and lessons learned. The project discussed was a provider task force improving connections among services providers, streamlining services, and enhancing care coordination efforts. The SNA provided a baseline on how the task force members work with each other by asking four questions:

Do you know this person?
Have you gone to this person for information in the last year?
Have you worked with this person to obtain new resources in the last year?
Have you referred a client to their organization in the last year?

The analysis was done in Gephi, a free software for conducting social network analyses. Data cleaning was the most tedious part of the project and was done manually; however, there are ways to bypass the manual data cleaning. After the data is set-up in the appropriate Node and Edges file, they are uploaded into Gephi. Once in Gephi, the steps detailed in their manuals were followed to take it from the initial map to the finalized map. Following Kelly’s discussion of her project, others in attendance spoke of their own experiences of using social network analysis in their work.

Key Challenges and Lessons Learned:

The roundtable participants discussed a few challenges and lessons learned when conducting a SNA, including:

New analytical methods and techniques, like SNA, can require a lot of patience and time to learn and master. Be sure to invest the necessary time when learning how to conduct a SNA for the first time.
A high response rate means A LOT of follow-up to ensure the data is representative of the population you are analyzing. Be sure to invest the necessary time and resources to doing follow-up for your project.
Make sure the questions being asked are the right questions as it’s difficult to change directions once the project and analysis has started.
Continually ask yourself and/or your team(s): Do I need to collect new data or is there already collected data I can use for the SNA?
SNA can be frustrating to administer and master at times. Patience during the process is key to ensuring a successful outcome.
The visual map was key for the task force in understanding the analysis.

Resources:

Kelly’s presentation can be found in the Roundtable Resources page (members only)
Members can also access in the Roundtable Resources section the handout Laura Beals had put together for her SNA presentation back on December 5^th, 2017.
Gephi - SNA software
UCINET Software
The Fletcher School at Tufts University has a Social Network Analysis class

Organizational Transformation: Is Feminist Evaluation the Answer?

Tue, 28 May 2019 17:12:56 GMT

Feminism, at its core, is about the transformation of power—but how do you know that’s happening at the organizational level? How can you understand the core drivers of that transformation? How can your own process of evaluating that transformation democratize the evaluators’ power?

Taylor Witkowski and Amy Gray are evaluation and learning specialists at Oxfam America, designing and testing feminist monitoring, evaluation and learning processes for a gender justice-focused organizational change initiative.

WHAT IS FEMINIST EVALUATION?

Everything is political – even evaluations.

Traditional evaluations, even when using participatory methods, prioritize certain voices and experiences based on gender, race, class, etc., which distorts perceptions of realities. Evaluators themselves carry significant power and privilege, including through their role in design and implementation, often deciding which questions to ask, which methodologies to use, and who to consult.

Feminist evaluation recognizes knowledge is dependent upon cultural and social dynamics, and that some knowledge is privileged over others – reflecting the systemic and structural nature of inequality. However, there are multiple ways of knowing that must be recognized, made visible and given voice.

In feminist evaluation, knowledge has power and should therefore be for those who create, hold and share it – therefore, the evaluator should ensure that evaluation processes and findings attempt to bring about change, and that power (knowledge) is held by the people, project or program being evaluated.

In other words, evaluation is a political activity and the evaluator is an activist.

APPLYING FEMINIST EVALUATION TO ORGANIZATIONAL TRANSFORMATION AT OXFAM AMERICA

Oxfam America is seeking to understand what it means to be a gender just organization—from the inside out.

In order to do this, Oxfam America recognizes that a holistic, multi-level approach is required. We believe that transformational change begins at the individual level and ripples outwardly into the organizational culture and external-facing work (Figure 1).

(Figure 1)

This is why we are investing in a feminist approach to monitoring and evaluation—because even though feminist values—adaptivity, intersectionality, power-sharing, reflexivity, transparency—seem like good practice, without mainstreaming and operationalization they would not be fully understood or tied to accountability mechanisms at the individual, organizational or external levels.

Therefore, as evaluators, we are holding ourselves accountable to critically exploring and implementing these values in our piece of this process. The foundational elements of this emergent approach include:

Power-Sharing: The design, implementation and refinement of the monitoring and evaluation framework and tools are democratized through participatory consultations with a range of stakeholders—a steering committee, the senior leadership team, board members, gender team and evaluation team.

Co-Creation: Monitoring includes self-reported data from project contributors as well as process documentation from both consultants and evaluation staff, and data is continually fed into peer validation loops for ongoing reflection and refinement.

Transparency: Information regarding the monitoring and evaluation framework, approach and activities are communicated and made accessible to staff on a rolling basis as they evolve.

Peer Accountability: Monitoring mechanisms that capture failures and the cultivation of peer-to-peer safe spaces to discuss them create new opportunities for horizontal learning and growth. This includes a social network analysis (SNA) of perceived power dynamics within teams (contributed by team members via an anonymous survey), followed by a group discussion in which they reflected on the visual depiction of their working dynamics through the lens of hierarchy and intersectionality.

EVALUATORS AS ACTIVISTS

As the monitoring, evaluation and learning (MEL) staff working on this initiative, we recognize that we have an opportunity to directly contribute to change. We therefore see ourselves as activists, ensuring MEL processes and tools share knowledge and power as well as generate evidence that reflects diverse realities and perspectives, and can be used for accountability and learning at multiple levels. As a result of this feminist approach to MEL, participating Oxfam staff can see and influence organizational transformation.

How have you used feminist evaluation in your work? Do you have any tips, resources or lessons learned you’d like to share? Do you think this would make a good roundtable discussion?

CONTACT

Amy Gray (amy.gray@oxfam.org)
Taylor Witkowski (taylor.witkowski@oxfam.org)

RESOURCES

Applying Feminist Principles to Program Monitoring, Evaluation, Accountability and Learning (2017) https://policy-practice.oxfam.org.uk/publications/applying-feminist-principles-to-program-monitoring-evaluation-accountability-an-620318
Oxfam’s Guide to Feminist Influencing (2019) https://www-cdn.oxfam.org/s3fs-public/file_attachments/oxfams_guide_to_feminist_influencing_english.pdf

Question of Practice: Do You Have More Data Than You Need?

Mon, 29 Apr 2019 19:12:35 GMT

As evaluators, we sometimes collect more data than we can use.

What are 1 or 2 methods or tricks you use to make your data collection process more meaningful and/or more aligned to your evaluation questions?

February GBEN Roundtable Summary: Impact Evaluation

Wed, 27 Mar 2019 14:52:59 GMT

On Tuesday, February 5^th, GBEN and Northeastern University’s Public Evaluation Lab (NU-PEL) co-hosted a panel on Impact Evaluation. This was the first GBEN event of 2019 and the first event co-sponsored with NU-PEL. The event saw the greatest turnout in the history of GBEN with 66 attendees!

The panel featured five local internal and external evaluation leaders who have recently undergone randomized-control trial (RCT) or quasi-experimental impact evaluations.

The five panelists were:

Lisa Goldblatt Grace, Co-Founder and Executive Director or My Life My Choice (MLMC), and Amy Farrell, Professor in the School of Criminology and Criminal Justice at Northeastern. Both panelists discussed a collaborative, multi-year, multi-site evaluation of the MLMC interventions.
Jack McDevitt, Director of Northeastern’s Institute on Race and Justice. Mr. McDevitt discussed a randomized control-trial study of the 2016 pilot implementation of body-worn cameras worn by Boston Police Department officers.
Ira Nichols-Barrier, Senior Researcher at Mathematica Policy Research. Mr. Nichols-Barrier discussed a large scale evaluation of KIPP (Knowledge is Power Program) schools.
Jessica Britt, Research and Evaluation Manager at Year Up. Ms. Britt discussed a series of RCT studies conducted by Year Up.

More and more, non-profits must demonstrate impact in order to ensure their ability to grow and innovate. The purpose of the panel discussion was to explore what drives non-profits to engage in an impact evaluation, how to choose methodology, and lessons learned about communicating results.

Here are some of the key takeaways from the engaging panel discussion:

Methodological Rigor vs Reality

Several of the panelists discussed the push-and-pull between ideal methodological rigor and what is actually possible and/or ethical for programs. In particular, Ms. Britt and Mr. Nichols-Barrier spoke to being able to do or not do randomization based on program over-subscription. On the flip side, Ms. Goldblatt Grace and Professor Farrell from My Life My Choice shared a powerful anecdote about overcoming skepticism to their project’s rigorous methodology to allow a research assistant to be present at the mentor-mentee match sessions.

Organizations Conduct Impact Evaluations for Lots of Reasons

The motivating factors behind the decision to evaluate impact are diverse. Organizational values, political context, and funders can all play a role in the decision to conduct an evaluation as well as decisions around study methodologies.

Communicating Results

Several of the panelists shared helpful tips about communicating results, specifically going beyond sticking them in a report that few people read. Ms. Britt shared a strong example of Year Up planning a year-long plan for communicating parts of their results throughout the whole organization, including a big celebratory kick-off event.

GBEN would like to thank the five panelists for being a part of this incredible event as well as NU-PEL for co-hosting. Be on the look-out for future co-hosted events with NU-PEL!

December GBEN Roundtable Summary: Budgeting for Evaluation

Wed, 19 Dec 2018 15:48:49 GMT

On Tuesday, December 4^th, GBEN hosted its last roundtable of the year about budgeting for evaluation. Like all of our roundtables this year, we had great a turnout with 17 members participating. The roundtable was designed to be an open dialogue focusing on the critical issues evaluators face when budgeting for evaluations. Guiding questions for the discussion included:

How do non-profits and other organizations fund evaluation staff, data management systems, and other elements of successful evaluations?
Is there a magic "rule of thumb" on how to allocate budget resources to evaluations?
Are there funders interested in supporting evaluation capacity building?
What are other processes or best practices to consider when budgeting for evaluation?

Here are some of the key takeaways from the group discussion:

Evaluation staffing and capacity varies across organizations

Some organizations still have little-to-no dedicated evaluation staff on the payroll. For many organizations, having a dedication evaluator is a new organizational initiative. For those with dedicated evaluation staff, many are grant-funded, meaning once the grant is over, the position may be reduced or eliminated.

Evaluation budgets can span multiple departments

Many organizations spread evaluation costs and budgets across varying departments and/or programs. Evaluation budgets can include staff who may not have the word “evaluation” in their job title or job responsibilities, including front-line program staff, data management and/or administrative staff, and/or information technology and systems staff. For example, an organization may employ a Database Specialist through their IS/IT department who does critical evaluation-related work by using Efforts to Outcomes (ETO) software.

Commitment from organizational leadership is key

Like all facets of a good evaluation initiative, commitment from senior management within an organization is important for evaluation funding. Senior leadership and management are often best positioned to seek out, advocate for, and request funding for evaluation from key funding partners.

Monitoring your evaluation work helps make the case for future or additional evaluation

Evaluation staff may hold the key to the magical data kingdom, but often time we don’t directly experience or see how the evaluation results are used to change or improve programmatic processes. Be sure to document the collaborative process between the evaluation team and front-line program staff to highlight programmatic improvements that are a result of the evaluation findings.

Relationships with development staff are key

It is very helpful to have a relationship between the development team and evaluation team, especially for grant writing. Development staff understand funder needs and wants and can effectively communicate impressive evaluation results.

Assert your evaluation needs!

Most people have no idea what it takes to make an evaluation successful. Often time it takes more than just a staff person. Evaluation needs can include certain software, equipment, consultants with specific expertise, and other administrative needs like postage and mailing supplies. Make a wish list of things you need and share it!

Seek out ways to reduce evaluation costs

While planning and conducting and evaluation, it’s important to always ask the question: “what do we want and what do we already have?” For example, is there data already being collected or existing data systems that could serve your evaluation project? Finding ways to lower evaluation costs may make it easier to acquire the necessary budget resources for an evaluation initiative.

Resources: (click here to access members-only resources from this roundtable)

Budgeting for Evaluation: Key Factors to Consider – a rubric developed by Westat for assessing how much to budget for evaluation
Budgeting for Evaluation from the 2014 Americorps Symposium

October GBEN Roundtable Summary: Community Health Needs Assessments

Wed, 31 Oct 2018 17:52:25 GMT

On October 5, 2018, Danelle Marable, GBEN President and Senior Director for Evaluation, Assessments, and Coalitions at the Massachusetts General Hospital Center for Community Health Improvement (MGH/ICHI), discussed two community health needs assessments that her team is working on:

Partnering with the North Suffolk Public Health Collaborative, municipalities, other healthcare providers, community coalitions, and organizations to conduct an assessment for Chelsea, Revere, and Winthrop.
Partnering with all Boston hospitals to conduct a Boston-wide assessment.

In 2011, the Affordable Care Act required every hospital to conduct a community health needs assessment (CHNA), develop strategic plans, and post findings to the public. Non-profit status would be revoked if the CHNA was not conducted.

A CHNA is a systematic examination of the health status indicators for a given population that is used to identify key problems and assets in a community. The goal of a CHNA is to develop strategies to address the community’s health needs and identified issues. A CHNA is instrumental in identifying the social and environmental conditions as well as social determinants that can impact the health of these communities, such as childhood experiences, housing, income, employment, healthcare, community. MGH identified Revere, Chelsea, Charlestown and Winthrop as primary communities to target with CHNA and a community health improvement plan (CHIP).

During the roundtable, Danelle discussed the process around the assessments, community engagement strategies, data collection efforts, and implementation.

Process

The CHNA and CHIP is a year-long process that occurs every three years. MGH uses the MAPP Framework (Mobilizing for Action Planning and Partnerships) for the CHNA and CHIP that in short outlines a process of engaging partners in comprehensive data collection and strategic. MGH/ICHI dedicates one year for visioning, assessment, and identification of strategic issues and then their Board of Trustees reviews and grants approval, then allowing for 100 days to develop an implementation plan. The MGH Trustees are required in MA to go to community advisory committee meetings in addition to other meetings.

Community Engagement and Needs Assessment

MGH/ICHI engages multiple sectors during the CHNA process: resident, local leaders, community-based organizations, educators, as well as other hospitals in the communities. MGH/ICHI started engaging other hospitals so they don’t burden the community with repeated data collection. Assessment and identification of needs can be a challenge as prioritization is not always straightforward. For example, residents in the community of Chelsea spoke of issues of community violence and safety, but other data sources showed decreases in instances of violence in the community.

Data Collection

Data collection for the CHNA involved survey data, focus groups, in-depth interviews, and secondary data sources, including the quality of life survey (addresses more social determinants such as housing, transportation, overall community). Surveys are translated and are distributed online and in print through various community networks.

Implementation and Evaluation

A crucial part of the process of a CHNA is the identification and implementation of evidence-based strategies around certain community needs and issues. Part of this process involves community dialogue – asking key questions about what can be done, what resources are available, and is there the will to implement solutions. This process not always leads to straightforward answers. For example, communities identified opioid use as a crisis and access to Narcan as a response, however the community balked at increasing access to Narcan under the suspicion that it leads to increased opioid use. The group had to take 3 steps back to educate the community regarding the benefits of Narcan. Once strategies have been implemented, the final process is comprehensive progress monitoring, implementation evaluation, and impact measurement.

Copies of Danelle’s PowerPoint slides can be found in the member roundtable resources section (members only!). For questions, reach out to Danelle by email at: dmarable@partners.org.

September GBEN Webinar Summary – Webinars that Don’t Suck!

Fri, 28 Sep 2018 13:19:32 GMT

On September 6, 2018, Dana Benjamin-Allen of the Boys and Girls Club of America - as well as a long time GBEN member - hosted GBEN's first professional development webinar titled “How to Host Webinars that Don’t Suck!” The webinar – a live meeting and discussion that occurs via the internet – can sometimes have a reputation for being boring, disengaging, and a waste of an audience member’s time. For evaluation professionals, creating a data-rich yet engaging webinar can be a challenge.

Fourteen GBEN members joined the webinar to learn about tools to increase interactivity and engagement, best practices for online presentations, and the benefits of different distance engagement platforms. Dana used several tools and activities to model ways to engage the audience and keep them interested in the content of the webinar. You can access her slide deck on the Roundtable Resources section of the website (members only). Below are some take away points for engaging your webinar audience.

Framing the Webinar is Important!

How you frame and present the webinar is critical to gaining interest and maintaining audience engagement once the webinar starts. Start the webinar with an introductory slide that presents the intentions and objectives that best resonate with your audience. It’s important to describe the presenter's relevant background and experience.

In addition, it is important to ‘level set’ your audience, that is, identifying the knowledge and experience level among your audience members regarding the webinar topic. You may have audience members with diverse backgrounds, varying knowledge levels, and varying experience levels with the topic. Level setting can help you hone in on certain content areas that may be most relevant to this diverse audience as well as identify gaps in knowledge. Hot tip: create interactive activities using online polls and tools like GroupMap to level set your audience.

Language matters so be sure to use a catchy title to make the webinar more inviting. The webinar title “Webinars that Don’t Suck!” is a perfect example! The timing of your webinar is very important. For some audience members, mornings are better, for others it may be lunch time. If you are reaching a diverse geography of audience members, be sure to schedule it conveniently for all time zones. Lastly, experiment with different alternatives to the word webinar to increase engagement. Hot tip: call it an online meeting, online course, virtual workshop, “brown bag”, or like the American Evaluation Association, a virtual “coffee break.”

Choose the best facilitation tools and methods based on audience needs and intended engagement level:

There are various webinar platforms on the market today with varying functions. Do you need a live whiteboard? Do you want to share and/or pass the screen to multiple presenters? Do you want a live camera to accommodate live video of all participants and presenters? These factors should be considered before choosing a platform that best fits the format of your presentation, the needs of your audience, and the intended engagement level. Hot tip: using shared experience activities, such as real-time polling applications, helps connect audiences as well as solicit feedback.

In addition to picking a webinar platform, it is important to identify which facilitation method is right for you and your audience. Is there one or multiple presenters? Are you hosting a moderated panel discussion? Is the presentation interactive and involve audience participation? Hot tip: co-hosting the webinar with a colleague who has credibility with your audience can increase audience interest and engagement.

Lastly, plan accordingly if you plan to take questions and how you will manage the questions. Do you need a colleague to help with this? Will it be interactive (i.e. audio for all participants)? Between 6-10 questions may be good for a live interactive presentation, but plan extra time as participants may interact during each question.

Presentation is Important:

The design of your presentation slides is very important. Keep your slides clean and easy to read. If you are going to use a lot of text in a slide, plan accordingly to walk through the text with the audience. Also, create intrigue, confusion, and or excitement from slide to slide using questions, images, and or illustrations. Lastly, use a strong, exciting, and engaging title. Hot tip: there are online alternatives to PowerPoint like Prezi, Visme, or Haiku Deck.

During this webinar, Dana used the following tools (follow the links for more details):

GotoWebinar for the question and chat format and polling feature
Groupmap
Poll Everywhere for voting and ranking (free for under 100 respondents)

August GBEN Roundtable Summary - Conducting Qualitative Data Analysis

Fri, 24 Aug 2018 15:00:00 GMT

On Tuesday, August 7^th, Carrie Fisher, PhD, Research and Evaluation Manager at the Institute for Community Health, presented to over 20 people on qualitative data analysis at the Jewish Family and Children’s Services in Waltham. Ms. Fisher is an anthropologist with interests in applied and evaluation research, innovative research methods, health and public health, and research with difficult-to-reach populations.

Ms. Fisher provided a comprehensive overview of qualitative data analysis that began with a very interesting discussion about how concepts and meanings of “truth” and “reality” can vary in qualitative data analysis. The presentation then focused on qualitative data analysis choices and methods, data management, data analysis, and key issues around data interpretation. Below are a few key summary points from the discussion.

Reality and Truth in Qualitative Data Analysis

There is not just one truth when dealing with qualitative statements and there are things that can affect how truth and reality are defined such as interviewer, analytical, and interpretive bias. Even false statements can provide important lessons in the analysis. It’s important to remember that qualitative data analysis is always selective and can be impacted by individual backgrounds, education, and even the mission and vision of the organization sponsoring the qualitative data project.

Qualitative Data Analysis Choices and Methods

Starting a qualitative data analysis project can be challenging as there are many questions that need to be considered such as: quantity and type of data, validity of analysis, how will the findings be used, identifying the audience, staff capacity to conduct the analysis, time and money for the analysis, and whether or not to use technical qualitative data analysis software.

Ultimately, a qualitative data analysis project starts when you decide on a data collection method. Qualitative analysis is an ongoing iterative process and researchers and evaluators should meet weekly with those doing the data collection to review responses to questions to determine if a change to question design is necessary.

Qualitative Data Management

It is critical to plan ahead for data management of qualitative data and to consider small, yet important, factors such as how you are going to record the data (written, audio recordings, video recordings), how you are going to conduct data entry (Excel, Word, other database), and which tools and/or software you may use to manage and analyze the data.

For example, one key data management consideration for in-depth interviews and focus groups is whether or not to use transcription services. Verbatim transcription can be expensive – just one hour of audio or video can require up to four hours of transcription, which can cost hundreds of dollars. The website www.rev.com is a cost-effective alternative for transcription with rates around $1 per minute.

If you are using qualitative data analysis software like Dedoose, NVivo, ATLAS.ti, or others, it’s important to remember that these software only manage data and do not automatically perform analysis for you. Coding and analysis of qualitative data using these software are additional steps which require sufficient time and technical training in the software itself.

Qualitative Data Analysis

It’s critical to choose your analytical approach with the end in mind. Key issues to consider include: level of detail, level of rigor, audience, how the information may or will be used, time and resources for analysis. There are many different approaches to analyzing qualitative data including pragmatic thematic analysis, case studies, content analysis, and coding trees to name a few.

Once your data has been collected and managed, schedule extra time to “swim in the data.” Get to know your data well by reading over all notes and transcripts before doing any coding. Write down preliminary thoughts on main themes, points of interests, and gaps in the data.

Lastly, coding of qualitative data is a critical step. It’s important to organize codes into meaningful categories and to create a descriptive codebook to document definitions and themes in codes. This is particularly important if your project has more than one analyst conducting coding in order to maintain consistency.

Qualitative Data Interpretation

When you are ready to begin interpreting your data findings, begin by listing key points and themes such as: key confirmations, major lessons, new ideas, and applications to other settings and/or programs. Some evaluators choose to summarize qualitative findings using quantitative outcomes (i.e., “9/10 respondents agreed with this idea…”). While this can increase confidence in results, it should be used with caution.

Ultimately, the evaluator should always ask: “Would I feel comfortable with the participant reading this.” “Would the participants agree with my interpretation of the findings?”

Ms. Fisher’s full presentation slides can be found here (members only).

2 Things That Need To Be In Place Before You Can Build An Evaluation System

Mon, 30 Jul 2018 16:37:13 GMT

A full evaluation program requires data systems and analytic capacity. Many organizations or programs feel like they’re not prepared to start building their internal evaluation capacity until they have developed logic models and databases. But there are only two elements that need to be in place in order to start evaluating your program. They are clarity about your organization’s mission and clarity about your organization’s target population. These elements are required to focus all of your data collection and analysis.

Mission clarity is a single, shared understanding of why your program is in business. While most organizations know very well what they do, there is sometimes a lack of clarity about why they do it. For example, if you run an after-school program for teens, it could be intended to improve the youths’ academic skills (an educational goal), help them explore careers that might interest them (a workforce development goal), or pair them with a caring adult (a positive youth development goal). In order to begin building an evaluation program, the program’s mission needs to be clear. Without clarity about your program’s real mission, it will be impossible to prioritize performance measures or design performance measures that communicate your program’s value to all of your stakeholders.

One way to document your organization’s mission is a simple strategy map, documenting the activity, the immediate outcome for your participants and the long-term outcome that you envision for them. It can be as simple as this:

tutoring à passing grades à high school graduation à careers à economic well-being

Developing this strategy map can be challenging, particularly when there are many possible positive outcomes based on the activity that your organization provides. It’s a necessity because your data collection systems will be organized around outcomes that serve that one central purpose.

The second thing you need is consensus about who exactly your organization serves. Who is best positioned to benefit from your program? Who is the most important population for your program to be serving? For example, in the above example, if your program serves an academic goal, the most important population to serve would be youths who are at risk of leaving high school without a diploma. If the goal is a workforce development one, the target population might be youths who are early enough in their high school careers to choose electives based on their future career plans. If your program’s goal is mentorship, your target population might be young people who are developmentally most able to benefit from a mentoring relationship. Measuring how closely the population you serve mirrors your ideal population requires perfect clarity in that area. Clarity about who you serve will organize all of your data collection and analysis and define your unit of analysis.

With these two things in place, your organization will be ready to develop measures that really indicate whether your organization is helping the population that you intend to serve and whether it is helping to move your participants or clients toward their goals. Equally important, this clarity will help you avoid collecting unnecessary data or focusing on metrics that don’t really drive your organization’s performance.

By Pieta Blakely, PhD

Pieta Blakely is a researcher and evaluator, specializing in quantitative evaluations of workforce development programs and local economic development initiatives. She has led multiple evaluation projects spanning education, labor economics, and urban economic development topics.

As the Principal of Blakely Consulting, LLC, she focuses on working with not-for-profit organizations to build evaluation capacity, integrate evaluation into their program logic models, and use learning for strategic planning. Her clients include a range of anti-poverty and social justice organizations, particularly those that serve disadvantaged and minoritized youth.

Dr. Blakely received her BA from Brown University in Organizational Behavior and Management and Anthropology, her MS in Administrative Science from Boston University, her MEd from Harvard University, and her PhD from Brandeis University in Social Policy.

June 2018 Blog: Six Questions with Leslie Goodyear, AEA President

Wed, 20 Jun 2018 19:20:07 GMT

At our spring GBEN social meet-up, GBEN was very fortunate to have Leslie Goodyear, American Evaluation Association (AEA) President and Principal Research Scientist at EDC, join to talk about the upcoming AEA conference in November as well as her vision for the AEA.

In addition to joining the spring meet-up, Leslie generously agreed to participate in a short six-question email interview with GBEN member, Marion Cabanes. Enjoy!

GBEN: In 2016 you were formally elected the next president of the American Evaluation Association. You expressed a hope to foster a collaborative approach among evaluators within the association and among other organizations, policymakers, scholars and practitioners. Reflecting back on more than one year of this vision, what collaborative dialogues have you witnessed and how do you envision continuing and developing further dialogues on the importance and use of evaluation?

Leslie: Whoa! I said that?!? Just kidding. Both during my time as president-elect, and this year as president, I’ve had the chance to see how evaluators are influencing the field and those with power to influence policy. I speak mostly of what’s happening in AEA, but we know that every day, evaluators provide evidence and information to guide decision making and policy at multiple levels.

You may know that through the Guiding Principles Task Force and the Evaluator Competencies Task Force, members have influenced the review and revision of our ethical principles and helped to define a set of AEA Evaluator Competencies. Both Task Forces took their charge to solicit member input very seriously, and implemented processes that incorporated the ideas and opinions of hundreds of members (through surveys, focus groups, and other calls for information and input). The AEA Board just voted to adopt both, and we’ll be rolling them out before and at the 2018 conference. In addition, our Evaluation Policy Task Force has had multiple successes this year in influencing the development and implementation of federal evaluation policy. Their work is not necessarily loud, but has resulted in strong relationships with policymakers and other influencers in Washington. I have no doubt we’ll see more from them in the coming years, too.

GBEN: The field of evaluation has been evolving rapidly and seeing a greater diversity of evaluators of different stripes. What do you think is the next era for evaluation in terms of innovations? Or what new fields could evaluation reach out to for informed decision making and organizational learning?

Leslie: When I started in evaluation – many years ago – we were a field of academics and we debated whether it was appropriate to advocate for the programs we evaluate and we argued about quantitative versus qualitative methods. Things have changed a lot since then! Now, we’re primarily an association of evaluation practitioners, and we’ve moved on to incorporate and debate new approaches to evaluation (e.g., feminist, developmental, systems) and new positions on everything from equity and inclusion, advocacy, methods, program theory, and 3D logic models. We offer more opportunities for professional development now, and we have new and more dynamic ways to present data (qualitative and quantitative!) and disseminate findings. I’m not a psychic, so I’m not sure what’s coming next, but I’m excited by the passion people have for making the evaluation field more diverse, and its processes and products more directly tied to decision making and action.

GBEN: You have travelled across the country and met many other local/state evaluation associations. What do you think makes for a strong local evaluation association or network?

Leslie: It’s been a real pleasure to get to meet with so many evaluators as I’ve visited local affiliates and attended their meetings. As an organization, I think we can do more to connect the national organization to its local affiliates, whether through co-sponsored events, the common brochure, the common member registration, or other ideas. I love that there was an #EvalTwitter chat with the Local Affiliate Council and that so many people participated! I’ve heard from local affiliates that they would love more opportunities to share lessons learned and strategies, and that they would appreciate more opportunities to connect more with each other. I’ve been ensuring that the ideas that have been shared with me are passed right along to AEA staff who can collect them and, when possible, implement them.

GBEN: In addition to our bi-monthly roundtables, this blog post is our way of engaging with GBEN members. What would you like to know from the GBEN community to that could influence your work at the national level?

Leslie: First, let me just say thanks for the invitation to contribute to the GBEN blog. I loved getting to meet GBEN members at the social event in Cambridge, and I look forward to connecting again soon, whether at the AEA conference in Cleveland or another GBEN event.

I’d love to know what you’re seeing in your evaluation worlds! Are funders and programs drinking the evaluation kool-aid? Are there opportunities the association, or local affiliates, could capitalize on? Are there challenges you’d like to share that you think might be more common than just to you in your work? What do new evaluators need from AEA? What about seasoned evaluators? What are you seeing with regard to trends and concerns?

GBEN: What is your favorite part about being an evaluator as well as being a part of the local and national professional evaluation community?

Leslie: Easy! My favorite part of being an evaluator, and being part of the local and national community of evaluators: meeting smart, dynamic, engaged, quirky, diverse, passionate, thoughtful, people who want to use their amazing skills to make this world a better place! (However you define, operationalize, and measure that. Ahem.)

April GBEN Roundtable Summary - Having Conversations with Stakeholders Unfamiliar with Evaluation

Wed, 23 May 2018 19:51:54 GMT

Having Conversations with Stakeholders Unfamiliar with Evaluation

By Marion Cabanes

On April 4^th, GBEN hosted a roundtable discussion with members to share their experiences talking with stakeholders unfamiliar with evaluation.

Members discussed cartoons from the well-known evaluation website, www.freshspectrum.com. Many of these cartoons portray situations where an evaluator experiences resistance to, or lack of understanding about, evaluation from a client, and how to go about changing the client’s perspective on the role of evaluation for a particular project.

When you implement your project activities every day for many years, one may think they know the nuts and bolts of a project and what the clients need. The reality, however, can be much different. For example:

Evaluation tends to be overlooked, or project staff simply don’t see how evaluation can capture their successes or areas in need of improvement of the project. They might also lack time for evaluation as they focus on implementing their project activities, especially where their clients’ immediate livelihoods are at stake (i.e., locating beds for the homeless).
Or, there is still a failure to understand how evaluation can showcase project activities and results. Without evaluation, you lose the utility of the data you’re routinely collecting to serve the purpose of reflecting on your project and learning from it. It then gets harder to show the “specialness” of your work.

When talking about the “specialness” of a project, make sure goals are explicit and you are able to explain how the project will succeed or is meeting goals. Never underestimate spending extra time developing an evaluation plan at the beginning of your project as this will help you stay on track and/or measure how far off you are from reaching the ‘real’ goal, la raison d’être, of your project.

Evaluation can oftentimes feel like just another administrative task to tick off at the end of a project’s life cycle. GBEN members offered up some great tips on how to frame the role of evaluation differently with clients:

Stakeholders have an ethical responsibility to act, intervene, and assess that their efforts and investment are worthwhile.
Evaluation is an opportunity to showcase how good a project is and communicate successes and learnings to project funders, stakeholders, and beneficiaries.
Evaluation is an integral part of developing a strong culture of learning. Evaluation will help you and your team perform better and learn as you are solving problems along the way. Evaluation provides data to help make feedback-based informed decisions for corrective actions.
It’s important to scope your evaluation appropriately by choosing wisely what’s important (and not important) to evaluate. What is the right level of evidence that shows whether or not your project is working as designed and intended?

A Cool Resource to help project staff measure project progress and performance and demonstrate results to the world is a “results scorecard” software like Clear Impact and its Scorecard software or Asana.

Click here to access the presentation from the roundtable (members only!).

- Marion Cabanes is a local Boston evaluation specialist working in international development as well as an active member of GBEN.

Welcome to GBEN's new website!

Danelle Marable — Tue, 02 Jan 2018 15:42:29 GMT

Along with our new website comes official membership! GBEN's membership dues are reasonable and in-line with other AEA affiliates. Once you are a paying member of GBEN, you'll have access to the monthly newsletter, featuring events, jobs, and networking opportunities. You'll also have access to the member forum, where you can pose questions to other members. Looking to network with specific people? You'll have access to the member directory where you can search for people in specific fields or organizations. You get all of this for $25/year. Are you a student? Sign up with your student email address and enjoy the benefits for only $15!