Evaluating Fieldwork in the Digital Space: A Case of Introduction to Evaluation Tools for Socio-Musical Projects in Latin America

Authors: Michael Rivera (USA), Gustavo Samudio (Panama)

1. INTRODUCTION

Knowledge and experience in program evaluation is essential to the growth and sustainability of social organizations, both to demonstrate visible impact to funders and partners as well as to improve the internal processes and structures upon which these organizations operate. While knowledge about evaluation is widely available online and in the academic setting, there are few Spanish-langauge resources and opportunities that provide program evaluation knowledge for professionals in the area of music and social change. The goal of our fieldwork assignment was twofold: to provide a Spanish-language virtual seminar to teach basic evaluative tools for such programs, and to provide a space for collaboration and networking for organizations with common missions across the region.

The current report will present the insights about the fieldwork we conducted between March 27th and June 24th— also known as Seminario de Introducción a las Herramientas de Evaluación para Proyectos Socio-Musicales en Latinoamérica— as part of our final project for the Global Leaders Program (GLP). The information is organized systematically, starting with a general introduction to the Latin American region and why we have decided to focus on this specific demographic. The report will then cover the methodology and curriculum design for each of the seminar sessions, the evaluation and results of the collected data, and an analysis and discussion about the future implications of these results. Based on our observations, we suggest that virtual fieldwork is a valid and often overlooked opportunity for knowledge sharing that is available to GLP members and, despite deviance from the traditional fieldwork model, presents high potential for customizability and far-reaching impact.

2. RATIONALE – EXPLANATION FOR MODIFIED FORMAT

After reviewing the possible options for the fieldwork assignment based on the broad offerings of The Global Leaders Program, we were unable to proceed with most of them for various reasons, many of which were related to the current global pandemic. As a result, we collaborated to design a creative, virtual webinar on evaluation tools that would provide value to our fieldwork host(s), even at a distance. Each of us had individual motivations to pursue this unique virtual fieldwork opportunity. Gustavo wanted to extend his Latin American network in the music field and gain value and meaning from all the music education experience gained during the last seven years of his career in the US. Michael wanted to synthesize his professional knowledge as a first-year program evaluator for a music for social change nonprofit and apply this knowledge within the cross-cultural context, specifically for Spanish-speaking communities.

Consequently, we worked with the GLP fieldwork staff to propose a solution that is now the subject of this case study. By tapping into the GLP network in Latin America, we designed a project that allowed us to share our specialized skills in program development and evaluation on a larger scale. To our knowledge, a virtual presentation or webinar about tools for evaluation designed for Spanish-speaking arts organization leaders has not been developed before as a focus of the GLP fieldwork assignment. This report therefore seeks to describe the results attained through this new venture as well as reflect on the possibility of designing future fieldwork assignments of the same nature.

a. Socioeconomic and cultural backgrounds in Latin America

It is essential to understand the dynamics of the economy in Latin America to be able to find the best suitable solutions for different social problems in various countries in the region. The work of social and music organizations in Latin America faces challenges at many levels. Most of them are related to the inherited economic issues of post-colonialism that still hinder the development of different countries in this region.

Based on the 2013 report by CEPAL (Economic Commission for Latin America and the Caribbean), economic equality still faces structural change and shows specific trends in economics and social development. For instance, the data clearly shows a reduction in the inequality gaps between 2002 and 2011 (Appendix A). The crisis has shifted to a financial crisis in which private over-indebtedness directly influences long-term employment, and recoveries are vulnerable to economic and financial volatility. (Cepal)

Even before the current pandemic crisis, the Latin America and Caribbean region showed resilience with GDP growth of 3.1% (above the global average of 2.2%), a fall in employment, increased investments, and fiscal prudence. These facts are relevant when compared to the global GDP, particularly in countries such as China with a slowdown from 9.2% to 7.7%, and modest growth in the US between 1.8% and 2.1%; and financial and fiscal imbalances in Europe with a recession of -0.5%. Further, when thinking about long-term challenges, there seem to be positive changes in inequality, investments, productivity, taxation policies, and environmental sustainability. All these will be possible if policymakers develop economic policies based on relevant, long-term sustainable vision at the macroeconomic, productive, and territorial levels that will take advantage of international context with a new equation:

State-market-society(Cepal).

At a cultural level, Latin America is one of the richest regions in the whole world due to diverse migrations that have happened over generations, blending African, Caribbean, and natives of the region and creating a melting pot of traditions, thus influencing their food, dances, poetry, music and more. Hence, it is not surprising to notice the significant number of artists in different arts fields from this region despite their many limitations throughout the years. Further, it is crucial to realize that their governments have abandoned the Latin American Arts movement in many cases. However, politicians and social leaders have recently started to realize the great potential in arts, particularly in music, to change those social patterns and produce new generations of creative professionals. This shift in leaders’ mindsets can offer a new future to creative individuals, changing their life trajectory, and consequently enhancing their communities.

b. Content accessibility in Latin America

Since the beginning of our project, we acknowledge the possible challenges in finding specific academic content for our participants, particularly with content tailored specifically for social and music organizations. However, this challenge represented an excellent opportunity for us to tailor, provide, and present a seminar for all those organizations and individuals who were willing and eager to learn about these fascinating fields related to music and social change.

Therefore, we decided to specifically focus on program evaluation since it is the expertise that was readily available for our team, developing the whole curriculum around that theme with a series of five sessions that will be further discussed in this current report.

Most of the information given to the participant was tailored for cultural and music projects using the Zoom sessions and our presentations. However, to introduce the different themes, the curriculum was curated cumulatively, having preparatory assignments (readings), which included texts drawn from various disciplines, including public health and educational research.

c. Limitations due to Covid-19

With the unprecedented situations related to the global pandemic, the Global Leaders Programs, the partner organizations, and the participants faced challenges, including the inability to travel safely to different sites worldwide. At the same time, some participants were able to conduct an alternative version of their fieldwork with some of the global allies of the GLP’s network. However, other sites could not achieve those goals due to specific local problems and restrictions, mainly due to the lack of accessibility to technology and reliable internet. These challenges across the board revealed how GLP members from their own perspective adapted and reacted during these difficult times. Our final fieldwork project represents a summary of those challenges through months of trial and error in different meetings with various organizations and GLP administration, until we were able to deliver our final product of the seminar.

3. METHODOLOGY AND CURRICULUM DESIGN

a. Course Objectives and Outline

We created the course based on several core competencies that serve as the foundation of evaluation: knowledge of what program evaluation is, the reasons for evaluating a program, and how and when to use evaluation. While providing a comprehensive course in evaluation is impossible in the span of five one-hour sessions, our objective was to provide foundational knowledge about evaluative tools that can be applied to music for social change organizations in Latin America.

The seminar included four lecture-style sessions of 60 minutes each and a final interactive workshop of 120 minutes (Figure 1). We used Microsoft Powerpoint to synthesize and present our curriculum, and hosted and recorded live each session on Zoom. All materials, presentations, bibliographies, and email correspondence were delivered in Spanish, and all materials found in English were translated by the presenters.

 Seminario de Introducción a las Herramientas de Evaluación para Proyectos

Socio-Musicales en Latinoamérica (Seminar: Introduction to Evaluative Tools for Social and Music Organizations in Latin America)

Session 1: What is evaluation and why do it? An Introduction

• Understand and define the types of evaluation (i.e. formative, process, impact)

Session 2: Understanding the Structure of an Organization: The Logic Model

• Understand how to create and interpret a logic model
• Participant Activity: Present mission statement of organizations in less than 30 seconds

Session 3 : Evaluation Methodology and Design

• Ethical evaluation design
• Quantitative tools overview: survey design
• Qualitative tools: interview design, focus groups, observation

Session 4: Evaluation instruments: implementation

• Continuation of qualitative and quantitative tools
• Data analysis and presentation
• Participant Activity: Breakout room discussion of shared experiences in evaluation

Session 5: Workshop and Practical Application of Evaluative Tools

Figure 1: Outline of the five-session seminar

b. Participant Selection

We originally envisioned the series of five seminars to be a series of informational webinars, in which participants could attend the sessions of their choosing and ask questions at the end of each session. However, in recognizing the potential value of cross-organizational collaboration and networking, we decided to shift the seminar into a more interactive format, allowing for smaller breakout rooms, more frequent Q&A breaks, and individualized coaching, particularly in the fifth session. In order to maintain a manageable presenter to participant ratio, we limited the seminar registration to 15 organizations, with two representatives from each organization, resulting in a total limit of 30 participants.

c. Overview of Sessions 1 – 4

The first four sessions were presented in lecture-style format, with opportunities to ask questions live or using the chat box, and also integrated opportunities for group work throughout. As shown in Figure 1, the first session explored the questions of “What is evaluation?”, “Why evaluate?”, and “When should we evaluate?” This particular session also covered the major types of evaluation that a program might undergo throughout its lifespan.

The second session covered the logic model as a tool for evaluation, based on the idea that one cannot evaluate a program without first knowing the program’s purpose. Using examples of model organizations given through previous GLP materials (i.e. Play on Philly), we led an entire session on what a logic model is, why organizations should use a logic model, and how to create one.

The third and fourth sessions covered the practical tools involved in evaluation, as well as the ethical considerations involved during design and implementation. These sessions specifically covered qualitative methods (i.e. observation, interviewing, focus groups) and quantitative methods (surveys), as well as types of biases involved with each.

d. Practical Application: Fifth Session Workshop

During our fifth session, we designed an interactive workshop focused on developing a logic model and evaluation plan for a hypothetical organization in Brazil called Amigos por el Arte en Alcántara (see the Appendix B). This session lasted for a total of 120 minutes and included two parts. In the first part, we reviewed the logic model and divided the group into three separate breakout rooms, instructing each team to focus on developing a different aspect of the

organization’s logic model. In the second half of the workshop, we provided a short tutorial on using Google Forms, and instructed the groups to create their own surveys according to their target audience and the guidelines established for the group. These target audiences and situations included the following: 1) parents interested in enrolling their children in a newly founded music program, 2) music teachers on staff with the program, and 3) students who have finished the program. These situations intentionally reviewed knowledge of question design specific to the three major types of evaluation: formative, process, and impact.

4. EVALUATION AND RESULTS

As presenters of a seminar on evaluation, we also took it upon ourselves to evaluate the impact and efficiency of our sessions. At the end of every session, we sent out a survey measuring the effectiveness of our presentation with regards to content and delivery. All surveys were written in Spanish and designed using Google Forms, and we sent them out in a follow-up email 1-2 days after the end of each session.

The post-survey included the following 6 items:

We also administered a pre- and post-assessment, which we called Conocimientos y Experiencia (Knowledge and Experience), in order to gauge any measurable change in the participants’ familiarity and comfort level with understanding and using evaluative tools.

a. Knowledge and Experience: Quantitative Data

Ten participants took the pre-survey and eight participants took the post-survey. In the final question of both Knowledge and Experience surveys, we asked participants to rate their overall comfort level with creating an evaluation plan for their organization; as demonstrated in Figure 3, nearly half of the participants who took the survey moved up by one point, demonstrating that they had gained some level of familiarity with using evaluation tools.

We also measured experience and knowledge of specific tools after sessions 1 and 5 using a four point scale: 1 = I have no no knowledge; 2 = I have basic knowledge ; 3 = I am

knowledgeable and have used this tool for my organization ; 4 = I am very knowledgeable and use this tool regularly. The nine “tools” presented were as follows:

1. Recognizing the different types of evaluation
2. Creating an evaluation plan
3. Creating a logic model
4. Designing surveys and/or questionnaires
5. Designing individual and group interviews
6. Implementing individual interviews
7. Implementing focus groups
8. Analyzing evaluation results
9. Applying and/or presenting evaluation results

Of the nine, we focused on analyzing the results of the three major themes that were covered in the first four sessions: 1) Recognizing the different types of evaluation, 2) creating a logic model, and 3) designing surveys and/or questionnaires. These results are shown in Figure 2.

As far as overall increase in knowledge and experience, the three major groups saw increases after the fifth session. Regarding knowledge of the logic model, it seems that 40% of participants who took the survey had no knowledge of this particular tool before the seminar. In the post-survey, nearly two-thirds of participants reported that they were knowledgeable of the logic model and had used it for their own organization over the duration of the seminar. One notable success is that one person reported that they now feel they are very knowledgeable in using the logic model and recognizing different types of evaluation, in addition to now using these tools regularly for their organization– indicated by the green bars in the post-survey.

With regards to knowledge of survey design, everyone reported that they had at least a basic knowledge of the tool, as opposed to the wider variance in the pre-survey. The amount of people who use surveys regularly did not change, as it seems that the majority of our participants had encountered survey design for the first time during our seminar.

Q: What is your current knowledge of each of these evaluation tools?

Pre-Survey

Post-Survey

Figure 2: Pre- and post- survey results of Knowledge and Experience for Item 4: Designing Surveys/Questionnaires, Item 3: Creating a Logic Model, and Item 1: Recognizing the different types of evaluation.

Figure 3: Q7 asked (in Spanish): “In general, I feel that I am able to make an evaluation plan for my organization.” The pre-survey (top) shows a lower average response than the post-survey (bottom), which was implemented five sessions later.

Knowledge and Experience: Qualitative Data

Our surveys also collected small segments of qualitative data– that is, short- and long-form responses — through the use of Google Forms. We asked two open-ended questions: 1) Do you have any suggestions that could help us improve the delivery of the session? And 2) Do you have any suggestions that could help us strengthen the content of the session? The comments can be grouped into 1) the need for more time and practice, 2) the desire to interact more with other participants, and 3) miscellaneous feedback and thanks.

The majority of comments involved the desire for more time and more practice to hone the new evaluation skills. It is worth noting that while our original time limit for the Session 5 Workshop was 90 minutes, we extended it to 120 minutes based on high participant interest and engagement with the activity.

In addition to comments about needing more time to practice skills, it appears that participants were also interested in the collaborative and networking aspect of the activity. These types of comments point to the benefits gained from cross-organizational work.

One comment was very telling about the overall benefit of a cross-cultural presentation. Fieldwork is an opportunity to exchange perspectives in a respectful dialogue; for this participant, it seemed that their understanding of the logic model improved when presented from a different angle.

Finally, one commenter referenced orthographical errors in the slide presentation. This type of feedback will certainly be taken into account in the future because presenting information in the cross-linguistic context also involves adhering to the correct usage of formal, academic Spanish.

5. DISCUSSION: VALUE ADDED

a. Impact of Curriculum

As our objective for the seminar focused on providing a foundation upon which participants could build and expand their knowledge, we expected a self-reported knowledge increase in the surveys. One must also consider the number of people who took the K&E survey, including only 8-10 participants out of the 29 enrolled. We know that not all participants attended all sessions and that not all participants were actively engaged. However, of the 8-10 who responded (10 took the pre-survey, 8 took the post-survey), they were generally the same participants who consistently participated and reported positive gains in their evaluation knowledge and overall experience in the program. The survey results of participant comfort levels in creating their own evaluation plan mirrors this overall sentiment of increased-self efficacy when it comes to implementing these tools on their own.

During the development of the curriculum, we also experienced particular challenges in obtaining and synthesizing content. In general, there is a lack of available content about program evaluation in Spanish, and there is much less customized content for organizations that focus on music for social change. In order to fill this lacuna in the literature, we combined the information we could find (all of which can be found in the bibliography) with real-life examples from organizations and models for music and social change. While we could not find any current models of evaluation for such programs in Latin America, we took current models from the US, such as Play on Philly, and presented this content in Spanish. Several participants requested a bibliography of resources to continue learning about evaluation past the end of the seminar; this list is found in the Works Cited section.

b. Considerations for Cross-Organizational Fieldwork

Due to the nature of our webinar-style format, we were able to take advantage of our situation and invite multiple organizations to participate. Based on our observations, we recommend offering GLP members the possibility of projects that involve in-person fieldwork with multiple sites. The opportunity for GLP members to collaborate with various organizations within the same region can offer multiple perspectives that working with one organization cannot. These experiences may range from differing models to sharing the challenges experienced by each organization in their individual contexts. Perhaps of equal importance is the opportunity for organizations in the same region to network among each other; as we witnessed during our

seminars, many organization representatives were eager to share their social media and contact information with each other— some of these organizations even originating from the same country.

c. The Future of GLP Fieldwork: Flexibility and Customizability

There is no doubt that COVID has impeded the quality of many learning opportunities, including the benefits gained from in-person instruction and travel. However, this “pilot year” for virtual GLP fieldwork has revealed several unique advantages that could benefit future cohort members interested in participating in remote fieldwork. As mentioned before, the online seminar has provided the opportunity for multiple organizations to convene and network despite geographic distance. Participants were able to share about their organizations, ask questions, and collaborate with and learn from each other comfortably and cost-effectively. Additionally, virtual opportunities for fieldwork can encourage substantial participation in fieldwork for those who cannot otherwise engage due to previous professional and personal responsibilities and financial constraints. Virtual fieldwork can also encourage GLP members to find their own creative solutions to challenges present only in the digital space. These may include gaining familiarity with technology in the classroom (music teaching or otherwise), as well as different methods of lecture presentation and group facilitation.

Nevertheless, we expected and still experienced regular disadvantages that occurred when conducting presentations virtually. Many of these impediments included technical issues related to software and hardware. In particular, the online setting can encourage disengagement and/or multitasking due to the on/off camera setting. In every fieldwork situation, there will always be some lack of involvement among a small group of participants; in the virtual setting, this group number tends to be higher. While we cannot (and perhaps should not) equate a closed camera to disengagement, the classroom community on Zoom does not feel as strong when there are fewer faces present.

Regarding the co-designed nature of the program, we understand that we have taken liberties with the traditional fieldwork format, as described and justified in Part 2: Rationale. While the traditional format of GLP Fieldwork and Case Study is an informative and beneficial model, we do not believe that fieldwork options should be limited to one or two participants to work with one organization. Traditional fieldwork assignments are generally taken over the course of 10 intense days, which would allow a cohort member to achieve some sort of short-term goal while focusing on developing a relationship with one host organization. However, one could also

argue that the value of the fieldwork and case study is also to allow the GLP participant to reflect and synthesize their experiences in the GLP, in addition to providing added value to the organization(s) with whom they are working with. Fieldwork, therefore, represents a high potential to become individualized to the personal and professional interests of the participants, and modifying the traditional fieldwork and case study format is one way to achieve this customization. Our personal interest in Latin America as a collective region and the lack of Spanish language presentational material on evaluating music for social change programs thus gave way to a multi-organizational, multi-week seminar.

6. CONCLUSION

Despite the chaos and complications brought on by the COVID-19 pandemic, these past two years have also given individuals and organizations the opportunity to self-reflect and consider larger existential questions as related to their personal mission and action. In the same vein of self-reflection, evaluation is the methodical process by which an organization can truly understand itself, its actions, and its relationship to its many stakeholders. By providing the opportunity to various organizations to learn about the language of evaluation in the virtual space, we hope to have increased access to this body of knowledge during this relevant period of reflection. While we cannot claim to make any one organization an evaluation expert after five introductory sessions, our objective was to provide a working foundation of knowledge upon which these organizations can continue learning and experiment with individualized evaluation plans. Based on the feedback received from our own evaluative tools, we see our success not only as an increase in theoretical and practical knowledge, but most importantly as increased self-efficacy and confidence to know how and where to learn more about evaluation.

In addition to adapting evaluation knowledge in linguistically and culturally appropriate ways, we were also able to create a space in which like-minded individuals could interact, collaborate, and share experiences. This unique aspect of our fieldwork assignment provides valuable implications for the future: firstly, that working with and facilitating multiple organizations can provide benefits not found in the traditional GLP fieldwork model, and second, that the fieldwork assignment and case study as defined by GLP can be successfully carried out in alternative forms (most evidently in the digital space). While fieldwork can often be a multi-party negotiation between the varying agendas of the GLP member and host organization(s), flexible fieldwork models customized to a GLP member’s interest can help to provide one creative solution to this balancing act.