Research Plan: External Evaluation of Steel Mosaic Orchestra
The stakeholders for the external mixed-method, naturalistic evaluation of Mosaic Steel Orchestra are Side-Stage Ensemble participants, board members and full-time teachers. The initial consultation will determine the best method of measurement in regard to how effective is Mosaic Steel Orchestra at instilling leadership, focus, and teamwork in members of Stage Side Ensemble. The evaluator will discuss specific options with members of the Board of Directors and instructors through email or telephone.
Performance indicator statements for the anonymous survey will be agreed upon by the aforementioned parties. The evaluator will distribute the survey to program participants in order to receive feedback about Mosaic Steel Orchestras' impact on the skills that are believed to build psychological resilience in a diverse population of young adults. An interview with the Director will discuss how accessible Mosaic Steel Orchestra's after school programs are for the at-risk youth population in the community.
During several Mosaic Steel Orchestra band practices, the evaluator will engage in non-participant observation. This will also take place during a public performance in Norfolk, Virginia. The naturalistic information will be recorded through written field notes. This will include observations of participants and teachers, and personal interviews using open ended questions. The evaluator will integrate these experiences in order to “refine my initial impression” (Posovac, 2011). This leads to revision of evaluation questions which can benefit the assessment in a more specific way.
Mixed resources will be consulted in order to get a scope of the impact MSO provides, and to propose beneficial changes for stakeholders. The evaluation will include naturalistic observation during band practice and a performance, participation through personal interviews with stakeholders, and written anonymous surveys for Side Stage Ensemble members. The use of multiple instances of observation allowed for a replication of the design, to see how members behave in different settings. If the behavioral patterns are similar, it can be confidently concluded that collaboration in Mosaic Steel Orchestra has an impact on the behavior.
Anonymous written surveys will be distributed to all members of Mosaic Steel Orchestra Side Stage Ensemble in order to get an honest view of the relationship “between outcomes achieved and the outcomes projected”(Posovac, 2011). Students will answer questions about strengths, weaknesses, opportunities and threats related to the program. S.W.O.T. surveys require feedback about internal and external aspects of the program from an insiders perspective. Anonymous surveys are useful for improvement focused assessments because participants have first hand knowledge of the program's efficacy- they can best identify discrepancies and problems. This information will be consulted and the overall data will be compared. The external evaluator will suggest a change for a positive improvement in the program based upon the mixed methods of research.
References
Posovac, E. (2011.) Program Evaluation, Methods and Case Studies. Upper Saddle River, NJ: Prentice Hall.