On this page, I’m sharing my writing for PME 802, a course in Program Inquiry and Evaluation.
My Program Evaluation Design website
Step #1. Select and describe program context.
Focus and Goals
The program I have chosen to evaluate is the Concours d’art oratoire, run by the Canadian Parents for French (CPF). As a Core French teacher, this is a program that connects directly to my teaching context, providing opportunities for both me and my students to get involved. Although this program is run by an external organization, we are invited to participate each year, and the target audience is school-aged children registered in French language programs in public schools across Canada.
I moved to a new school a year and a half ago, and am now working in a school that also houses a French Immersion program. This provides me with a first-hand opportunity to observe aspects of French Immersion programming and its appeal to or impact on students. While both French Immersion and French as a Second Language students can participate in Concours, there are separate categories for each program so that students from these two programs are not competing against each other.
The focus of the Concours d’art oratoire is aligned with the umbrella organization’s mandate of “furthering bilingualism [in Canada] by promoting and creating opportunities for students to learn and use French.” The Concours d’art oratoire is the largest annual French public speaking program in Canada for second language students in grades 6-12, in both French Immersion and Core French programs, as well as Francophone students. Although the Concours is a national program, it is structured to begin in local classrooms. The event is organized by parents and school staff, and requires a high number of volunteers, as close to 10,000 students across Canada participate each year. Volunteers also include university and college professors, teachers, native Francophones, and former Immersion students. Students who are successful at each level progress through school district, provincial and national level competitions. The stated aim of this program is “to stimulate the interest of students learning French, to improve their speaking skills, and to give them experience presenting in public.” I have been involved previously at the classroom and district levels, and my interest was piqued by the potential to evaluate this program in light of some questions I have (more on that below).
Size of staff
The portion of the Concours which takes place in schools is run by volunteers, and depends on teachers to volunteer to begin the initial phases. Teachers serve as coaches, coordinators and judges within their own schools, and must follow the rules and evaluation rubric laid out on the CPF website. Once the school-based competition has determined the winners, those students progress to the school district level. In my district, this competition is held on the campus of a local university, where more volunteers coordinate the event. If a teacher is sending a student to a competition, he or she is requested to serve as a judge in order to help support the event.
The CPF website lists a number of resources for teachers to use at the classroom level to prepare for Concours, supporting students in selecting topics, preparing speeches, providing lessons connected to public speaking, as well as additional resources and evaluation criteria. Not directly related to Concours, the CPF website also lists more general resources for youth, parents and educators, promoting second-language literacy as well as collaboration between educators and parents in support of youth.
Given that the Concours begins in schools, the instruction and facilities are integrated into programs of instruction and take place largely within the context of a school day. Once the higher levels of competition are reached, events take place at local schools and universities.
I am choosing to look at this program through the lens of its use and impact in my school district. I work in Surrey, BC, which is home to the largest school district in the province. There are 125 schools district-wide, including elementary and secondary schools. There are currently almost 74,000 students registered in Surrey schools, and the district offers Core French, Intensive French, and French Immersion programs. The popularity of French Immersion programs in particular continues to increase province-wide. There is a local chapter of Canadian Parents for French who primarily communicate through their Facebook page, which is used to organize and promote local activities for students and parents. The Twitter account for CPF-BC is used in both English and French to promote stories of interest, news, and events.
Surrey is a diverse and multicultural district, with 195 languages other than English represented in our schools. Of those other languages, those with the highest percentage of representation within the student body are Punjabi, Tagalog, Mandarin, Hindi and Arabic. A speech and film contest for Punjabi-speaking students is in its third year of operation in Surrey schools, and is co-sponsored by OMNI TV and community organizations. Given the multicultural nature of our community, it is important to consider the impact of Concours d’art oratoire in schools as well as its appeal and impact on students from a wide range of cultural and linguistic backgrounds.
Step #2. Identify purpose for evaluation and specify evaluation questions.
The purpose of my evaluation is to look at the impact of the stated goal of Concours on learning for Core French students in my district. Although this program started primarily for French Immersion students, it has incorporated Core French students as well. This tends to mean that the teachers who volunteer have less experience as Concours participants, and that can impact participation rates. In my evaluation, I would like to investigate the answers to the following key evaluation questions, which are based on the practical requirements for running Concours as well as the mission statement of the parent organization (Canadian Parents for French):
- Do Surrey Core French teachers regularly participate in implementing and supporting Concours?
- Is Concours providing a meaningful opportunity for Surrey Core French students to learn and use French?
- What recent trends have emerged in Surrey School District Core French student participation in Concours?
- Are we seeing diversity in the students choosing to participate in this program?
- Are we seeing diversity in the students who are successful at the school or district level?
- Do the participation rates indicate that this program is reaching its intended audience?
- What do these trends tell us about the impact of Concours in our community? Does this impact tell us whether and how anything should be changed in the way we are inviting students to participate?
Step # 3: Construct a program theory.
In the table below, I have summarized a basic model of what the Concours d’art oratoire would look like at the school level, which is the level at which it will impact the greatest number of students.
|Activity||IF the activity is provided, THEN what should be the result for participants? WHY do you believe the activity will lead to this result?||WHAT evidence do you have that this activity will lead to this result (data from your own or other programs, published literature, etc.)?|
|Students participate in French language public speaking activities run by teachers in their school on topics of interest to the students. Teachers will use the resources listed here on the Canadian Parents for French website.||If this activity is provided, then the result for students should be increased skill and confidence in public speaking in French.||It is generally accepted that when people practice a skill, they will likely improve in their performance of that skill over time. With the support of a teacher and peers, as well as resources from the Canadian Parents for French (and other resources the teacher may have at their disposal) in the familiar setting of their own classroom, this improvement is more likely to occur. Speaking is a normal part of second language programs, and is incorporated in the BC Core French curriculum.|
|Students select topics of interest and write and memorize speeches on those topics, in French, with assistance from their teachers.||By selecting topics of personal interest, students will be more engaged and motivated. In writing and memorizing their speeches, students will engage directly in the process of learning to deliver their message to their intended audience.||Bandura’s social learning theory supports the interaction of the processes of attention, retention, motor reproduction and motivation as being at the heart of social learning, which incorporates both thinking and behaviour. Selecting a topic of personal importance will also lead to a more meaningful experience for student participants.|
|Students will enter a school-based French language public speaking competition to be judged by teachers and other adult volunteers, using criteria from the Concours d’art oratoire. Students will also respond in French to questions from the judges relating to the content of their speech.||Students will be motivated to revise, memorize, and perform well at a school-based competition and will practice responding to questions on their speeches.||Self-regulated learning, which is in part built from the theories developed by Albert Bandura, supports student learning, motivation and efficacy through repetition of the cycle of planning, monitoring performance, reflecting on performance, and then using the results to guide the next performance. This cycle is at the heart of instructional design for teachers, and supports student learning, motivation and performance in tasks of this nature. This is not an add-on to incorporating activities related to the program in classes because self-regulated learning theories were based on observations of what people do naturally while engaged in learning.|
|The winners of the Concours at each grade level will go on to compete, using the same speech, at the district level.||Students will be more confident after having won at the school level, and will be more motivated to improve their performances for the district level competition.||As mentioned above, social learning theory and self-regulated learning will support student efforts at this level. In addition, the added motivation of winning at the next level increases the competition and motivation of students to do well.|
Step #4: Identify, describe, and rationalize your evaluation approach:
Program Theory – rationale, description and approach
My goal with this program evaluation is to look at the impact of the Concours d’art oratoire on Core French classes in my school district. Although the Concours is a national program, I am looking specifically at my local context.
For this evaluation, I am using the Evaluation for Learning and Use model explained here by Dr. Jennifer Green. The program I am looking at has been in existence in Canada for more than 30 years, and I want to know if it is having its intended impact on students in my district.
The table above outlines the activities contained in basic model of the Concours d’art oratoire operating at the school level, which is where most students would be impacted by it. The activities are expressed mostly from a student perspective, although there is significant input required from teachers as well. The activities that teachers will engage in at the school level may or may not be integrated into their regular classroom instruction, which could potentially impact their ability or willingness to participate. The model (as laid out by the parent organization) presupposes that teachers will either volunteer time outside of class to run the program, or will find ways to integrate it into their classroom instruction. Regardless of which option teachers choose, the model also assumes that there will be a fit with programs and provincial curricula that will prepare students for public speaking and oral presentation on a topic of interest to them, rather than a topic integrated into a specific textbook or program. In my experience, these are reasonable expectations, an assumption which is also supported by the longevity of this program.
Because I am not looking at the setup of a new program, I am not looking at aspects of program setup that would involve looking for volunteers and finding facilities. This program has an existing infrastructure that supports those aspects. My purpose is to contribute to the quality of the program and its impact on students in my district. With this in mind I am also considering aspects of stakeholder theory as described by Chen, specifically the ones impacted by this program: the mission and philosophy of the organization, the budget and personnel restraints, the theoretical justification, and the base of evidence. As part of considering the impact of this program on my community, it will also be helpful to consider its relevance to the majority of students enrolled in Core French programs. This aspect is also incorporated by Dr. Green, when she recommends that evaluations of this type should examine how well the program under consideration addresses the priority needs of the diverse program participants.
Step #5: Identify data collection methods and analysis strategies.
I am designing an impact evaluation for a specific portion of a program that has been in existence for many years. Although the Concours d’art oratoire is a national program (Canada-wide), I am taking a narrower focus for my evaluation, looking specifically at its impact in the Core French programs of public schools in Surrey, BC.
The demographics of the area included in this focus are used in part to determine my methods to collect information. As mentioned in my summary of program context, there are 125 public schools in Surrey, with a total of approximately 74,000 students, and there are three basic models of French instruction offered at those schools, including Core French, Intensive French, and French Immersion. Core French is offered at the greatest number of schools, but the Concours was initially introduced in French Immersion programs and has a strong base of support and participation there. Core French programs offer the widest possibility for student participation, and because of the way in which Core French is staffed and delivered in elementary schools in Surrey, it is likely that most students will not participate until they reach high school, which is in grade 8 in BC.
Because of the numbers of students, schools, teachers and parents impacted by this program, the data collection methods will be easy to administer and will not require a great deal of time. Data collection will largely take the form of questionnaires and surveys, and will measure variables related to the key evaluation questions.
In order to measure these aspects of the evaluation, three similar but distinct surveys and questionnaires will be designed and administered. The target groups are teachers, students, and parents. If time and resources allow, a fourth target group could be identified, comprised of support people on the periphery of Concours who have the capacity to impact its success in a positive manner. These people could include administrators, counselors, teacher-librarians, EAs, peer tutors, former participants and community volunteers. Concours does not have a major media support campaign. Communication from the organization to teachers typically goes through a chain that involves a local chapter of Canadian Parents for French, a district helping teacher or program coordinator, district head office, school-based administrators, department heads, and finally classroom teachers who spread the message to their students. Although students are the ones primarily involved in the activities that form this program, each of the groups mentioned above is part of a communication and support chain that is integral to the success of the program.
Teacher data collection
The teacher survey will assume that teachers running Concours in their classes will have a basic proficiency in speaking and understanding French and will need minimal training in pedagogy relating to public speaking and assessment. Some variables that could be informative include other languages spoken by a teacher, and his or her prior participation in Concours either as a student participant, teacher sponsor, whole class implementation, or helping as a volunteer. If a teacher has participated previously but has dropped out of the program, tracking the reasons for dropping out are useful. It is also useful to track reasons for ongoing participation. Additional questions for teachers would include years of experience in Core French instruction, awareness of how to get information about the program, where to get resources to support the program, and ways to integrate activities for public speaking into classroom instruction.
While the majority of teacher information would come from participant surveys, another avenue of data collection would be focus groups or interviews conducted at individual schools. Surrey school district has been divided by district administrators into zones which form distinct regions, and are comprised of families of schools. Within each zone, there are slight differences in staffing, funding, available programs, service delivery, and demographics. The focus groups would be comprised of one school within each zone, and teachers within those schools could be asked to participate in interviews to give in depth responses to the questions asked in the questionnaire. These interview questions would also be an opportunity to get more information about what barriers may exist to participation, in the form of familiarity with the language, logistics, or other factors.
Student data collection
Student data collection would be most informative if it were comprised of all Core French students in the district. While this is not practicable for most evaluations, the goal would be to have as many respondents from as wide a range of schools and grade levels as possible. Questions for students could include grade level, other languages spoken, experience in public speaking, level of parental support and motivation for participation. Student motivation is a key component of program success, but is tricky to manage. It cannot be marks-based because the program is voluntary, and the incentive of tuition at select Canadian universities is only available to those at the highest level in grade 12. Answers to this question will be qualitative and will vary. They may be difficult to categorize but will be informative to assessing program impact. Students should also be asked about topic selection and whether they are able to speak publicly in French about topics that are important and meaningful to them. Past topics of winning students are listed in the database made publicly available by Canadian Parents for French, but there is no general database of topics selected by all participants.
While participation rates are being measured, there are some additional metrics of participation that can be examined, may vary from site to site, and may impact whether students continue to higher levels of competition. These include what kind of participation students engage in (speaking, coaching, judging, or volunteering), whether participation is voluntary and open or requires teacher recommendation, whether participation is integrated in classroom instruction or must be done outside of class time. Measuring these variables could be predictive of best practices for implementing this program to get maximum impact for students.
In schools where interviews are conducted with teachers, the same could be done with a select group of students. Students in this case would represent a range of grade levels and language ability, but in-depth responses to interview questions would be helpful in assessing program impact for students. In addition, students can be asked about barriers to participation such as language ability, other languages spoken, stigmatization among peer groups, or logistics.
Parent data collection
At the entry level of Concours, parental consent is not required, but parental support is important to the success and impact of the program. Measures of parent data could include languages spoken, ability to volunteer in support of the program, and views on language learning. While views and opinions are sometimes difficult to measure using surveys (which would be the primary method of information collection due to work schedules), 5-point scales can be used to measure degrees of support for language learning in general and French language learning in particular. Given the demographics of Surrey, it is important that this survey be available in multiple languages and supported by graphics where possible or necessary for situations in which language or multiple meanings could be a barrier to understanding.
Additional sources of data
Additional information is found on the CPF website in their database of past winners, grade levels, and topics. I am not aware of any public records pertaining to general participation in the initial levels of Concours, and so historical data will only contain information from those who were able to get to the highest levels of competition.
Observation of program implementation is another tool that can be used, but would likely take the form of another type of survey regarding time, location, frequency, resources and sponsorship of activities. Travelling site to site to observe programs in action in the beginning stages of public speaking activities in classes would be time consuming, costly, and could likely discourage student participation through the presence of strangers watching them speak publicly in a second language.
Step #6: Describe approach to enhance evaluation use
Data will be gathered by online surveys, questionnaires and interviews, in a manner that preserves participant anonymity and confidentiality. Interview participants will be able to review, edit, retract or revise statements before submission. Data will be gathered by schools, and would most efficiently be administered through the same communication channels used by CPF to notify schools of Concours. Data would be shared between the school district and the local chapter of CPF, as it would benefit both parties. In order to address the last key evaluation question, this survey would ideally be administered over at least two consecutive years, in order to measure impact with a more long-term view. With this in mind, it is especially important that surveys be clear, concise, and easy to complete.
Because the majority of data will be gathered through online surveys, the information gathered can be used to produce graphics which can then be presented to stakeholder groups including teachers, students, parents, district personnel and other support staff. This information can be used to advocate for additional resources and supports identified (for example, through questions about repeat participation, dropping out, and barriers to participation) as being important to program success.
Information will be used to support students participating in Concours and to encourage greater levels of participation. It will also be used to support teachers who may not be familiar with the program, and to encourage such things as teacher collaboration or sharing of resources, practices and times and spaces needed for program implementation. It will also be used to inform parents of the benefits of student participation and to involve them in the process as volunteers and student supports.
Step #7: Commitment to Standards of Practice.
The evaluation plan above adheres to the Standards for Program Evaluation in the following ways:
- Primary stakeholders have been clearly identified as being teachers, students, and parents. This program is an initiative coming from a parent organization, and directly impacts teachers and students.
- My credibility as an evaluator has pros and cons – I am a teacher with direct first-hand experience in this program as a classroom coach, volunteer and judge. I have first-hand knowledge of the communication structure and organization of this program in my community, and my years of experience working in Surrey also contribute to my understanding of district administration practices and structure. My proximity to the program also removes the possibility of viewing me as an unbiased third party, and my profession as a second language teacher implies my bias towards valuing second language learning.
- Information gathering has been selected to include those details that will inform participants of the program’s impact and will allow them to evaluate ways to improve it in future.
- The evaluation process I have laid out is feasible and frugal. A possible area of political controversy is in the questions about other languages spoken, as they imply cultural background and its connection to participation or achievement. These questions will be carefully worded so that bias and misinterpretation are eliminated to the greatest extent possible. While the focus is on the fact that participation in this program is based solely on knowledge of French, it is commonly acknowledged that knowing more than one language and being able to speak multiple languages fluently leads to a greater level of comfort in public speaking and second language use.
- The propriety of this evaluation is assured by the fact that survey participation is voluntary and anonymous, that interview participants will be able to review, edit, retract or revise statements before submission. It will be made clear that information gathered will be anonymized and shared with the school district and with the local chapter of CPF, for the purposes of assessing program impact and improving the program.
- The accuracy of the information gathered is assured through making questions as objective as possible and gathering a range of both quantitative and qualitative data. Quantitative data will largely be gathered by surveys, while qualitative data will be gathered through interviews. Both types of data will be incorporated in the final analysis of program impact and will be reported in a way that is impartial. Information will be gathered in a way that represents all impacted grade levels and all zones within the Surrey school district.
- Information will only be gathered from teachers, students and parents who are direct participants in this program in order to assess its impact.
- 2019/2020 FACT SHEET – Surrey Schools
- Canadian Parents for French
- Canadian Parents for French – BC and Yukon Chapter
- CPF Surrey Facebook page
- Data Collection Methods video: https://www.youtube.com/watch?v=NJ-gW6adQTc
- Data Collection and Analysis for Impact Evaluation Video: https://www.youtube.com/watch?v=HFGVJJMDo4I
- Evaluation Design Checklist: https://www.wmich.edu/sites/default/files/attachments/u350/2014/evaldesign.pdf
- Greene, Jennifer. “Approaches to Evaluation.” YouTube, Education at Illinois, 12 Apr. 2017, www.youtube.com/watch?v=zpwB_nSv6HM.
- Huey-Tsyh, C. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage PublicationsStandards for Program Evaluation: http://www.oecd.org/dev/pgd/38406354.pdf
- Key Evaluation Checklist: https://www.wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20 checklist.pdf
- Qualitative and Quantitative Data video: https://www.youtube.com/watch?v=dwFsRZv4oHA
- Surrey School District website