An Analysis of the Effects of an Academic Summer Program for Middle School Students, 2012 Archival Version


An Analysis of the Effects of an Academic Summer Program for Middle School Students, 2012, funded by the Edna McConnell Clark Foundation's Social Innovation Fund, was conducted in the Summer of 2012 in three school districts (District A, B, C) that were new partnerships for the Building Educated Leaders for Life program (BELL). Of the three study districts, District A and B offered the BELL program in one middle school each; District C offered the program in three schools. Several types of quantitative data were collected about students and BELL teachers in the study. The nature and purpose of these data sources are described below. (1) Spring (baseline) characteristics and test scores:During the application process, parents provided information about their child's socioeconomic characteristics (racial or ethnic group, parents' education, and so on. In addition, schools provided information about whether students in the study were eligible for free or reduced-price lunch, whether they had an individualized education plan (IEP), and whether English was their second language. Schools also provided students' scores on the spring 2012 math and reading assessments administered by their state; these scores were used to determine whether students were proficient, based on local cutoff scores on their state test. (2) Attendance records: BELL provided the evaluation team with the attendance records of students in the study during summer 2012, including any students in the non-BELL group who may have attended the program. (3) Fall reading and math tests: Students in the study were encouraged to take standardized tests in math and reading in fall 2012. In the average study district, students took the tests six weeks after the end of the program, or one week after the start of the school year. Students' reading achievement was assessed using the Group Reading Assessment and Diagnostic Examination (GRADE), and their math achievement was assessed with its math counterpart, the Group Mathematics Assessment and Diagnostic Examination (GMADE). The GRADE contains two subtests (reading comprehension and vocabulary), and the GMADE contains three (concepts, operations, and processes). (4) Fall student survey: In the same session as the GRADE and GMADE were administered, students also completed a short survey asking about the extent to which they were engaged in various aspects of instruction when they returned to school in the fall (for example, whether they paid attention in class and whether they completed their homework on time). The survey also included a set of items asking students to describe their activities during the summer. (5) BELL teacher survey: BELL administers a survey to its teachers as part of regular program monitoring and evaluation activities. The survey captures teachers' characteristics as well as their perceptions of various aspects of the BELL program (such as training, materials, and staffing), of their own performance in the classroom, and of their students' performance and engagement. BELL administers the survey to all of its teachers -- academic (English Language Arts [ELA] or math or both) and enrichment teachers -- as well as mentors (teaching assistants). However, given the academic focus of this evaluation, the target population for the study's purposes was academic teachers who taught students in the study. This collection contains two datasets. Part 1 is a student-level dataset that includes all information collected about students in the study sample (baseline data, GRADE and GMADE test scores, student survey data, and attendance data). Part 2 is a teacher survey dataset that includes teachers in the analysis sample (i.e. academic middle school teachers who responded to the survey and who taught students in the study). Demographic information about the students include socioeconomic characteristics like racial or ethnic group, parents' education, and so on. Demographic information about the teachers in the sample include their education and experience backgrounds. The purpose of this study was to evaluate the impact of BELL's summer program for middle school students who are performing below grade level. Also, the results can be useful for generating suggestive or preliminary evidence about the potential effects of a full day, academically oriented summer program model for middle school students. BELL aims to serve students who are struggling academically, and so eligibility for the study was limited to students in three study districts who were performing below grade level. The evaluation of BELL's middle school program uses a student-level random assignment research design to examine BELL's effects on student outcomes. Some of the districts that partner with BELL operate voluntary summer programs where there are more eligible students than BELL has the capacity to serve. In these oversubscribed voluntary programs, random assignment was used to determine which students would be invited to attend the BELL middle school program (the BELL group) and which students would participate in "business as usual" summer activities (the non-BELL group). In summer 2012, three of the ten districts that partnered with BELL to serve middle school students had oversubscribed voluntary programs and were willing to participate in the evaluation. Of the three study districts, District A and B offered the BELL program in one middle school each; District C offered the program in three schools. The schools in Districts A and C served only rising seventh- and eighth-grade students, whereas the middle school in District B served students in all three middle school grades. In order to make random assignment possible, a further requirement was that students had to be attending the program voluntarily (as opposed to being required to attend because they had not met some criteria on the state test). To be included in the study, students and their parents also had to complete the BELL application form and sign the informed consent form. In total, 1,032 rising sixth-, seventh-, or eighth-grade students applied to the middle school program in the three study districts and agreed to be part of the study. Of these 1,032 students, 385 students are from District A; 127 students are from District B; and the remaining half (520 students) are from District C. Random assignment was then used to determine which of these students would be invited to participate in the BELL middle school program (the BELL group) and which students would not be invited to participate in BELL (the non-BELL group). In order to ensure that each grade-level classroom in the BELL study sites would have 20 students, the research team conducted a separate random assignment lottery for each grade level, as well as for each school that students attended in the spring before the summer program. In total, 643 students (62 percent of study participants) were randomly assigned to the BELL group, while the remaining 389 students (38 percent of study participants) were placed in the non-BELL group. No weight variables are present in the data collection. ICPSR data undergo a confidentiality review and are altered when necessary to limit the risk of disclosure. ICPSR also routinely creates ready-to-go data files along with setups in the major statistical software formats as well as standard codebooks to accompany the data. In addition to these procedures, ICPSR performed the following processing steps for this data collection: Checked for undocumented or out-of-range codes.. Presence of Common Scales: GRADE test, GMADE test, and Skinner student engagement scale. Middle school students served by the BELL program in Summer 2012 in a subset of program sites. Smallest Geographic Unit: School district (anonymized) Convenience sample of schools (program sites) and students in the program (students who consented to study participation).

                        record abstracts, cognitive assessment test, on-site questionnaire, web-based surveyThe student and teacher data files can be merged using the BELLCLASSROOM variable, which is an identifier for the students' and teachers' classroom. More information can be found within the Description of Data Files Documentation.
                                    For additional information on "An Analysis of the Effects of an Academic Summer Program for Middle School Students, 2012", please visit the BELL Web site.
Related Identifier DOI: 10.3886/ICPSR36531.v1
Metadata Access
Creator Grossman, Jean;Somers, Marie-Andree
Publisher ICPSR - Interuniversity Consortium for Political and Social Research
Contributor Social Innovation Fund (Edna McConnell Clark Foundation & Corporation for National and Community Service)
Publication Year 2017
Rights Delivery;One or more files in this study are not available for download due to special restrictions; consult the study documentation to learn more on how to obtain the data.
Contact ICPSR - Interuniversity Consortium for Political and Social Research
Language English
Resource Type Dataset;administrative records data, event/transaction data, experimental data, survey data
Discipline Mathematics
Spatial Coverage {"United States","Time period: 2012","Collection date: 2012"}
Temporal Coverage {"United States","Time period: 2012","Collection date: 2012"}