Title: Targeted school‐based interventions for improving reading and mathematics for students with, or at risk of, academic difficulties in Grades 7–12: A systematic review
Abstract: Campbell Systematic ReviewsVolume 16, Issue 2 e1081 SYSTEMATIC REVIEWOpen Access Targeted school-based interventions for improving reading and mathematics for students with, or at risk of, academic difficulties in Grades 7–12: A systematic review Jens Dietrichson, Corresponding Author Jens Dietrichson [email protected] VIVE—The Danish Center for Social Science Research, Copenhagen, Denmark Correspondence Jens Dietrichson, VIVE—The Danish Center for Social Science Research, Herluf Trollesgade 11, DK-1052 Copenhagen K, Denmark. Email: [email protected] for more papers by this authorTrine Filges, Trine Filges VIVE—The Danish Center for Social Science Research, Copenhagen, DenmarkSearch for more papers by this authorRasmus H. Klokker, Rasmus H. Klokker VIVE—The Danish Center for Social Science Research, Copenhagen, DenmarkSearch for more papers by this authorBjørn C. A. Viinholt, Bjørn C. A. Viinholt VIVE—The Danish Center for Social Science Research, Copenhagen, DenmarkSearch for more papers by this authorMartin Bøg, Martin Bøg Lundbeck A/S, Copenhagen, DenmarkSearch for more papers by this authorUlla H. Jensen, Ulla H. Jensen Professionshøjskolen Absalon, Roskilde, DenmarkSearch for more papers by this author Jens Dietrichson, Corresponding Author Jens Dietrichson [email protected] VIVE—The Danish Center for Social Science Research, Copenhagen, Denmark Correspondence Jens Dietrichson, VIVE—The Danish Center for Social Science Research, Herluf Trollesgade 11, DK-1052 Copenhagen K, Denmark. Email: [email protected] for more papers by this authorTrine Filges, Trine Filges VIVE—The Danish Center for Social Science Research, Copenhagen, DenmarkSearch for more papers by this authorRasmus H. Klokker, Rasmus H. Klokker VIVE—The Danish Center for Social Science Research, Copenhagen, DenmarkSearch for more papers by this authorBjørn C. A. Viinholt, Bjørn C. A. Viinholt VIVE—The Danish Center for Social Science Research, Copenhagen, DenmarkSearch for more papers by this authorMartin Bøg, Martin Bøg Lundbeck A/S, Copenhagen, DenmarkSearch for more papers by this authorUlla H. Jensen, Ulla H. Jensen Professionshøjskolen Absalon, Roskilde, DenmarkSearch for more papers by this author First published: 01 April 2020 https://doi.org/10.1002/cl2.1081Citations: 4 Linked Article: Protocol Plain language summary on the Campbell website AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat 1 PLAIN LANGUAGE SUMMARY 1.1 Targeted school-based interventions improve achievement in reading and maths for at-risk students in Grades 7–12 School-based interventions targeting students with, or at risk of, academic difficulties in Grades 7–12 have on average positive effects on standardised tests in reading and maths. The most effective interventions have the potential to considerably decrease the gap between at-risk and not-at-risk students. Effects vary substantially between interventions, however, and the evidence for using certain instructional methods or targeting certain domains is weaker. 1.2 What is this review about? Low levels of literacy and numeracy skills are associated with a range of negative outcomes later in life, such as reduced employment, earnings and health. This review examines the effects of a broad range of school-based interventions targeting students with, or at risk of, academic difficulties on standardised tests in reading and maths. Included interventions changed instructional methods by, for example, using peer-assisted learning, introducing financial incentives, giving instruction in small groups, providing more progress monitoring, using computer-assisted instruction (CAI) and giving teachers access to subject-specific coaching. Some interventions targeted specific domains in reading and maths, such as reading comprehension, fluency and algebra, while others focused on building for example meta-cognitive and social-emotional skills. This Campbell systematic review examines the effects of targeted school-based interventions on standardised tests in reading and maths. The review analyses evidence from 71 studies, 52 of which are randomised controlled trials. 1.3 What studies are included? Included studies examine targeted school-based interventions that tested effects on standardised tests in reading and maths for students in Grades 7–12 in regular schools. The students either have academic difficulties, or are deemed at risk of such difficulties on the basis of their background. The interventions are targeted as they aim to improve achievement for these groups of students, and not all students. The review summarises findings from 71 studies. Of these, 59 are from the United States, four from Canada, three from the UK, two from Germany, two from the Netherlands and one from Australia. Fifty-two studies are randomised controlled trials (RCTs) and 19 are quasiexperimental studies (QESs). 1.4 What are the main findings of this review? The interventions studied have on average positive and statistically significant short-run effects on standardised tests in reading and maths. This effect size is of an educationally meaningful magnitude, for example, in relation to the gap between groups of at-risk and not-at-risk students. This means that the most effective interventions have the potential of making a considerable dent in this gap. Only seven included studies tested effects more than three months after the end of intervention, and there is, therefore, little evidence of longer-run effects. Effects are very similar across reading domains. Interventions have larger effects on standardised tests in maths than on reading tests. Small group instruction has significantly larger effect sizes than CAI and incentive components. 1.5 What do the findings of this review mean? The review provides support for school-based interventions for students with, or at risk of, academic difficulties in Grades 7–12. However, the results do not provide a strong basis for prioritising between earlier and later interventions. For that, estimates of the long-run cost-effectiveness of interventions would be needed. The lack of long-run evidence should not be confused with a lack of effectiveness. We simply do not know whether the short-run effects are lasting. More research about long-run effects would therefore be a welcome addition to the literature. More research is also needed from non-English speaking countries; a large share of the included studies is from the United States, Canada, or the UK. There are also more interventions that have been tested by reading tests than maths tests, and interventions targeting maths seem like a promising research agenda. Many studies are not included in the meta-analysis due to low methodological quality. The most important improvement to research designs would be to increase the number of units and students in intervention and control groups. Lastly, the instruction given to control groups is often not described in detail. Variation in control group instruction is therefore difficult to analyse and a likely source of the effect size variation. 1.6 How up-to-date is this review? The review authors searched for studies up to July 2018. 2 EXECUTIVE SUMMARY/ABSTRACT 2.1 Background Low levels of numeracy and literacy skills are associated with a range of negative outcomes later in life, such as reduced earnings and health. Obtaining information about effective interventions for educationally disadvantaged youth is therefore important. 2.2 Objectives The main objective was to assess the effectiveness of interventions targeting students with or at risk of academic difficulties in Grades 7–12. 2.3 Search methods We searched electronic databases from 1980 to July 2018. We searched multiple international electronic databases (in total 14), seven national repositories, and performed a search of the grey literature using governmental sites, academic clearinghouses, and repositories for reports and working papers, and trial registries (10 sources). We hand searched recent volumes of six journals and contacted international experts. Lastly, we used included studies and 23 previously published reviews for citation tracking. 2.4 Selection criteria Studies had to meet the following criteria to be included: Population: The population eligible for the review included students attending regular schools in Grades 7–12, who were having academic difficulties, or were at risk of such difficulties. Intervention: We included interventions that sought to improve academic skills, were performed in schools during the regular school year, and were targeted (selected/indicated). Comparison: Included studies used a treatment-control group design or a comparison group design. We included RCTs, quasirandomised controlled trials (QRCTs) and QESs. Outcomes: Included studies used standardised tests in reading or mathematics. Setting: Studies carried out in regular schools in an OECD country were included. 2.5 Data collection and analysis Descriptive and numerical characteristics of included studies were coded by members of the review team. A review author independently checked coding. We used an extended version of the Cochrane Risk of Bias tool to assess risk of bias. We used random-effects meta-analysis and robust-variance estimation procedures to synthesise effect sizes. We conducted separate meta-analyses for tests performed within three months of the end of interventions (short-run effects) and longer follow-up periods. For short-run effects, we performed subgroup and moderator analyses focused on instructional methods and content domains. Sensitivity of the results to effect size measurement, outliers, clustered assignment of treatment, missing values, risk of bias and publication bias was assessed. 2.6 Results We found 24,411 potentially relevant records and screened 4,244 in full text. In total 247 studies met our inclusion criteria and we included 71 studies in meta-analyses. The reasons for not including studies in the meta-analyses were that they had too high risk of bias (118), compared two alternative interventions (38 studies), lacked necessary information (13 studies), or used overlapping samples (7 studies). Of the 71 studies, 99 interventions, and 214 effect sizes included in the meta-analysis, 76% were RCTs, and the rest QESs. The total number of student observations in the analysed studies was around 105,700. The target group consisted of, on average, 47% girls, 73% minority students, and 62% low income students. The mean grade was 8.3. Most studies included in the meta-analysis had a moderate to high risk of bias. The average effect size for short-run outcomes was positive and statistically significant (weighted average effect size [ES] = 0.22, 95% confidence interval [CI] = [0.148, 0.284]). The effect size corresponds to a 56% chance that a randomly selected score of a student who received the intervention is greater than the score of a randomly selected student who did not. All measures indicated substantial heterogeneity across effect sizes. Seven studies included follow-up outcomes. The average effect size was small and not statistically significant (ES = 0.05, 95% CI = [−0.096, 0.192]), but there was substantial variation. We focused the analysis of comparative effectiveness on the short-run outcomes and two types of intervention components: instructional methods and content domains. Interventions that included small group instruction (ES = 0.38, 95% CI = [0.211, 0.547]), peer-assisted instruction (ES = 0.19, 95% CI = [0.061, 0.319]), progress monitoring (ES = 0.19, 95% CI = [0.086, 0.290]), CAI (ES = 0.17, 95% CI = [0.043, 0.309]) and coaching of personnel (ES = 0.10, 95% CI = [0.038, 0.166]) had positive and significant average effect sizes. Interventions that provided incentives for students did not have a significant average effect size (ES = 0.05, 95% CI = [−0.103, 0.194]). The average effect size of interventions that included none of the above components, but for example provided extra instructional time, instruction in groups smaller than whole class but larger than 5 students, or just changed the content had a relatively large, but statistically insignificant effect size (ES = 0.20, 95% CI = [−0.002, 0.394]). The differences between effect sizes from interventions targeting different content domains were mostly small. Interventions targeting fluency, vocabulary, multiple reading areas, meta-cognitive, social-emotional, or general academic skills, comprehension, spelling and writing, and decoding had average effect sizes ranging from 0.14 to 0.22, all of them statistically significant. Effect sizes based on mathematics tests had a relatively large effect size (ES = 0.34, CI = [0.169, 0.502]). Including all instructional methods and moderators without missing observations in meta-regressions revealed that effect sizes based on mathematics tests were significantly larger than effect sizes based on reading tests, and QES showed significantly larger effect sizes than RCTs. Small group instruction was associated with significantly larger effect sizes than CAI and incentive components. The unexplained heterogeneity remained substantial throughout the comparative effectiveness analysis. 2.7 Authors' conclusions We found evidence of positive and statistically significant average effects of educationally meaningful magnitudes (and no significant adverse effects). The most effective interventions in our sample have the potential of making a considerable dent in the achievement gap between at-risk and not-at-risk students. The results thus provide support for implementing school-based interventions for students with or at risk of academic difficulties in Grades 7–12. We want to stress that our results do not provide a strong basis for prioritising between earlier and later interventions. For that, we would need estimates of the long-run cost-effectiveness of interventions and evidence is lacking in this regard. Furthermore, there was substantial heterogeneity throughout the analyses that we were unable to explain by observable intervention characteristics. 3 BACKGROUND 3.1 The issue Across countries, a large proportion of students leave secondary school without the skills and qualifications needed to succeed in the labour market. In the member countries of the Organisation for Economic Co-operation and Development (OECD), 16% of all youth between 25 and 34 years of age have not earned the equivalent of an upper secondary education or high school degree (OECD, 2016a). According to the results from the Programme for International Student Achievement (PISA), on average around 20–25% of the participants are not proficient in reading and mathematics as 15 year olds (OECD, 2016b, 2019). Whilst the proportion of students that are not proficient in reading and mathematics is lower in some countries, it remains around 10% even in the best performing countries (OECD, 2016b, 2019). Thus, the share of students with academic difficulties is substantial in all OECD countries. Entering adulthood with a low level of educational attainment is not only associated with reduced employment and financial prospects (De Ridder et al. 2012; Johnson, Brett, & Deary, 2010; Scott & Bernhardt, 2000), it is also associated with numerous health problems and risk behaviours, such as drug use and crime, which have serious implications for the individual as well as for society (Berridge, Brodie, Pitts, Porteous, & Tarling, 2001; Brook, Stimmel, Zhang, & Brook, 2008; Feinstein, Sabates, Anderson, Sorhaindo, & Hammond, 2006; Horwood et al., 2010; Sabates, Feinstein, & Shingal, 2013). Improving the educational attainment and achievement for students with academic difficulties is therefore important. The group of students who experiences academic difficulties is diverse. It includes for instance students with learning disabilities, students who are struggling because they lack family support, because they have emotional or behavioural problems, or because they are learning the first language of the country they are living in. Some groups of students may not currently have academic difficulties but are "at risk" in the sense that they are in danger of ending up with difficulties in the future, at least in the absence of intervention (McWhirter, McWhirter, McWhirter, & McWhirter, 2004). Although being at risk points to a future negative situation, "at risk" is sometimes used to designate a current situation (McWhirter et al., 2004; Tidwell & Corona Garret, 1994), as current academic difficulties are a risk factor for future difficulties and having difficulties in one area may be a risk factor in other areas (McWhirter, McWhirter, McWhirter, & McWhirter, 1994). Separating students with and at risk of academic difficulties is therefore sometimes difficult. Test score achievement gaps are typically present well before secondary school (e.g., Heckman 2006; Lipsey et al., 2012; von Hippel, Workman, & Downey, 2018), and there are often large differences in risk factors for academic difficulties before children start primary school. For example, the gap between majority and minority ethnic children on cognitive skills tests is apparent when children are as young as 3–4 years old (e.g., Burchinal et al., 2011; Fryer & Levitt, 2013). Low-income preschool children can have more behaviour problems (e.g., Huaqing & Kaiser, 2003) and there is a strong continuity between emotional and behavioural problems in preschool and psychopathology in later childhood (Link Egger & Angold, 2006). Emotional and behavioural problems are in turn linked to lower academic achievement in school (e.g., Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Taylor, Oberle, Durlak, & Weissberg, 2017). Struggling readers tend to be persistently behind their peers from the early grades (e.g., Elbro & Petersen, 2004; Francis, Shaywitz, Stuebing, Shaywitz, & Fletcher, 1996) and early math and language abilities strongly predict later academic achievement (e.g., Duncan et al., 2007; Golinkoff, Hoff, Rowe, Tamis-Lemonda, & Hirsh-Pasek, 2018). The prenatal and early childhood environment appears to be an important factor that keeps students from realising their academic potential (e.g., Almond, Currie, & Duque, 2018).11 Hereditary factors do not seem like a major explanation for the achievement gap between at-risk and not-at risk groups, see e.g., Hackman and Farah (2009), Nisbett et al. (2012), and Tucker-Drob et al. (2013) for discussions. Currie (2009) furthermore documented that children from families with low socioeconomic status (SES) have worse health, including measures of foetal conditions, physical health at birth, incidence of chronic conditions and mental health problems. Immigrant and minority children are often overrepresented among low SES families and face similar risks (e.g., Bradley & Corwyn, 2002; Deater–Deckard, Dodge, Bates and Pettit, 1998; Morgan, Farkas, Hillemeier, & Maczuga, 2012). Family environments also differ in aspects thought to affect educational achievement. Low SES families are less likely to provide a rich language and literacy environment (Bus, Van IJzendoorn, & Pellegrini, 1995; Golinkoff et al., 2018; Hart & Risley, 2003). The parenting practices and access to resources such as early childhood education, health care, nutrition, and enriching spare-time activities also differ between high and low risk groups (e.g., Esping-Andersson et al., 2012; Morgan et al., 2012). Low SES parents also seem to have lower academic expectations for their children (Bradley & Corwyn, 2002; Slates, Alexander, Entwisle, & Olson, 2012), and teachers have lower expectations for low SES and minority students (e.g., Good, Aronson, & Inzlicht, 2003; Timperley & Phillips, 2003). Furthermore, low SES children are more likely to experience a decline in motivation during the course of primary, secondary, and upper secondary school (Archambault, Eccles, & Vida, 2010). The neighbourhoods students grow up in is another potential determinant of achievement (e.g., Campbell, Shaw, & Gilliom, 2000; Chetty, Friedman, Hendren, Jones, & Porter, 2018; Chetty, Hendren, & Katz, 2016). It seems likely that many students in high risk groups live in neighbourhoods that are less supportive of high educational achievement in terms of, for example, peer support and role models. To get by in a disadvantaged neighbourhood may also require a very different set of skills compared to what is needed to thrive in school, something which may increase the risk that pupils have trouble decoding the "correct" behaviour in educational environments (e.g., Heller et al., 2017). Regarding the relative importance of families and neighbourhoods, the review in Björklund & Salvanes (2011) indicates that family resources are the more important explanatory factor. After this review of risk factors for academic difficulties, it is worth noting that the life circumstances placing children and youth at risk are only partially predictive. That is, risk factors increase the probability of a having academic difficulties, but are not deterministic. As academic difficulties therefore cannot be perfectly predicted and may show up relatively late in a child's life, early interventions may not be enough and effective interventions in all grades may be needed to reduce the achievement gaps substantially. As the test score gaps between high and low risk groups remain relatively stable from the early grades, schools do not seem to be a major reason for the inequality in academic achievement (e.g., Heckman 2006; Lipsey et al., 2012; von Hippel et al. 2018). Further evidence is provided by the seasonality in achievement gaps. In the United States, the gap between high and low SES students tends to widen during summer breaks when schools are out of session (e.g., Alexander, Entwisle, & Olson, 2001; Gershenson, 2013; Kim & Quinn, 2013; although von Hippel et al., 2018, show that this pattern is not universal across risk groups, grades and cohorts). However, the stability of the test score gaps also implies that current school practice is not, in general, enough to decrease the achievement gaps. As schools are perhaps the societal arena where most children and youth can be reached, finding effective school-based interventions for students with or at risk of academic difficulties is a question of major importance. 3.2 The intervention This review focusses on interventions that are targeted at students with or at risk of academic difficulties and that aim to improve students' academic achievement. In line with the diversity of reasons for ending up with a low level of skills and educational attainment, we included interventions targeting students who for a broad range of reasons were having academic difficulties, or were at risk of such difficulties. We prioritised already having difficulties over belonging to an at risk group in the sense that if there was information about for example test scores, grade point averages, or low attendance, we did not require information about at risk status. Furthermore, we did not include interventions targeting high-performing students in groups that may otherwise be at risk. Interventions aimed at improving academic achievement are numerous and very diverse in terms of intervention focus, target group, and mode of delivery. This review focused on targeted interventions performed in schools and provided to students with or at risk of academic difficulties in Grades 7–12 (ages range from 12–14 to 17–19, depending on country/state), where academic skill building and learning were primary intervention aims. Many targeted interventions are delivered individually as a supplement to regular classes and school activities. However, targeted interventions can be delivered in various settings, including in class (e.g., paired reading interventions or the Xtreme Reading programme) or in group sessions (e.g., the READ 180 programme). This review restricts the settings to school-based interventions, by which we mean interventions delivered in school, during the regular school year, and where schools are a key stakeholder. This restriction excludes for example after-school programmes, summer camps and summer reading programmes, and interventions involving only parents and families (see, e.g., Zief, Lauver, & Maynard, 2006 for a review of after school-programmes, Kim & Quinn, 2013, for a review of summer reading programmes, and Jeynes, 2012, for a review of programmes that involve families or parents). We include a wide range of interventions that aim to improve the academic achievement of students by either changing the method of instruction—such as tutoring, peer-assisted learning or CAI interventions—or by changing the content of the instruction—for instance, interventions emphasising mathematical problem solving skills, reading comprehension or meta-cognitive and social-emotional skills. Many interventions involve changes to both teaching method and content of instruction, and very often consist of several major programmatic components. Thus, interventions were included in this review based on their aim to improve academic achievement of students with or at risk of academic under achievement and not on the type of components (or mechanisms) used in the intervention. For this reason, the review excludes interventions that may improve academic achievement as a side-effect, but do not state academic achievement as an explicit aim. For example, interventions where improvement in behavioural or social-emotional outcomes are the primary aim of the intervention, like Classroom Management or the SCARE programme, are not included. However, interventions with behavioural and social-emotional components may very well have academic achievement as one of their primary aims, and use standardised tests of reading and mathematics as one of their primary outcomes. If this is the case, and achievement is a primary outcome, such interventions are included. Thus, the content of the programme is less important than the primary outcome (academic achievement) and the target population (students with or at risk of academic difficulties). Universal interventions which aim to improve the quality of the common learning environment at school in order to raise academic performance of all students (including average and above average students) are excluded. Whole-school reform strategy concepts such as Success for All, curriculum-based programmes like Elements of Mathematics (EMP), as well as reduced class size interventions and general professional development interventions for principals and teachers that do not target at-risk students were also excluded. However, we do include interventions with a professional development component, for example, in the form of coaching of teachers during the implementation, as long as the intervention specifically targeted students with or at risk of academic difficulties. 3.3 How the intervention might work Given the spectrum of interventions that are included in this review, it is unsurprising that they represent a range of diverse strategies to achieve improvement in academic outcomes. This diversity reflects the varying reasons that might explain why students are struggling, or are at risk. In turn, the theoretical background for the interventions also vary. It is therefore not possible to specify one particular theory of change or one theoretical framework for this review. Instead, we briefly review three theoretical perspectives that characterise the majority of the included interventions. We also discuss and exemplify how targeted interventions may address some of the reasons for academic difficulties in light of the theoretical perspectives. 3.3.1 Theoretical perspectives The reasons why students may be struggling are multifaceted and the theoretical perspectives underlying interventions are therefore likely to be broad. Nevertheless, three superordinate components are characteristic for the majority of the included interventions. These components can be abridged to: Adaptation of behaviour (social learning theory). Individual cognitive learning (cognitive developmental theory). Alteration of the social learning environment (pedagogical theory). We emphasise that the following presentation of theoretical perspectives is not all-encompassing and although components are presented as demarcated, they contain some conceptual overlap. Social learning theory has its origins in social and personality psychology and was initially developed by psychologist Julian Rotter and further developed especially by Bandura (1977; 1986). From the perspective of social learning theory, behaviour and skills are primarily learned by observing and imitating the actions of others, and behaviour is in turn regulated by the recognition of those actions by others (reinforcement) or discouraged by a lack of recognition or sanctions (punishment). According to social learning theory, creating the right social context for the student can therefore stimulate more productive behaviour through social modelling and reinforcement of certain behaviours that can lead to higher achievement. Cognitive developmental theory is not one particular theory, but rather a myriad of theories about human development that focus on how cognitive functions such as language