Study Finds Student Growth Percentile Is Unreliable, Limits Access to Charter Public Schools

Share on Facebook

Share on Twitter

Share on

LinkedIn

High degree of error, especially in small school districts, leads to small districts replacing larger ones in bottom 10 percent

BOSTON – The Student Growth Percentile (SGP) that the Massachusetts Department of Elementary and Secondary Education (DESE) now uses as 25 percent of the formula for determining school district rankings has a high degree of error. While the SGP may have a role to play as part of discussions around holding districts accountable for performance, it should not be used for high-stakes policy decisions, including which districts are eligible for an increase in the charter public school cap, according to a new study published by Pioneer Institute.

A 2010 state law requires DESE to rank school district performance. Districts in the bottom 10 percent statewide are subject to an increase in the amount of tuition that can be paid to charter schools from 9 percent to 18 percent of net school spending (NSS). Prior to 2015, rankings were based solely on the percentage of students scoring proficient on MCAS ELA, math and science tests.

“Using absolute proficiency as the only measure to determine eligibility for an increase in the charter school cap better aligns with the 2010 law,” said Jamie Gass, director of PioneerEducation. “The law’s intent was to provide students and families in struggling districts with different public school options.”

SGP compares changes in an individual’s MCAS score to those of students who scored similarly in previous years.

But SGP scores suffer from wide margins of error. Several recent studies report that a 95 percent confidence interval for the scores is a spread of 50 points.

“If one school received an SGP of 30 while another got a score of 70, researchers couldn’t be confident that the school with the higher score actually helped its students progress more than the one with the lower score,” said Dr. Cara Candal, who co-authored “An Analysis of How Massachusetts’ ‘Student Growth’ Model Limits Access to Charter Public Schools” with Dr. Bruce Bean and Rebekah Paxton.

When district enrollment is higher, SGP scores cluster around a “typical mean,” which may not reflect the diversity of real SGP scores within a district. In districts where enrollment is comparatively low, SGP is more volatile. As a result of the spurious relationship between SGP and district enrollment, some low-performing large districts where charter demand is high are exiting the bottom 10 percent and being replaced by higher performing small districts.

[ytp_video source=”5hLm2HUBppc”]

In the 2018-19 school year, Lynn moved out of the bottom 10 percent due to an increase in its SGP score, even though it remained solidly in the lowest 10 percent in terms of absolute proficiency. The change resulted in the charter cap falling from 18 percent to 9 percent of NSS, and meant none of the 1,920 unique students on charter waitlists would be able to gain access to charter schools, which generally outperform the district.

In 2018, one charter school application from Lynn “substantially met the criteria for approval,” according to then-Acting Commissioner of Elementary and Secondary Education Jeff Wulfson. But he was unable to recommend the school for approval by the Board of Elementary and Secondary Education (BESE) “because of upcoming changes in the NSS cap for Lynn.”

This increased volatility among districts entering and exiting the bottom 10 percent, and therefore BESE’s ability to authorize more charters, discourages operators from applying for a charter in high-demand areas that are close to exiting the lowest 10 percent.

Three quarters of the school districts entering the lowest 10 percent under SGP have enrollments below 2,500. They include districts like Hawlemont, Easthampton, Greenfield, Palmer, and Leicester that are among the smallest in Massachusetts. The result is an overall reduction in the number of available charter school seats.

About the Authors

Dr. Cara Stillings Candal is Director of Educational Opportunity at the Foundation for Excellence in Education and a senior fellow at Pioneer Institute. She was formerly research assistant professor and lecturer at the Boston University School of Education and a founding team member of the National Academy of Advanced Teacher Education. Candal holds a B.A. in English literature from Indiana University at Bloomington, an M.A. in social science from the University of Chicago, and a doctorate in education policy from Boston University. She is the author/editor of several books, including The Fight for the Best Charter Public Schools in the Nation.

Dr. Bruce Bean has been involved in various forms of data analysis for over forty years, creating custom software for many uses. Working as an accountant, he created a complete custom software general ledger package and other software programs. As the founder of his own economic statistics software company, he created custom software that contained over 14,000 economic and financial time series and was used nationwide. For the past twenty-one years, he has been the data manager of The Community Group in Lawrence, MA, where he has been involved in analyses of DESE and local school data both for in-house and consulting. He earned a Ph.D. in Physics from the University of New Hampshire.

Rebekah Paxton is a Research Assistant at Pioneer Institute. She first joined Pioneer in 2017 as a Roger Perry intern, writing about various transparency issues within the Commonwealth, including fiscal policy and higher education. Since then, she has worked on research projects under PioneerPublic and PioneerOpportunity, in areas of state finance, public policy, and labor relations. She recently earned an M.A. in Political Science and a B.A. in Political Science and Economics, from Boston University, where she graduated summa cum laude.

About Pioneer

Pioneer Institute is an independent, non-partisan, privately funded research organization that seeks to improve the quality of life in Massachusetts through civic discourse and intellectually rigorous, data-driven public policy solutions based on free market principles, individual liberty and responsibility, and the ideal of effective, limited and accountable government.