Vermont educator helps create software to parse testing data

The federal No Child Left Behind Act, which required the nation’s schools to make “adequate yearly progress” by meeting proficiency standards on statewide tests, has been distressing Vermont schools for more than a decade.

As of last year, the 2001 law expected every public school across the country to reach 100 percent proficiency, but nearly every school in Vermont failed to reach the goal, leaving 290 schools with a black mark. Like more than half the states, Vermont has a waiver – a two-year reprieve from using the test scores for accountability purposes while it transitions to the Smarter Balance Assessment Consortium (SBAC) test.

A recently released Web-based tool called Polarys promises to help educators look at the information gleaned from the tests in a new way, with a “much better performance indicator” that levels the playing field and allows administrators to use data to improve their school, according to its creators at ReContext Data Solutions.

“By controlling for factors that affect state exam results but are outside the school’s control, we are able to demonstrate the changes in outcome at the district, school and classroom level,” according to the ReContext website.

The point of testing students annually is to make sure that the most vulnerable students are not falling behind. The idea was to provide educators with the kind of classroom level information that would help them help their students. Instead, a more cynical culture emerged, according to Polarys architect David Adler.

“Part of the problem with standardized exams is that educators believe the data is useless and rigged,” said Adler, who is also the curriculum director for the Orleans Southwest Supervisory Union.

The Annual Yearly Progress indicators are based on high stakes tests that can be misleading, Adler said. Schools that score well on the exams can miss opportunities to improve and a “bad” school may not have too far to go to raise performance criteria, Adler said.

Teachers, principals and administrators have access to a lot of data these days. Adler designed Polarys to turn it into actionable information using much more than just test scores. While improved student achievement is the ultimate goal there are many variables that can impact the outcomes.

“Factors like programming, school structures, spending, student demographics and teacher characteristics all impact outcomes. Figuring out what changes to make, if any, can seem like a Herculean feat,” according to Adler.

For example, the state average for proficiency might be 87 percent while a school is scoring at 60 percent, says Adler. It might look like the school needs to make up 27 percentage points, but when applying Polarys algorithms it turns out the gap between where the school is and where it should be is really only about 8 percent. At this point, an administrator can compare performance with similar schools and find out what they are doing to close the gap.

Put another way, Polarys works by normalizing NECAP results for factors that are beyond the school’s control, such as poverty or special education.

“We can norm the data and come up with a number that represents school performance,” said Mary Westervelt, director of operations for ReContext.

The free program also creates a snapshot by grouping schools with others that have similar characteristics. This feature allows administrators to compare performance in an “apples to apples way” and collaborate on solutions.

Richard Boltax, the school improvement coordinator at the Agency of Education, says Polarys is a “practical and useful tool.”

Boltax’s job is to help schools develop improvement plans and he finds that the principals and superintendents that he works with like Polarys because it is simple to use, automatically generates graphs and presents information in a way that is understandable.

All of the same raw data is available on the AOE’s website. “You have to know how to work it and comb through the data,” said Boltax. Adler’s program helps sort it and package it in a straightforward manner.

Adler has been designing the tool for the better part of a decade with the most significant adjustments coming in the past three years. “It started out as an Excel spreadsheet,” he said. He used publicly available information on school performance, demographic and structural data from the state. Polarys will soon integrate the SBAC results into the matrix.

About 50 school leaders in Vermont have been working with the Excel-based product since 2013. But this summer, Polarys was launched as a Web-based tool and is being offered in partnership with the Vermont Rural Education Collaborative free to Vermont educators for the next 18 months through an anonymous grant.

“Small schools provide many personalized learning opportunities for students, it will be helpful to have data that shows the impact of these opportunities on students success,” said Margaret Maclean, president of the board of directors for VREC and director of the Vermont Rural Partnership. The VREC feels it is particularly important for small rural schools to have tools like Polarys that sift through a wide range of data, Maclean said.

“What I really like about the program,” said Boltax, “is how it immediately pulls up the demographically similar schools so you can compare from school-to-school. It is a practical quick tool to paint a good picture based on accountability.”

Michael Hock, director of assessment at the Agency of Education said that American Institutes for Research, Vermont’s test contractor, provides tools and trainings on test interpretation and data usage. The agency welcomes anything that helps schools make better use of their data.

Vermont is a very collaborative environment, Adler said before adding that the purpose of Polarys is to provide meaningful analysis, meaningful feedback and spark meaningful conversations that will lead to our goal of educating students well.

Tiffany Danitz Pache

Comment Policy

VTDigger.org requires that all commenters identify themselves by their authentic first and last names. Initials, pseudonyms or screen names are not permissible.

No personal harrassment, abuse, or hate speech is permitted. Be succinct and to the point. Comments should be 1000 characters or fewer. If your comment is over 500 words, consider sending a commentary instead.

We personally review and moderate every comment that is posted here. This takes a lot of time; please consider donating to keep the conversation productive and informative.

The purpose of this policy is to encourage a civil discourse among readers who are willing to stand behind their identities and their comments. VTDigger has created a safe zone for readers who wish to engage in a thoughtful discussion on a range of subjects. We hope you join the conversation. If you have questions or concerns about our commenting platform, please review our Commenting FAQ.

Privacy policy
  • Max D. Meridi

    …”Polarys works by normalizing NECAP results for factors that are beyond the school’s control, such as poverty or special education.”…

    Normalizing?

    Is this actually saying that special ed kids somehow really affect a district’s aggregate NECAP scores?

  • Max D. Meridi

    If divining “data intelligence” is all that this tool ends up being used for, that’s fine.

    However, in my experience, politicians have a tendency to use tools like this to massage and politicize statistics (which I’m sure is not your tool’s intended function.)

    Political misrepresentations (intentional or not) of the statistical relevance of NAEP scores come to mind.

  • Rita Pitkin

    I get the idea behind this, but, geez, can we get rid of the words “normalize”and “normal”?

  • Jay Diaz

    While I’m no fan of testing, if we allow schools with achievement gaps between privileged and commonly marginalized student groups to claim that their lack of proficiency should be largely blamed on societal problems, we will be letting our education system off the hook too easily. For civic, economic, and social reasons, schools have legal and ethical obligations to provide equalized educational services to students with disabilities, students who speak English as a second language, and students in other commonly marginalized groups. Today, there are myriad educational programs, used by some schools in Vermont, that counter the impact of societal factors to narrow achievement gaps, improve proficiency, and improve school climate, for ALL students. School can be the great equalizer, but only if we make it so. To dismantle our educational disparities, we should use the data we have to determine where evidence-based programs are necessary, then fund and implement them. In short, let’s treat the problem, not use data “normalization” to ignore it. Our state’s future depends on it.

    • Hi Jay,

      I couldn’t agree more. We cannot let schools use the demographics of the students as an excuse to justify poor programming. The problem is that we’re misdiagnosing schools with both false positives and false negatives. Many schools are being praised for high achievement, when it’s really due to the demographics of the students and not the great job of the school system. Don’t those kids deserve more? We’re also using false metrics to tell schools that are improving outcomes for kids that they’re doing a poor job, when they’re actually moving their students in the right direction.

      I fundamentally believe that all students can learn and perform well on any academic indicator (standardized test or otherwise). It is true, however, that if we are to evaluate schools as a social practice, then we must do so with metrics that actually measure the impact of the school, and not ones that measure the demographics of the students. This is not excuse-making or lowering the bar. It’s giving institutions real, actionable feedback, so they can see where and how they can improve what they are offering for kids.

      Fifteen years of No Child Left Behind has served to expand the achievement gaps between the rich and the poor. A part of this problem is the way in which we’re evaluating our schools for effectiveness and how we’re trying to improve education with that information. Polarys is not a cure-all, but a step towards accountability with real metrics and information designed to help schools improve programming for kids in meaningful ways.

Thanks for reporting an error with the story, "Vermont educator helps create software to parse testing data"