The Massachusetts Consortium for Innovative Education Assessment (MCIEA) recognized that standardized testing—often biased by narrow-minded questions and an overemphasis on scores—fails to accurately capture student learning outcomes, and is a poor predictor of future success. The partnership seeks to provide more accurate and useful data about school quality, close pernicious opportunity gaps, and prepare all students for college, career and life.
The consortium partners with public schools and teacher unions from Attleboro, Boston, Lowell, Milford, Revere, Somerville, Wareham and Winchester, MA. Through active MCIEA membership, these districts administer professional learning opportunities for teachers, school coaching, student and teacher climate surveys, community focus groups and performance assessment resources.
Providing practical solutions for learning opportunities
MCIEA engaged with VMware Tanzu Act, a program that partners with nonprofit organizations to leverage the resources of developers, designers and product managers from VMware Tanzu Labs. Tanzu Act identifies, designs, and delivers practical solutions and products to address critical challenges at a discounted rate. Tanzu Act collaborations help hone the skills of nonprofit personnel so they can maintain the technology well into the future. And the program contributes to the VMware 2030 agenda, a decade-long environmental, social, and governance (ESG) commitment to foster a more sustainable, equitable, and secure world.
Through the engagement, MCIEA built a dashboard that offers school district leaders, instructional leaders and stakeholders a fair, actionable view of school performance.
Measuring performance based only on test score poses challenge
There are many technology tools and programs teachers can integrate into their classrooms, and others for administrators to track how their districts are performing. Many districts then use data gathered from student test scores to better understand school quality. Schools identify patterns in student or teacher responses to survey topics of interest and then develop improvement action plans. For example, in response to concerning data regarding student academic stress, one district put together a student mental health committee composed of students, teachers and parents.
“It’s really been amazing to see the roles technology, software and data play in public education,” says Peter Piazza, project director, MCIEA. “Across public education in general, software is trying to solve problems that are major and very new in some cases regarding how students are learning material.”
However, there are challenges school districts face regarding using technology and collecting data. For over 20 years, states across the country have relied primarily on standardized test scores to assess the performance of public schools, and students and teachers in schools that receive low scores often suffer significant consequences. Schools that receive low scores suffer enrollment declines, which means per-pupil funding lost, and can be subject to state oversight or even closure. By relying on test scores that measure a narrow and limited range of skills, assessments capture only a small sliver of what students learn in class. MCIEA aims to change the accountability system by using other means to assess learning and school performance.
“The research is very clear about the relationship between demographics and test scores. Essentially, test score outcomes are picking up things that happen outside of the school, factors that are outside of the control of teachers and school administrators, which are highly correlated with test scores. There is also emergent research on the way test scores can be racially biased as well,” says Piazza.
Piazza provided an example from the Massachusetts Comprehensive Assessment System (MCAS) test. A reading comprehension question asked students to read a section of a novel about slavery and respond to it from the point of view of the slaveholder.
“Asking students, especially students of color, to embody a slaveholder who has oppressed their ancestors, is an awful way of measuring knowledge and harmful for these students. That question made it through rounds of vetting, yet people looked at this and decided it was a good way of measuring student reading comprehension,” Piazza says. This example emphasizes findings from MCIEA research that shows standardized tests are too narrow and often contain or promote racial bias. A significant number of students were unable to process, nor answer, this question. This altered participation led to a misrepresentation in test scores.
The Tanzu Act engagement was all about developing a user-friendly dashboard that can help visualize our data and make it actionable for our school partners. We collect a lot of data: annual student survey data, teacher survey data, and administrative data from each of the 200 schools of our consortium. Our online dashboard is a key resource we provide to our district and school partners.”Peter Piazza, Project Director, Massachusetts Consortium for Innovative Education Assessment
Tanzu Act helps to develop impactful products
In an effort to change the way schools are understood, MCIEA developed an evidence-based School Quality Measures (SQM) framework and dashboard that offers a fair, comprehensive overview of school performance. The SQM framework includes multiple measures of student engagement and achievement while emphasizing performance assessments as a meaningful way to understand student learning.
MCIEA sought input from stakeholders in all eight districts to build a framework reflecting what the public wants to know about its schools. Built around multiple measures, the framework comprises five major categories: teachers and leadership, school culture, resources, academic learning, community and well-being. This dashboard displays the framework for each school to show data the consortium has gathered.
“The Tanzu Act engagement was all about developing a user-friendly dashboard that can help visualize our data and make it actionable for our school partners,” says Piazza. “We collect a lot of data: annual student survey data, teacher survey data, and administrative data from each of the 200 schools of our consortium. Our online dashboard is a key resource we provide to our district and school partners.”
Tanzu Act engagements lead with empathy, turning problems into solutions for the organizations they serve. And the teams listen to organization feedback, using interviews to understand how to build successful solutions.
“We did user interviews on a regular basis to show how we incorporated feedback. What we ended up with is better and more usable, and the results from those conversations dramatically shaped every aspect of the dashboard,” says Piazza.
There is something long-lasting in the service provided by Tanzu Labs to our consortium, and there is also something long-lasting we can share with other districts so that they can get out of the test score rut. Now I have something I can show and click through to share.”Peter Piazza, Project Director, Massachusetts Consortium for Innovative Education Assessment
Looking ahead: The future of measuring school quality
Going forward, MCIEA plans to update and expand on the Tanzu Labs solution. “I hope to keep building the dashboard, making it better, and figure out how to operationalize it. It provides a really useful proof-ofconcept for districts that aren’t involved in our consortium,” Piazza says.
The solution’s success has put the spotlight on MCIEA. Other districts have reached out with questions about doing similar projects. Other districts want to improve the way they measure school quality, and the consortium’s Tanzu Act engagement offers an inspiring paradigm. Piazza says, “There is something long-lasting in the service provided by Tanzu Labs to our consortium, and there is also something long-lasting we can share with other districts so that they can get out of the test score rut. Now I have something I can show and click through to share.”