Garbage In, Garbage Out

By Jane Weinkrantz



Although I have fond memories of P.S. 156 in Laurelton, I have never been back to visit the Queens elementary school I left when my family moved to the suburbs. I took a look at my old elementary school today, but it was virtual; I checked P.S. 156 on to get a sense of how the New York City Teacher Data Reports, a sort of Zagat’s of teachers in which teachers were given a percentile rank based on actual student performance on statewide math and English assessments compared to state expectations adjusted for pupil past performance and demographics, works. According to the city’s evaluation system, 0% of P.S. 156’s math teachers are considered above average and some teachers are perceived to have a value-added score of  -.25. Anything that makes me wistful for can’t be good.

But then, I thought about the saying “Garbage in, garbage out.” The New York City ratings, which are sure to leave chaos in their wake, are flawed in so many ways that they would be funny, if only they weren’t tragic. The scores have a margin of error that can reach 50%, so even if you do believe that a teacher’s career can be reduced to a number, these are not the right numbers. How irresponsible is this formula? Imagine your child coming home to tell you that he received a 50 on a test, but it could actually have been anywhere between a 25 and a 75. Not a very informative assessment? Too bad.  Worse, imagine teaching a student in any discipline to formulate answers that could be up to 50% off. Parents and administrators would be right to correct a teacher who permitted such wide variability in accuracy. Yet, our colleagues in New York City have no choice but to accept an erroneous and demeaning system of professional assessment.

Whether a single number could ever measure teacher efficacy ---even with no margin for error---is doubtful.  When combined with out-of-date numbers and data from tests the city has admitted are flawed, the rankings become a pointless exercise in numbers-crunching and heart-wrenching.

Teachers of gifted students are particularly harmed by the data-driven rankings because their students only have so much room for improvement. According to The New York Times which chose to publish the rankings in spite of their inaccuracies but invited teachers to comment on their individual rankings, “In one extreme case, the formula assigned an eighth-grade math teacher at the prestigious Anderson School on the Upper West Side the lowest possible rating, a zero, even though her students posted test scores 1.22 standard deviations above the mean — normally good enough to rank in the 89th percentile. Her problem? The formula expected her high-achieving students to be 1.84 standard deviations higher than the average — roughly the 97th percentile.”

Consider what the results might be for teachers who were on maternity or medical leave, who switched grades or content areas or schools during the three year look -back period or who went from part-time to full-time.  Any variation in a teacher’s career would be sure to make terribly unreliable figures downright absurd. Rebecca Joseph, a teacher trainer, associate professor at California State University and writer for The Huffington Post looks back on her teaching career: “If you had looked at my test scores my first year, you would have not have seen how weak I was, because my sixth graders had another amazing reading teacher on the team whose skills almost counteracted my lack of teaching knowledge and experience. During my sixth year of teaching, when I was much better trained and competent, my test scores were lower, because my students' skills were far below grade level.” To paraphrase, numbers do lie.

            “’What people don’t understand is that they are just not accurate,’ Elizabeth Phillips, the principal of P.S. 321 told The New York Times. ‘We are talking about minute differences in test scores that cause a teacher to score in the lowest percentiles,’ like a teacher whom she finds great and who scored in the sixth percentile because her students went to a 3.92 average test score from a 3.97, out of a possible 4. And that is not to mention one teacher who had test scores listed for a year she was out on childcare leave, Ms. Phillips said.

“The only way this will have any kind of a positive impact,” she said, “would be if people see how ridiculous this is and it gives New York State pause about how they are going about teacher evaluation.” In other words, if good teachers have taught New Yorkers how to properly think critically, our state should conclude that the City’s system for evaluating educators is deeply flawed and demand change.

            Even Alison Epstein, a second grade gifted-and-talented teacher whose top rating was featured in The New York Times is quick to point out that test scores are an inadequate method of evaluating students and educators. She told reporters, “Unfortunately, the schools have become incredibly data-driven, which at times detracts from the overall curriculum…The pressure for teachers and children to perform for tests that do not really show how intelligent a student is, or how amazing a teacher might be is substantial.”

            Ms. Epstein’s words reminded me of my beloved trig teacher, Mr. Edmund Miles and a test that did not show how amazing he was. Mr. Miles was a gifted and enthusiastic teacher whose lessons were creative, engaging and clear. I understood every topic he taught---the day after the test that I invariably failed.   As he went over each test, the topic would finally click in my brain. It was too little and too late. I was failing trigonometry. I always did my homework and got extra help outside of school. Mr. Miles understood that I had other talents in English, social studies and foreign language, so he never made me feel bad about my lack of success in his class. In fact, Mr. Miles praised my achievement in my other classes, but privately he must have thought I would definitely fail trig. for the year. Calamity struck when Mr. Miles had a heart attack about six weeks before the Regents. Another teacher covered his classes, but it wasn’t the same. My chances of passing were decreasing every day. So committed to our class was Mr. Miles that he left the hospital against medical advice and returned for some last-ditch review sessions.  He was still obviously weak but determined.

The day of the Regents came and the test was harder than any sample test we’d taken in class. (We later heard it was the hardest in a decade.) I did my best and tried to concentrate, calculating, showing all work and then crossing out and erasing until my eraser had worn holes in my paper. “Never do the logarithm on the Regents,” Mr. Miles had cautioned us. With a shudder, I realized the logarithm was my only shot at partial credit. I knew how to set it up at least.  To make matters worse, the test was administered in a room adjacent to the school handball courts. Donna Summer’s “Bad Girls” blared from the boombox of a handballer outside----“Bad girls! Talking bout bad, bad girls! Beep! Beep!”--- making concentration even more difficult. Ultimately, I didn’t really finish my Regents paper so much as  I abandoned it. I went home assuming that I had failed and hoping that colleges might have some compassion for an English major who stunk at math.

            Well, I got a 67 on that Regents, a grade I am prouder of than many much higher grades I received in high school and college. Mr. Miles was proud, too. “Janie Bird! You passed! 67!” he shouted, smiling. I was particularly proud of the two points over 65; they proved that my grade wasn’t made only of mercy. Mr. Miles died several weeks later after giving his trig class of 1979 everything he had---literally.

            If Mr. Miles were alive and evaluated by New York City today, my very best efforts on that Regents would have brought down his rating. I was not learning disabled; I spoke English; I was not underprivileged; I was not a member of any minority group.  I was placed in many honors classes. The score the formula would have anticipated for me would undoubtedly have been higher than a 67. Were my math limitations Mr. Miles’ fault? Would his rating have suffered because he was out for a few weeks due to a heart attack? Would he have been held responsible for the Donna Summer distraction? Probably yes to all of that. Mr. Miles loved numbers for their beauty, logic and possibilities, but he also knew that they weren’t the only way to measure a person’s worth.  It is imperative that New York City ’s mayor and chancellor grasp that same philosophy and put an end to Teacher Data Reports.  

return to pct homepage