The Dartmouth College study, published in Science Advances journal, challenges the effectiveness of software called COMPAS, for “Correctional Offender Management Profiling for Alternative Sanctions, which has been used to evaluate more than a million people since 1998.
Supposedly more accurate than humans, COMPAS uses 137 variables about a person to make its assessment.
To test this, researchers compared the program’s results to those of workers found online through the Amazon Mechanical Turk crowd-sourcing marketplace.
When results were pooled, humans were accurate 67% of the time, while COMPAS had 65.2% accuracy, which a statement on the study’s results noted are statistically the same.
The problem is not limited to COMPAS: “A separate review cited in the study found that eight of nine software programs failed to make accurate predictions,” it said.
“Claims that secretive and seemingly sophisticated data tools are more accurate and fair than humans are simply not supported by our research findings,” said Julia Dressel, one member of the student-faculty research team.