Special Report
Education Funding

Peer Reviewers Winnow Race to Top Hopefuls

By Michele McNeil — February 26, 2010 7 min read
  • Save to favorites
  • Print

In the competition for $4 billion in Race to the Top grants, states have made their best pitches, a secret jury has debated and scored their applications—and now U.S. Secretary of Education Arne Duncan must decide who’s good enough to make it to the final round.

Mr. Duncan’s announcement of the Race to the Top Fund finalists, which is expected as early as this week, caps a dash by the Department of Education to recruit, vet, and train peer reviewers who wield tremendous power in determining who will win this high-stakes education reform competition.

In this first round of competition, all but 10 states applied for awards financed by the economic-stimulus package that will range from an estimated $350 million to $700 million—badly needed money that would help cash-strapped states.

Once the first-round awards are made in April, the department will publicize the scores and comments for each applicant from the 49 peer reviewers who have been given responsibility for judging 30 criteria that measure a state’s commitment to improving teacher quality, low-performing schools, data systems, and academic standards.

Despite all that influence, the ultimate power rests with Mr. Duncan, who will have the final say in awarding those cash prizes. But if the secretary deviates from the highest scores in awarding the Race to the Top grants, department officials have said he pledges to justify his reasons.

Inside the Peer Review Process

Each of 49 peer reviewers evaluated four or five state applications, which were randomly assigned. Comments and scores were due February 8, 2010.

BRIC ARCHIVE

IMAGES: iStockphoto.com/auras
SOURCES: U.S. Department of Education; Education Week

Mr. Duncan also has sole discretion in choosing how many finalists will come to Washington the week of March 15 to make presentations to the peer reviewers.

The purpose of the presentations, said Joanne Weiss, the department’s Race to the Top director, is not only to verify and clarify what’s in the applications, but also to ensure that those who must implement the plan know and understand what’s been promised.

“The real issue is, we’re giving away hundreds of millions of dollars, and we don’t want to just rely on what you wrote on a piece of paper,” Ms. Weiss said. “We want to look you in the eye.”

Looking for Conflicts

In August, the Education Department put out a call for peer reviewers to judge the applications and drew 1,500 applicants. They will pull from this pool for the second round of the competition, which starts in June when applications are due and ends in September, when the final dollar is awarded.

Vetting the list for potential conflicts of interest was no easy task. Any nominee who works for a state or district was automatically eliminated. So was anyone who helped out in any way on a state’s application, or who has a financial interest in the outcome of the competition.

Lawyers and ethics officers from the department did most of that work, although four career and four political staff members also reviewed the applicants to make sure they had expertise in the four school improvement areas emphasized in Race to the Top.

The department also hired a contractor to do a public-records search—even a Google search—into nominees to uncover information not previously disclosed. The department was also on the lookout for anyone who might have “significant identification with a specific pedagogical or philosophical viewpoint that might give an unfair advantage or disadvantage to an applicant,” according to department documents.

“It’s wasn’t like, ‘Do you have a belief?’ ” Ms. Weiss said. “It was more about, ‘Do you have an agenda?’ ”

Of the 49 peer reviewers, the department granted 12 waivers for “indirect” conflicts of interest, most commonly that a reviewer had worked for a state or school district within the past 12 months.

The reviewers receive $5,000 for what amounts to hundreds of hours of work that will span at least two months and involve tedious reading of applications that can exceed 1,000 pages each, counting appendices.

Though the department released basic demographic information on the reviewers—most, for example, have doctoral degrees and at least a dozen served on a state board of education or as a local or state school superintendent—federal officials have caught criticism for keeping their identities secret.

One reason the reviewers’ names should be made public is because of Mr. Duncan’s pledge to run a transparent competition, said Frederick M. Hess, the director of education policy studies for the American Enterprise Institute and a blogger hosted by Education Week. “We’re talking about an extraordinary sum of cash in play, and that’s why the identities of the judges are so important.”

In fact, the peer reviewers had to sign confidentiality agreements declaring they won’t disclose their identities or talk to the press—in effect, a gag order that applies until the department lifts it. Department officials have said they will make the identities public once the winners are announced, although it does not intend to reveal which reviewers scored which application.

Process So Far

In the course of the judging process, each peer reviewer received four or five applications to score; they were not allowed to judge applications from the states where they live. Then, the reviewers came together in Washington to form five-member panels to discuss and debate each application.

Since the reviewers rotate among panels, forming a unique one for each state, the department hopes to smooth out any scoring variances that might occur with different groups reviewing different applications.

Peer reviewers could refine their scores and comments based on their panel discussions, and the five scores were averaged to arrive at a final score. Ms. Weiss said any outliers whose grades were particularly low or high compared with the other panelists were not pressured to come to consensus.

Throughout the process, peer reviewers were instructed to do no independent research and to rely entirely on what’s written in the application.

Checks and balances were built into the system to fact-check the states’ applications. Ms. Weiss said, for example, that making the applications public, including the scores, comments, and video from the finalists’ presentations in Washington, will help ensure accountability.

Although the department has established a detailed, 500-point scoring guide, peer reviewers’ judgment will still ultimately come into play. For example, a peer reviewer will have to decide how to weigh an overly ambitious application, with a price tag that far exceeds the department’s nonbinding award estimates, against a less ambitious plan that stays within those estimates. “It’s entirely the peer reviewers’ judgment,” Ms. Weiss said. “You will see the reviewers made comments such as, ‘This is underfunded. Or this is overfunded.’ ”

Wild Card

Such judgment is the wild card in the competition, said Michael J. Petrilli, a vice president of the Thomas B. Fordham Institute, a think tank in Washington.

“What this comes down to is people. And invariably there are going to be judgment calls being made even as prescriptive as the criteria are,” said Mr. Petrilli, who worked with some peer-reviewed grant programs while at the department under President George W. Bush. “That’s when policy leanings are going to come into play.”

Still, the peer-review system was probably the best choice for the department in judging Race to the Top, said Mark Schneider, the vice president for education, human development, and the workforce program at the American Institutes for Research.

“Peer review is probably the best system we’ve come up with for judging products on merit,” said Mr. Schneider, a former commissioner of the National Center for Education Statistics. “If it’s done right, many people, especially losers, are unhappy but it does have a justification and a long history of support.”

There’s been less clarity to how Secretary Duncan and his staff will decide how many finalists will be selected—and how many awards will be given out.

The department has said that the applications will be ranked in order by score, with the top scorers getting money, until the money runs out. And Mr. Duncan has promised to be tough in setting a high bar for states to win.

Ms. Weiss said she couldn’t talk specifically about the process the department would use in determining the number of finalists or awards. But factors will include whether there are “natural breaks” that separate the superior applications from the not-so-good—and how much money is involved in awarding the grants.

One example: If a few big states with large grants are top scorers, there might be fewer finalists and fewer winners in the first round because the department has promised to leave a sizable pot of money for Phase 2.

And what will the losers get?

The same feedback as the winners, an average of seven to 10 pages of comments from each reviewer, which will serve as a roadmap for Phase 2.

Coverage of the American Recovery and Reinvestment Act is supported in part by grants from the William and Flora Hewlett Foundation, at www.hewlett.org and the Charles Stewart Mott Foundation, at www.mott.org.
A version of this article appeared in the March 03, 2010 edition of Education Week as Reviewers Winnow Race to Top Hopefuls

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Funding Biden's Budget Proposes Smaller Bump to Education Spending
The president requested increases to Title I and IDEA, and funding to expand preschool access in his 2025 budget proposal.
7 min read
President Joe Biden delivers remarks on lowering prices for American families during an event at the YMCA Allard Center on March 11, 2024, in Goffstown, N.H.
President Joe Biden delivers remarks on lowering prices for American families during an event at the YMCA Allard Center on March 11, 2024, in Goffstown, N.H. Biden's administration released its 2025 budget proposal, which includes a modest spending increase for the Education Department.
Evan Vucci/AP
Education Funding States Are Pulling Back on K-12 Spending. How Hard Will Schools Get Hit?
Some states are trimming education investments as financial forecasts suggest boom times may be over.
6 min read
Collage illustration of California state house and U.S. currency background.
F. Sheehan for Education Week / Getty
Education Funding Using AI to Guide School Funding: 4 Takeaways
One state is using AI to help guide school funding decisions. Will others follow?
5 min read
 Illustration of a robot hand drawing a graph line leading to budget and finalcial spending.
iStock/Getty
Education Funding A State Uses AI to Determine School Funding. Is This the Future or a Cautionary Tale?
Nevada reworked its funding formula hoping to target extra aid to students most in need. What happened could hold lessons for other states.
13 min read
Illustration of robotic hand putting coins into jar.
iStock / Getty Images Plus