Once statistics for an NBA game have been collected, their lifespan of usefulness is only just beginning. Players rely on them to hone skill development; teams leverage them for game-planning and scouting; fans and media alike use them to build or further a plethora of narratives.
The data the NBA collects on its referees is no different. That collection itself is just the beginning; this data is put to work in several ways that make an ongoing impact on the product seen on the floor.
Who has access to it? How is it used within the NBA’s officiating department, both for training and advancement purposes? How involved can NBA teams be in this process, and how do team decision-makers feel about it?
“We understand that referees learn best through video work, similar to players,” Matthew Futterman, the NBA’s director of referee operations, told FiveThirtyEight.
Futterman is the creator of the Referee Engagement and Performance System (REPS), a proprietary league database that officials and their supervisors have used since the start of the 2020-21 NBA season. Before REPS came along, referees largely communicated with supervisors and one another via a disorganized system of calls and emails; REPS centralized those exchanges.
Accessible via any device, REPS is made available not just to NBA officials, but also to those in the WNBA and the G League. (Some refs cross over between leagues.)
Video is at the foundation of many REPS components. The same video angles available to the NBA’s team of referee reviewers are also offered to officials themselves for the purposes of review, training and even dispute in some cases. If referees are confused about how or why they were graded on a call, they can tag their supervisor — or even fellow referees — for further discussion. All conversations among referees are kept private; supervisors don’t have access to them.
NBA
Management, particularly the NBA’s developmental advisers who oversee and train individual officials, can also use the system for ongoing development. They can tag single plays, “playlists” of 15 to 20 plays, or even long-term trends to single refs or groups of refs to focus their attention on the necessary areas. Every interaction is logged so there’s a history to draw from.
REPS also allows supervisors to view the league’s complete analytical profile and accuracy rankings for each ref.1 In coordination with the NBA’s data scientists, these features are often used to spot bird’s-eye officiating trends and take the appropriate action.
Mark Wunderlich, a former NBA official now serving as the league’s vice president of referee operations, oversees the developmental advisers who work with refs individually. Wunderlich gives a simple example from this past offseason of how the fusion between data and on-court work has improved the league’s processes.
“We realized we had [too many] incorrect decisions made from the lead position,” Wunderlich told FiveThirtyEight. “When we did the tracking of those misses, we realized the official was too tight.
“Any strong-side or down-lane drive, if an official was inside 8 feet to the baseline to the rim, he didn’t have a big enough picture to see the play. When we got outside 9 feet and a little bit wider, our vision angle expanded, and we ended up reducing our error count because the analytics proved to us that width was better from the lead position. … So when we went to [preseason] camp, we banged home plays that we had missed being too tight on video, through the help of the analytics.”
This is a level of precision that was simply not available to the NBA a decade ago, much less in prior generations.
“We’re starting to see more plays,” Wunderlich said. “I’ve had officials call me and say it’s like a whole new world out there.”
As with several other parts of the officiating program, limiting bias is a constant effort. REPS offers supervisors the ability to view trends and call data blindly, with actual referee names hidden until the analysis is completed to avoid potential favoritism.
Simple accuracy isn’t the only quality management is looking at, either. Developmental advisers also use REPS to rate their officials on themes like communication, confidence, rules knowledge and their ability to work as part of a three-ref crew while filling the proper roles.
Referees struggling with performance can use REPS as part of their pathway to improvement, a process the NBA hopes to foster — within reason. Officials who aren’t meeting the league’s accuracy and consistency standards will be put on notice and given one year to improve in their areas of weakness, per Monty McCutchen, senior vice president of referee development and training. Many are able to; those who are not are dismissed, an event McCutchen reports doesn’t happen every season necessarily, but is still a periodical occurrence.
Beyond these specific forms of evaluation and training, REPS is also a central hub for daily operations. Some of its other features:
- A dashboard/notification system for all interactions, plus a schedule section for upcoming games.
- A personal notes section for each official.
- A search function that can comb through any referee, any player, any call type, etc. — and all games in the NBA’s video library, though games older than four years or so often have to be requested in advance.
- Weekly “rules tests” and other periodic exams given to all league officials on rules knowledge, protocols and more.
- A resources tab that includes the rulebook, protocols and a case book.
- A bulletin board with contact info, travel instructions and other details.
- An expenses/logistics dashboard covering such information as flights, daily itineraries and per diem.
NBA
The system has revolutionized referee training, communication and even operations. Futterman said the league logged roughly 1,200 phone and email interactions between officials and management during the 2019-20 season, the year before REPS was implemented; last year, they logged more than 12,000 within the client, and not just from the NBA — also from officials within the WNBA and G League. There’s been real buy-in across the officiating ranks.
“We’re trying to change the way we think about training and development, and make it a more community-based approach,” Futterman said. “We’re seeing a lot of really neat successes with the platform.”
Team access is another important element of the NBA’s focus on a data-centric approach to officiating, one that saw a seminal shift back in 2015.
Prior to that point, teams received no specific referee data from the league; they could make inquiries or request certain limited chunks of information, but only on a case-by-case basis. Starting that year, though, teams began receiving the full officiating breakdown for all their own games, compiled by league reviewers.2
This process is managed through a system known as Team Inquiry Website (TIW). All 30 NBA teams have access to TIW, both to view their game breakdowns and to interact with the league in three ways:
- Pre-TIW inquiry: If a team spots a particular play or trend from a given game in real time and wants to inquire about or dispute it, the team can flag it directly after the game — before NBA reviewers have even begun combing through the data. The idea is to ensure reviewers pay particular attention to that area.
- Post-TIW inquiry: More commonly, teams will wait until receiving their game report to make any inquiries. If they have any major disputes, they can submit them to the NBA for review from a member of the league’s analytics team. A response will typically come back within 24 to 48 hours, per multiple team sources; in certain cases, the league will change a given play grade based on team disputes.
- Thematic inquiries: Teams can also submit inquiries based on longer-term trends involving a player, a play type or even a certain referee. One major caveat here, however: Any such inquiry is public within TIW, meaning all 29 other teams can see its contents and any response the league gives. For this reason, sources with multiple NBA teams who use the system say they’re often reticent about sacrificing a competitive advantage through this form of inquiry.
Opinions vary around the league on TIW and its ultimate value, as does the degree to which — and the ways in which — different teams use the data available to them.
Teams hoping for constant vindication from perceived officiating injustices are often left unsatisfied. While the league will occasionally change call grades based on team input, that sort of retroactive “told you so” rhetoric isn’t the primary goal; rather, the goal is to use team viewpoints to inform continuous referee training and improvement.
Many executives within NBA franchises surveyed for this story care more about a different area of referee grading and data: The role this information plays in determining which officials are promoted from the G League, which are given crew chief assignments and, perhaps most vitally, which are selected to officiate playoff and NBA Finals games.
The NBA maintains that three primary variables are at play for such forms of promotion:
- Referee grading analytics compiled by game reviewers and league analysts.
- Specific input/rankings from each NBA team regarding individual referees (typically provided by a team’s coach and/or general manager).
- Assessments of referees from the NBA’s developmental advisers and other top management/training staff.
The NBA steadfastly refuses to disclose the weights each of these three variables holds during any promotion or playoff assignment decisions. Each is given “substantial weight,” per Byron Spruell, NBA president of league operations and the man in charge of all things referee-related. Spruell also reports these scales were notably changed a few years ago — and when the league retroactively applied the new weights to the prior season, they found that the alteration to variable weights “doesn’t change [things] significantly.” Most in the department viewed that as a distinct positive, a sign that each input variable was leading to similar results.
Multiple league staffers were also quick to point out that these three variables are not the end-all, be-all for these decisions. Important intangibles like communication and confidence are also factored in, especially for vital playoff assignments. Spruell gave an example of these sorts of elements the league will consider when selecting its 12 primary Finals officials each season.
“Most of [the officials being considered for the Finals], if not all, are crew chiefs [normally],” Spruell told FiveThirtyEight. “When you get to the final 12, not everybody can be a crew chief. The ability to be a team player and play your role in the context of the playoffs still means a lot in terms of team composition.”
This realm inspires some definite consternation in certain NBA executive circles. Multiple team sources — requesting anonymity so as to avoid possible fines — told FiveThirtyEight, in essence, that they doubt the league’s truthfulness here. Some believe that when it really comes down to it, vital playoff and Finals assignments are based on favoritism above all else.
At least generally, you can understand where they’re coming from. It’s true that some of the same names are constant presences in the NBA’s biggest games. Six referees — Tony Brothers, James Capers, Marc Davis, Scott Foster, John Goble and Zach Zarba — have officiated in each of the past five NBA Finals. (David Guthrie has officiated each of the last four; Kane Fitzgerald and Eric Lewis, each of the last three.)
Is that favoritism, though, or is it the system working as intended?
“Players who make it into the Hall of Fame do it because they’re consistent year after year,” McCutchen told FiveThirtyEight. “It’s no different for referees.”
While noting that the league regularly experiences movement among referees between various playoff rounds, including three refs who worked their first Finals in 2021 (Courtney Kirkland, James Williams and Sean Wright), McCutchen pushed back strongly on the idea that these decisions are made by a “good old boys system.”
“Your year has to matter as an official. You can’t just roll out the ball and get to a [certain] position,” McCutchen said. “But neither are we going to penalize people who year after year, teams have ranked them in the top five. … When you have that kind of aggregate spit out the names year after year after year, that means they’re putting in the work. This isn’t referee operations capriciously deciding who we like and don’t like — the aggregate drives [these decisions].”
Some level of side-eye between teams and the league’s officiating department is unavoidable even in a perfect system. These are sensitive subjects; jobs and millions of dollars are regularly on the line.
Transparency is a constant tug-of-war. Some teams advocate for a sharing of the full referee grading and analytics data set with the entire league; the NBA has no such plans in the near future, per both Spruell and McCutchen. That’s for several reasons, including simple logistics, but one stands out in today’s era of newsbreakers: teams selectively leaking referee data to the media. It’s easy to envision a team losing a close game and feeling slighted by a given official, then leaking that official’s low rank in the league’s grading system to sway public opinion.
Some even call for this data to be released to the general public rather than just teams themselves. Spruell, a longtime executive at Deloitte whose process-oriented approach was praised by several of his subordinates,3 raised an interesting comparison in response.
“How many companies do you know that would just open up their confidential employee files? Not many,” Spruell said.
Spruell makes a fair point. So might those who reply that most companies aren’t like the NBA, which has public box scores for games dating back decades and clearly operates in the public eye. It’s easy to see both viewpoints, frankly.
Desires for transparency can border on quests for blood in some cases — understandable at the fan level, perhaps, but less so at the team level. Does being “transparent” mean sending a press release every time an NBA official is fined for misapplication of the rules (something that, per multiple league sources, happens several times per season)? Should the league publicize it when a referee is outright fired for continuously failing to meet the NBA’s accuracy and other quality thresholds?
Some earnestly believe the answer to those questions is yes. Is that really transparency, though, or are those just examples of punitive responses that do little to improve future performance?
There are also specific efforts aimed at fostering open lines of communication between teams and the officiating department. For several years, each NBA team has designated its own “team rules liaison,” usually a member of the coaching staff, who participates in monthly one-on-one meetings with league staff. These meetings can take many forms, from teams airing concerns about specific calls or player trends to video sessions that help teams understand why certain calls are being interpreted in a given way.
One team source put it best: “Why would the league want bad refs?”
Basketball officiating has come a long way since the days of VCR training and regional game graders. The resources devoted solely to grading and analyzing NBA, WNBA and G League referees rival those of many major corporations.
Is the system perfect? Assuredly not. Mistakes come with the territory. For those present across several generations of NBA officiating, though, today’s use of data and analytics allows for a greater understanding of — and improvement on — those mistakes than they could have ever envisioned.
“When I refereed, we were always behind the players. They always had something new,” Wunderlich said. “Now, I think we’re getting ahead of it a little bit.”