Fifteen NBA teams are currently using camera technology that tracks not only what their players did while on the court, but what they should have done. Sounds crazy? It is.

This system, called SportVU and produced by Stats LLC., keeps track of every NBA player on the floor throughout the entire game. In the above video, taken from a Raptors/Knicks game, the Raptors’ analytics team went back after the contest and turned plays into playable video files written from code. The code was designed to understand everything from where a pass was thrown and whether it connected, when a player initiated a pick-and-roll, and where all players on the court were while a play unfolded.

The technology goes deep into calculations, even taking into account the player’s skill levels. The program understands when a player is more dangerous from mid-range than another, and when one player is taller than his defender.

Once everything is factored in, the program calculates what the players should have done. In the video, the clear circles indicate where a player should have been during a play. It combines data gathered from scouting reports of opposing players, coaching personnel and their team’s defensive tendencies, and the value of each decision is calculated multiple times a second.

In short, this technology takes the best elements statistics have to offer and combines them with visuals that players and coaches can watch to make better decisions during a game. The team says the system isn’t even running at full strength yet. In other words, this is the Super Nintendo version. Although the technology has been around for a few years, this is the first time one of the teams agreed to talk about their secret weapon.

[via Gizmodo]