Sony patent reveals AI predicting game frames before rendering
Sony has filed a patent for a new artificial intelligence technology designed to predict the next frame of a game before GPU rendering is complete. The aim is to reduce visual stuttering.
A newly revealed patent application indicates Sony is developing an artificial intelligence system capable of predicting a game's next frame before the graphics processing unit (GPU) has finished rendering it. The primary objective is to smooth out game frame rates and mitigate stuttering.
Traditionally, games often wait for two rendered frames before inserting an interpolated frame. Sony's approach suggests the AI will predict the upcoming frame based on the current one, thereby reducing latency. This method bypasses the need to wait for two full frames to be rendered before interpolation.
The technology utilizes a jointly trained encoder-decoder codec. Rendered images are converted into a compact numerical representation, which the decoder uses to generate the predicted frame. The system operates in parallel with the graphics rendering pipeline. As the hardware renders the next frame, the AI processes the previous one and sends its prediction to the display buffer.
Furthermore, the system can incorporate motion vectors and recent player input. This historical data from player actions can form the basis for predicting the next frame's movements. The AI is trained using sequential video frames, comparing its predictions to actual subsequent frames to minimize errors.
The practical effectiveness of this technology will depend on the AI's ability to generate accurate predictions rapidly enough to fill potential gaps in frame output when rendering speed decreases.