Deep learning super sampling

Deep learning super sampling (DLSS) is an image upscaling technology developed by Nvidia and exclusive to Nvidia graphics cards for real-time use in select video games, using deep learning to upscale lower-resolution images to a higher-resolution for display on higher-resolution computer monitors. Nvidia claims this technology upscales images with quality similar to that of rendering the image natively in the higher-resolution but with less computation done by the video card allowing for higher graphical settings and frame rates for a given resolution.[1]

As of December 2020, this technology is available exclusively on GeForce RTX 20 and GeForce RTX 30 series GPUs.

History

Nvidia advertised DLSS as a key feature of the GeForce RTX 20 series GPUs when they launched in September 2018.[2] At that time, the results were limited to a few video games (namely Battlefield V[3] and Metro Exodus) because the algorithm had to be trained specifically on each game on which it was applied and the results were usually not as good as simple resolution upscaling.[4][5]

In 2019, the videogame Control shipped with ray tracing and an improved version of DLSS, which didn't use the Tensor Cores.[6][7]

In April 2020, Nvidia advertised and shipped with driver version 445.75 an improved version of DLSS named DLSS 2.0, which was available for a few existing games including Control and Wolfenstein: Youngblood, and would be available later for upcoming games. This time Nvidia said that it used the Tensor Cores again, and that the AI did not need to be trained specifically on each game.[2][8]

A side effect of DLSS 2.0 is that it seems not to work very well with anti-aliasing techniques such as MSAA or TSAA, the performance being very negatively impacted if these techniques are enabled on top of DLSS.[9]

As of April 2020, DLSS 2.0 must still be included on a per-game basis by the game developers.

Release history
Release Release Date Highlights
1.0February 2019First version, using AI and specifically trained for certain games, including Battlefield V and Metro Exodus[3]
2.0 (first iteration)August 2019First 2.0 version, also referenced as version 1.9, using an approximated AI of the in-progress version 2.0 running on the CUDA shader cores and specifically adapted for Control[6][2][10]
2.0 (second iteration)April 2020Second 2.0 version, using Tensor Cores again and trained generically[11]

Algorithm

DLSS 1.0

Nvidia explained that DLSS 1.0 worked for each target game image by generating a "perfect frame" using traditional supersampling, then trained the Neural network on these resulting images. On a second step, the model was trained to recognize aliased inputs on the initial result.[12][13]

DLSS 2.0

DLSS 2.0 works as follows:[14]

  • The neural network is trained by Nvidia using "ideal" images of video games of ultra-high resolution on supercomputers and low resolution images of the same games. The result is stored on the video card driver. It is said that Nvidia uses DGX-1 servers to perform the training of the network.[15]
  • The Neural Network stored on the driver compares the actual low resolution image with the reference and produce a full high resolution result. The inputs used by the trained Neural Network are the low resolution aliased images rendered by the game engine, and the low resolution, motion vectors from the same images, also generated by the game engine. The motion vectors tell the network which direction objects in the scene are moving from frame to frame, in order to estimate what the next frame will look like.[16]

Architecture

DLSS is only available on GeForce RTX 20 and GeForce RTX 30 series GPUs, in dedicated AI accelerators called Tensor Cores.[16][17]

Tensor Cores are available since the Nvidia Volta GPU microarchitecture, which was first used on the Tesla V100 line of products.[18] Their specificity is that each Tensor Core operates on 16 bits floating point 4 x 4 matrices, and seem to be designed to be used at the CUDA C++ level, even at the compiler level.[19]

The Tensor Cores use CUDA Warp-Level Primitives on 32 parallel threads to take advantage of their parallel architecture.[20] A Warp is a set of 32 threads which are configured to execute the same instruction.

See also

References
  1. "Nvidia RTX DLSS: Everything you need to know". Digital Trends. 2020-02-14. Retrieved 2020-04-05. Deep learning super sampling uses artificial intelligence and machine learning to produce an image that looks like a higher-resolution image, without the rendering overhead. Nvidia’s algorithm learns from tens of thousands of rendered sequences of images that were created using a supercomputer. That trains the algorithm to be able to produce similarly beautiful images, but without requiring the graphics card to work as hard to do it.
  2. "Nvidia DLSS in 2020: stunning results". techspot.com. 2020-02-26. Retrieved 2020-04-05.
  3. "Battlefield V DLSS Tested: Overpromised, Underdelivered". techspot.com. 2019-02-19. Retrieved 2020-04-06. Of course, this is to be expected. DLSS was never going to provide the same image quality as native 4K, while providing a 37% performance uplift. That would be black magic. But the quality difference comparing the two is almost laughable, in how far away DLSS is from the native presentation in these stressful areas.
  4. "AMD Thinks NVIDIA DLSS is not Good Enough; Calls TAA & SMAA Better Alternatives". techquila.co.in. 2019-02-15. Retrieved 2020-04-06. Recently, two big titles received NVIDIA DLSS support, namely Metro Exodus and Battlefield V. Both these games come with NVIDIA’s DXR (DirectX Raytracing) implentation that at the moment is only supported by the GeForce RTX cards. DLSS makes these games playable at higher resolutions with much better frame rates, although there is a notable decrease in image sharpness. Now, AMD has taken a jab at DLSS, saying that traditional AA methods like SMAA and TAA “offer superior combinations of image quality and performance.”
  5. "Nvidia Very Quietly Made DLSS A Hell Of A Lot Better". Kotaku. 2020-02-22. Retrieved 2020-04-06. The benefit for most people is that, generally, DLSS comes with a sizeable FPS improvement. How much varies from game to game. In Metro Exodus, the FPS jump was barely there and certainly not worth the bizarre hit to image quality.
  6. "Remedy's Control vs DLSS 2.0 – AI upscaling reaches the next level". Eurogamer. 2020-04-04. Retrieved 2020-04-05. Of course, this isn't the first DLSS implementation we've seen in Control. The game shipped with a decent enough rendition of the technology that didn't actually use the machine learning
  7. "NVIDIA DLSS 2.0 Update Will Fix The Geforce RTX Cards' Big Mistake". techquila.co.in. 2020-03-24. Retrieved 2020-04-06. As promised, NVIDIA has updated the DLSS network in a new Geforce update that provides better, sharper image quality while still retaining higher framerates in raytraced games. While the feature wasn’t used as well in its first iteration, NVIDIA is now confident that they have successfully fixed all the issues it had before
  8. "HW News - Crysis Remastered Ray Tracing, NVIDIA DLSS 2, Ryzen 3100 Rumors". 2020-04-19. Retrieved 2020-04-19. “The original DLSS required training the AI network for each new game. DLSS 2.0 trains using non-game-specific content, delivering a generalized network that works across games. This means faster game integrations, and ultimately more DLSS games.”
  9. "Evaluating NVIDIA DLSS 2.0 Quality And Performance In Mech 5 And Control". hothardware.com. 2020-03-27. Retrieved 2020-04-07. One side effect of DLSS is that it doesn't seem to play nicely with MSAA (forced through the drivers) or TXAA enabled in the game. Performance actually tanked pretty hard with either of those anti-aliasing methods on top of DLSS 2.0, with the Quality mode only performing around half as fast as no DLSS
  10. Edelsten, Andrew (30 August 2019). "NVIDIA DLSS: Control and Beyond". nividia.com. Retrieved 11 August 2020. we developed a new image processing algorithm that approximated our AI research model and fit within our performance budget. This image processing approach to DLSS is integrated into Control
  11. "NVIDIA DLSS 2.0 Review with Control – Is This Magic?". techquila.co.in. 2020-04-05. Retrieved 2020-04-06.
  12. "NVIDIA DLSS: Your Questions, Answered". Nvidia. 2019-02-15. Retrieved 2020-04-19. The DLSS team first extracts many aliased frames from the target game, and then for each one we generate a matching “perfect frame” using either super-sampling or accumulation rendering. These paired frames are fed to NVIDIA’s supercomputer. The supercomputer trains the DLSS model to recognize aliased inputs and generate high quality anti-aliased images that match the “perfect frame” as closely as possible. We then repeat the process, but this time we train the model to generate additional pixels rather than applying AA. This has the effect of increasing the resolution of the input. Combining both techniques enables the GPU to render the full monitor resolution at higher frame rates.
  13. A Supercomputer & AI Will Power NVIDIA RTX GPU's - NVIDIA RTX 2080 Performance. JAGS gaming. 2018-08-23. Retrieved 2020-04-19.
  14. "NVIDIA's Deep Learning Super Sampling (DLSS) 2.0 Technology Is The Real Deal". Forbes. 2020-03-29. Retrieved 2020-04-07.
  15. "NVIDIA DLSS 2.0: A Big Leap In AI Rendering". Nvidia. 2020-03-23. Retrieved 2020-11-25.
  16. "NVIDIA DLSS 2.0: A Big Leap In AI Rendering". Nvidia. 2020-03-23. Retrieved 2020-04-07.
  17. "NVIDIA TENSOR CORES". Nvidia. Retrieved 2020-04-07.
  18. "On Tensors, Tensorflow, And Nvidia's Latest 'Tensor Cores'". tomshardware.com. 2017-04-11. Retrieved 2020-04-08.
  19. "The NVIDIA Titan V Deep Learning Deep Dive: It's All About The Tensor Cores". AnandTech. 2018-07-03. Retrieved 2020-04-08.
  20. "Using CUDA Warp-Level Primitives". Nvidia. 2018-01-15. Retrieved 2020-04-08. NVIDIA GPUs execute groups of threads known as warps in SIMT (Single Instruction, Multiple Thread) fashion
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.