You're digging through some old moving boxes you never unpacked, and you discover your old console with a bunch of retro games. You dust it off, and after managing to hook it up to your shiny 4K flat screen TV, you find that it looks, well, did Link always look so blurry? Part of the problem is that old consoles like the SNES, had a resolution of just 256 by 224 pixels, compared to the 3840 by 2160 pixels of a modern 4K TV. That means that if your TV were to show the game at its true native resolution, it would be this tiny. And you'd have to get so close to the screen, you'd go cross-eyed. So instead, a television will take that small media and upscale it so that it fills up most, if not all of the available pixels. To do this, the TV manufacturer had to choose just one of several different upscaling algorithms.
The most common ones are
- Nearest neighbor
With Nearest neighbor being the simplest and fastest to process, and Lanczos being the slowest and most complex.
However, on modern-day hardware, any of these algorithms can easily upscale an image in real time, which, for this video, I am defining as 16 milliseconds or less. And that's because, if you're playing a game at 60 frames per second, each individual frame will appear on screen for about 16 milliseconds.Therefore, each of these algorithms should work faster than that. Which also means that buffering multiple frames is not necessary. However, for AI or smart image upscaling, a multi-frame buffer can be necessary, which will result in a delay or input lag of much more than 16 milliseconds.
Assuming that each of these algorithms can work in real time, which one actually looks the best? Well, see for yourself. And, immediately you can see that only Nearest neighbor can deliver the perfect sharp pixels that make pixel art look good. All the others just kind of blend the pixels together, which blurs the image. So the upscaling alogorithm on your TV is probably some form of Bicubic or Lanczos, raising the question, why would anyone want an upscaling algorithm that's more complicated, slower, and looks worse? Because, while Nearest neighbor's scaling looks great for pixel art, it does not look good for most other images. Your TV doesn't know that you've plugged in a retro game console, as opposed to a DVD player or something. And for live action video, Bicubic or Lanczos scaling really are better choices. The real world is not made of pixels. And so, when we upscale a photograph, the results should not look pixelated. 2D Mario is made of pixels, and, therefore, when we upscale him, the results should be pixelated.
"But wait," I hear you say. "My TV already has a game mode setting. "Why doesn't it just use Nearest neighbor for that?" Well, the manufacturer would have to include a Nearest neighbor menu option, which doesn't sound so hard. As for why they don't do that, I don't know. TV manufacturers just don't care I guess. Maybe because not enough consumers know or care about this problem. But, if you really want those sharp pixels, I have good news. There are a lot of options like the RetroTINK-2X, FrameMeister, or Open Source Scan Converter. But, before you buy anything, I recommend you check out this great RGB video explainer series from My Life in Gaming because this stuff can get really complicated.
I ended up just buying a $14 component cable for my Wii, which provides much better quality than the stock composite cable. And, it's not perfect, but, you know what? It's good enough.
Now, if you're emulating your old games, you have way more options. Emulators like RetroArch have access to an arguably even better scaling method called Sharp-bilinear, which uses Nearest neighbor for prescaling to the closest integer and then uses Bilinear scaling for the remainder. This method even allows you to rotate the image without getting any jagged edges. There are also tons of other interesting scalers and shaders that can be mixed and customized to get just the look you want. And blurry pixel art is not just a problem for retro gamers. If you're a digital artist, you should know that programs like GIMP, Photoshop, and After Effects do have support for Nearest neighbor if you know where to look. But some creative software, including Premiere, does not.
Also, support for integer scaling from PC hardware manufacturers like AMD, NVIDIA, and Intel has been getting better lately, but we've still got a ways to go.