With the development of neural networks and machine learning technologies, the scope of their application is also expanding. If earlier neural networks were used exclusively for performing complex mathematical, medical, physical, biological calculations and forecasting, now these technologies are gaining wide popularity in a more "mundane" environment - in the field of entertainment. Taking only the first steps in this direction, they are already capable of demonstrating amazing and sometimes even outstanding results. Today we will analyze a few illustrative examples.
The process of video remastering is so complicated and time-consuming that we might never have seen many masterpieces of the world classics with a new, modern, clear and juicy picture. However, the world is full of smart fans and enthusiasts who are well versed in new technologies, and in particular neural networks and machine learning technologies, with which you can achieve amazing results even at home. For example, YouTube user Stefan Rumen with the pseudonym CaptRobau decided to demonstrate some of the capabilities of neural networks in processing videos of an old science fiction series.
His earlier work is Remako Mod, an "HD remake" of the classic and highly popular Japanese RPG Final Fantasy VII called. To do this, he used the AI algorithm AI Gigapixel, with the help of which he was able to scale the image of the original picture by 4 times, converting it to HD resolution without any significant changes in the original art design. Thus, while you are waiting for another decade until the moment when the Japanese developer and publisher of computer games Square Enix officially releases a remaster of perhaps one of the best parts of this game series, you can try Stefan Rumen's mod yourself by downloading it from this site.
By the way, recently, neural network technologies for remastering old games and bringing them to a more relevant and modern look without changing the general original concept has become a real trend among various modders. For example, not so long ago we talked about ESRGAN (Enhanced Super Resolution Generative Adversarial Networks) technology, which implements image scaling technologies with a 2-8x increase in quality. The algorithm is “fed” the original image with a low resolution, after which it not only increases the original resolution of the latter, but also improves the image quality, painting on realistic details and making textures “more natural”.
Comparison of texture quality: on the left is the original texture from the Morrowind game, on the right - processed by the neural network.
A character from Doom (left - was, right - became).
Background processing in Resident Evil 3.
Promotional video:
Be that as it may, in the intervals between the remastering of "The Seventh Final" Stefan Rumen decided to take up another project - to use the same machine learning technology, but this time to process frames of the classic science fiction series of the 90s. Rumen chose Star Trek: Deep Space Nine as the object for his experiments.
Scaling a live image of a TV series is very different in complexity from scaling a pre-rendered image of Final Fantasy VII, the author notes, so the final result, although it looks noticeably better than the original materials in low resolution, is still far from the ideal about which you could have dreamed since the first Blu-ray players hit the market. Occasionally, small "artifacts" appear on the screen. But, again, in general, everything looks more than worthy. But, in general, see for yourself.
For this project, Rumen also used the AI Gigapixel algorithm, which was trained to edit images based on real photographs. The author notes that the new picture was obtained in 1080p and 4k, but since Rumen does not have a TV or monitor with native 4K resolution, he cannot adequately assess the 4K version.
Unfortunately, you cannot watch the entire series in Full HD quality. The process of processing all the source material would have taken a very long time, so Rumen used only separate frames from different series for the demonstration. According to him, he took up this project for only one reason - to show that it is really possible. In his opinion, a whole team of professionals working in a large television company and having at their disposal more suitable and powerful computer equipment for such work will be able to cope with this task much better.
Using neural networks to simplify the work of processing old images from video games and movies are not the only areas where such technologies are able to show their talents. In the modern world, where panoramic cameras capable of producing 360 degrees, as well as virtual reality headsets, are gaining popularity, developers have begun to actively explore the potential of panoramic photography.
One of the latest developments in this direction is a neural network capable of sounding panoramic static images. It is authored by machine learning experts from Massachusetts Universities, Columbia Universities and George Mason University.
The created algorithm determines the type of environment and objects in the photo, and then selects and arranges sounds from the used database in accordance with the spatial calculation of the distance to their sources in this image. Thanks to this, the panoramic image acquires a realistic and spacious sound that allows you to evaluate the presented image in a completely new way.
According to the developers of this neural network, the technology may find interest among developers of VR content (films and games). The latter, in this case, will not have to manually overlay all the sounds on the panoramic image, the neural network will be able to do it all on its own.
Nikolay Khizhnyak
