Intel Photoreal Enhancement ML project demoed in GTA V

by Mark Tyson on 13 May 2021, 13:11

Tags: Intel (NASDAQ:INTC), Rockstar

Quick Link: HEXUS.net/qaeqk4

Add to My Vault: x

Researchers at Intel labs have been looking at enhancing the realism of synthetic images. Stephan R. Richter, Hassan Abu Alhaija, and Vladlen Kolten recently published a research paper and accompanying demo video showing a new ML-based technique in action - changing the great but obviously synthetic graphics in GTA V to a much more realistic and photoreal experience.

The new ML rendering method is described as enhancing the in-game graphics by leveraging a convolutional network that has been trained on the 'Cityscapes dataset'. The results are clearly different and appealing in a real-world way. However, it is explained that the 'Cityscapes dataset' isn't derived from driving around Southern California, but mostly from driving around cities in Germany with an automotive grade camera. One unintended side effect of using the German data is that the landscapes in San Andreas become greener, and the roads smoother and less pot-holed. Other effects like the shading and lighting of cars and buildings translate very well, but I think SoCal-centric data would make the whole photorealism effort a bit sunnier.

Above you can see a diagram which outlines the process each frame will go through in order to apply the new Intel Photorealism Enhancement process. A very important part of that flow is the use and analysis of game rendering buffers called 'G-buffers' which help the ML work by categorising. These buffers help by revealing data such as object distance to the camera, scene materials, light positioning, and object IDs – thus trees in the scene will be treated and enhanced differently to cars.

The Intel researchers learnt a lot about optimising this type of Photorealism Enhancement in the process of implementing it in GTA V, and that helped them refine this ML technique as well as proposing a new strategy for sampling image patches during training, and introducing multiple architectural improvements in the deep network modules used for photorealism enhancement. The new technique also stands up well against other photorealism techniques with "substantial gains in stability and realism" in comparison.

If you want to read more and play with some GTA V photorealism image comparison sliders then head on over to the project's official GitHub page.



HEXUS Forums :: 8 Comments

Login with Forum Account

Don't have an account? Register today!
Does it look more real? Definitely. Does it look better? Not really.
jimcumming
Does it look more real? Definitely. Does it look better? Not really.
This.
It looks way worse IMHO.

Big applause for the ML network they created but the learning set they used was “sad gray cities”, not the amazing hot and sunny coastal town.
DevDrake
jimcumming
Does it look more real? Definitely. Does it look better? Not really.
This.
It looks way worse IMHO.

Big applause for the ML network they created but the learning set they used was “sad gray cities”, not the amazing hot and sunny coastal town.

I can see this perhaps being very useful in a different game. But people play games for colourful, exciting stimulation. This just reminds me of how boring city life is.

Now, can we take a moment to appreciate how good the person playing GTA was trying to be? He clearly wanted to go on the wrong side of the road and flatten some motorcyclists….
Looking realistic VS looking good are far from what the average Joe wants / feel.

For instance for me, i have yet to see really good looking footage from any action camera, it is also the same with action cameras, here it is often really bad.
Another place you often see it is in commercial ( i want to say photography but really i should say paint-shopping )

The truth !!!!!! you cant handle the truth.
I can see how this would look better if the source data was taken on a sunny day in LA, or even better across several different weather conditions.

It would be interesting to see how this tech would translate to faces, although perhaps dystopian. And if it became common place in games set in the current day or near present/past it would inherently cause games set in the past or future to have a different art style.