Computer Graphics R&D: An industry perspective

Slides from a recent talk I gave at the University of British Columbia.

Download here.

Tone mapping & local adaption

My friend Bart recently wrote a series of great articles on tone mapping, showing how localized models can be useful in certain scenarios, and I strongly advise to head over to his blog to read that one first, which motivated me to finally dump some of my notes and small experiments on the blog...

Local tone mapping refers to the idea of varying the tone mapping per pixel, instead of applying a single curve globally on the frame. Typically the curve is adapted by using some neighborhood-based filter to derive a localized adaption factor, which can help preserve details in both bright and dark areas of the image.

At the extreme, local operators give the typical "photographic HDR" look, which looks very unnatural as it's trying to preserve as much texture information possible at the detriment of brightness perception.

HDR photography example from lifehacker

While these techniques help display the kind of detail we are able to perceive in natural scenes, in practice the resulting image only exacerbates the fact that we are displaying our content on a screen that doesn't have the dynamic range that natural scenes have, and the results are very unrealistic.

That's why global tonemapping is not only cheaper but most often appropriate for games, as we aim to preserve the perception of brightness even if it means sacrificing detail.
We like brightness preservation so much in fact, that two of the very rare instances of videogame-specific visual devices in photorealistic rendering are dedicated to it: bloom and adaption.

Veiling glare in COD:AW

Most of photorealistic videogames still, and understandably, borrow heavily from movies, in terms of their visual language (even more so than photography), but you will never see the heavy-handed exposure adaption videogames employ, in movies, nor you'll see what we usually call "bloom", normally (unless for special effect, like in a dreamy or flashback sequence).

Of course, a given amount of glare is unavoidable in real lenses, but that is considered an undesirable characteristic, an aberration to minimize, not a sought-after one! You won't see a "bloom" slider in lightroom, or the effect heavily used in CG movies...

An image used in the marketing of the Zeiss Otus 24mm,
showing very little glare in a challenging situation

Even our eyes are subject to some glare, and they do adapt to the environment brightness, but this biological inspiration is quite weak as a justification for these effects, as no game really cares to simulate what our vision does, and even if that was done it would not necessarily improve perceived brightness, as simulating a process that happens automatically for us on a 2d computer screen that then is seen through our eyes, doesn't trigger in our brain the same response as the natural one does!

Trent Parke

So it would seem that all is said and done, and we have a strong motivation for global tonemapping. Except that, in practice, we are still balancing different goals!
We don't want our contrast to be so extreme that nothing detail can be seen anymore in our pursuit of brightness representation, so typically we tweak our light rigs to add lights where we have too much darkness, and dim lighting where it would be too harsh (e.g. the sun is rarely its true intensity...). Following cinematography yet again.

Gregory Crewdson

How does local tone mapping factor in all this? The idea of local tone mapping is to operate the range compression only at low spatial frequency, leaving high-frequency detail intact. 
One one hand this makes sense because we're more sensitive to changes in contrast at high-frequency than we are at lower frequencies, so the compression is supposed to be less noticeable done that way.

Of course, we have to be careful not to create halos, which would be very evident, so typically edge-aware frequency separation is used, for example employing bilateral filters or other more advanced techniques (Adobe for example employs local laplacial pyramids in their products, afaik).

But this kind of edge-aware frequency separation can be also interpreted in a different way. Effectively what it's trying to approximate is an "intrinsic image" decomposition, separating the scene reflectance from illuminance (e.g. Retinex theory).

So if we accept that light rigs must be tweaked compared to a strict adherence to measured real scenes, why can't we directly change the lighting intensity with tone mapping curves? And if we accept that, and we understand that local tonemapping is just trying to decompose the image lighting from the textures, why bothering with filters, when we have computed lighting in our renderers?

This is a thought that I had for a while, but didn't really put in practice, and more research is necessary to understand how exactly one should tone-map lighting. But the idea is very simple, and I did do some rough experiments with some images I saved from a testbed.

The decomposition I employ is trivial. I compute an image that is the scene as it would appear if all the surfaces where white (so it's diffuse + specular, without multiplying by the material textures) and call that my illuminance. Then I take the final rendered image, divide it by the illuminance to get an estimate of reflectance.

Note that looking at the scene illuminance would also be a good way to do exposure adaption (it removes the assumption that surfaces are middle gray), even if I don't think adaption will ever by great done in screen-space (and if you do so, you should really look at the anchoring theory), better to look at more global markers of scene lighting (e.g. light probes...).

Once you compute this decomposition, you can then apply a curve to the illuminance, multiply the reflectance back in, and do some global tone mapping as usual (I don't want to do -all- the range compression on the lighting). 

Standard Reinhard on the right, lighting compression on the left.
The latter can compress highlights more while still having brighter mids.

I think with some more effort, this technique could yield interesting results. It's particularly tricky to understand what's a proper midpoint to use for the compression, and what kind of courve to apply to the lighting versus globally on the image, and how to do all this properly without changing color saturation (an issue in tone mapping in general), but it could have its advantages.

In theory, this lies somewhere in between the idea of changing light intensities and injecting more lights (which is still properly "PBR"), and the (bad) practices of changing bounce intensity and other tweaks that mess with the lighting calculations. We know we have to respect the overall ratios of shadow (ambient) to diffuse light, to specular highlights (and that's why tweaking GI is not great). 
But other than that, I don't think we have any ground to say what's better, that's to say, what would be perceptually closer to the real scene as seen by people, for now, these are all tools for our artists to try and figure how to generate realistic images.

I think we too often limit ourselves by trying to emulate other medias. In part, this is reasonable, because trying to copy established visual tools is easier both for the rendering engineer (less research) and for the artists (more familiarity). But sooner or later we'll have to develop our own tools four our virtual worlds, breaking free of the constraints of photography and understanding what's best for perceptual realism.

"Conclusions"

Adding lights and changing the scenes is a very reasonable first step towards dynamic range compression: effectively we're trying to create scenes that have a more limited DR to begin with, so we work with the constraints of our output devices.

Adding fill lights also can make sense, but we should strive to create our own light rigs, there's very little reason to constrain ourselves to point or spots for these. Also, we should try to understand what could be done to control these rigs. In movies, they adapt per shot. In games we can't do that, so we have to think of how to expose controls to turn these on and off, how to change lighting dynamically.

Past that, there is tone-mapping and color-grading. Often I think we abuse of these because we don't have powerful tools to quickly iterate on lighting. It's easier to change the "mood" of a scene by grading rather than editing light groups and fill lights, but there's no reason for that to be true! 

We should re-think bloom. If we use it to represent brightness, are the current methods (which are inspired by camera lenses) the best? Should we experiment with different thresholds, with maybe subtracting light instead of just adding it (e.g. bring slightly down the scene brightness in a large area around a bright source, to increase contrast...).

And also, then again, movies are graded per shot, and by masking (rotoscoping) areas. We can easily grade based on distance and masks, but it's seldom done.

Then we have all the tools that we could create beyond things inspired by cinematography. We have a perfect HDR representation of a scene. We have the scene components. We can create techniques that are impossible when dealing with the real world, and there are even things that are impractical in offline CGI which we can do easily on the other hand. 
We should think of what we are trying to achieve in visual and perceptual terms, and then find the best technical tool to do so.

Look closely

The BRDF models surface roughness at a frequency bigger than wavelength, but smaller than observation scale. 

In practice surfaces have details at many different scales, and we know that the BRDF has to change depending on the observation scale, that is for us the pixel projected footprint onto the surface.

Specular antialiasing of normal maps (from Toksvig onwards) which is very popular nowadays models exactly the fact that at a given scale the surface roughness, in this case represented by normal maps, has to be incorporated into the BRDF. The pixel footprint is automatically considered by mipmapping.

Consider now what happens if we look at a specific frequency of surface detail. If we look at it close enough, on an interval of a fraction of the frequency wavelength, the detail it will “disappear” and the surface will look smooth, determined only by the underlying material BRDF and an average normal over such interval.

If we “zoom out” a bit though, at scales circa proportional to the detail wavelength, the surface starts gets complicated. We can’t anymore represent it over such intervals as a single normal, nor it’s easy to capture such detail in a simple BRDF, we’ll need multiple lobes or more complicated expressions.

Zoom out even more and now your observation interval covers many wavelengths, and the surface properties look like they are representable again in a statistical fashion. If you’re lucky it might be in a way that is possible to capture by the same analytic BRDF of the underlying material, with a new choice of parameters. If you are less lucky, it might require a more powerful BRDF model, but still we can reasonably expect to be able to represent the surface in a simple, analytic, statistical fashion. This reasoning by the way, also applies to normalmap specular antialiasing as well...

The question is now, in practice how do surfaces look like? At what scales do they have detail? Are there scales that we don’t consider, that is, that are not something we represent in geometry and normal maps, but that are not either representable with the simple BRDF models we do use?

- Surfaces sparkle!

In games nowadays, we easily look at pixel footprints of a square millimetre or so, for example that is roughly the footprint on a character’s face when seen in a foreground full body shot, in full HD resolution.

If you look around many real world surfaces, over an integration area of a square millimetre or so, a lot of materials “sparkle”, they have either multiple BRDF lobes or very “noisy” BRDF lobes.

If you look closely at most surfaces, they won’t have smooth, clear, smooth specular highlights. Noise is everywhere, often "by design" e.g. with most formica countertops, and sometimes "naturally" even in plants, soft plastics, leather and skin notably.

Some surfaces have stronger spikes in certain directions and definitely “sparkle” visibly, even at scales we render today. Road asphalt is an easy and common example.

There are surfaces that have roughness at a frequency that is high enough to be represented always by a simple BRDF, but many are not. Some hard plastics (if not textured), glass, glazed ceramic, paper, some smooth paints.

Note: Eyes can be deceptive in this kind of investigation, for example Apple’s aluminum on a macbook or iphone looks “sparkly” in real life, but its surface roughness is probably too small to matter at the scales we render. 

This opens another can or worm that I won’t discuss, of what to do with materials that people “know” should behave in a given way, but that don’t really at the resolutions we can currently render at...

My Macbook Pro Retina

- Surfaces are anisotropic!

Anisotropy is much more frequent than one might suspect, especially for metals. We tend to think that only “brushed” finishes are really anisotropic, but when you start looking around you notice that many metals shows anisotropy, due to the methods used to shape them. 

Similar artifacts are sometimes found also in plastics (especially soft or laminated) or paints.

On the left: matte wall paint and a detail of its texture
On the right: the metal cylinder looks isotropic, but close up reveals some structure

It's also interesting to notice how the anisotropy itself can't be really captured by a single parameter, skewing the BRDF in a tangent space, but that there are many different distributions of anisotropic roughnesses.

- Surfaces aren’t uniform!

At a higher scale, if you have to bet how any surface looks at the resolutions we author specular roughness maps, it will always somewhat be non-uniform pixel to pixel.

You can imagine how if it’s true that many surfaces have roughness at frequencies that would require complex, multi-lobe BRDFs to render at our current resolutions, how it is even more likely that our BRDFs won’t ever have the exact same behaviour pixel to pixel.

Another way of thinking of the effects of using simpler BRDFs driven by uniform parameters is that we are losing resolution. We are using a surface representation that is probably reasonable at some larger scale, per pixel at a scale at which it doesn’t apply. So it’s somewhat similar to doing to right thing with wrong, “oversized” pixel footprints. In other words, over-blurring.

- Surfaces aren’t flat!

Flat surfaces aren’t really flat. Or at least it’s rare, if you observe a reflection over smooth surfaces as it “travels” along them, not to notice subtle distortions.

Have you ever noticed the way large glass windows for example reflect? Or ceramic tiles?

This effect is at a much lower frequency of course of the ones described above, but it’s still interesting. Certain things that have to be flat in practice are, to the degree we care about in rendering (mirrors for example), but most others are not.

- Conclusions

My current interest in observing surfaces has been sparkled by the fact I’m testing (and have always been interested) hardware solutions for BRDF scanning, thus I needed a number of surface samples to play with, ideally uniform, isotropic, well-behaved. And I found it was really hard to find such materials!

How much does this matter, perceptually, and when? I’m not yet sure. As it's tricky to understand what needs more complex BRDFs and what can be reasonably modeled with rigorous authoring of materials.

Textures from The Order 1886
Specular highlights are always "broken" via gloss noise,
even on fairly smooth surfaces

Some links:

• Also in the article:
• Survey of non-linear prefiltering
• Discrete stochastic microfacet models (check out the video as well)
• Debevec's facial microgeometry and Jorge's realtime approximation
• Rendering glints on high-resolution normal-mapped specular surfaces
• Further links:
• Nextlimit Maxwell Renderer and Thin Film
• Linear efficient antialiased displacement and reflectance mapping
• Psycopath renderer
• Rendering of grooved surfaces
• Ocean rendering with seamless transition between geometry and BDRF

Mental note: shadowmap-space filters

A thought I often had (and chances are many people did and will) about shadows is that some processing in shadowmap space could help for a variety of effects. This goes from blurring ideas (variance shadow maps and variants) to the idea of augmenting shadowmaps (e.g. with distance to-nearest-occluder information).

I've always discarded these ideas though (in the back of my mind) because my experience with current-gen told me that often (cascaded) shadowmaps are bandwidth-bound. To a degree that even some caching schemes (rendering every other frame, or tiling a huge virtual shadowmap) fail because the cost of re-blitting the cache in the shadowmap can exceed the cost of re-rendering.

So you really don't want to do per-texel processing on them, and it's better instead to work in screenspace, either by splatting shadows in a deferred buffer and blurring, or by doing expensive PCF only in penumbra areas and so on (i.e. with min/max shadowmap mipmaps to compute trivial-in shadow and trivial-out shadow cases and branch).

It seems though that lately caching schemes are becoming practical (probably they are already for some games on current-gen, by no mean my experience on the matter in Space Marines can be representative of all graphic loads).

In these cases it seems logical to evaluate the possibility of moving more and more processing in shadowmap space. 

Then again, a reminder that a great metaheuristic for graphics is to try to reframe the same problem in a different space (screen, light, UV, local, world... pixel/vertex/object...)

Just a thought.

Notes on Epic's area lights

Finally, PBR is becoming mainstream... But, it's not easy, especially if you want to stay "correct". Are you sure your pipeline has no errors? Do you know the errors you have? What is your ground truth? Acquired data? Path traced solutions?

I'm planning to share some of my notes on PBR methods and certain findings and thoughts, this is a starter, on Brian Karis' excellent "representative point" area lighting as explained in Siggraph's 2013 PBR course.

I won't say much about it actually, mostly because my own research is not cleared for publication yet. If I were you though, I would also keep an eye on GPU Pro 5, as Michal Drobot's publication might change the state of the art once more (and I'm not really teasing, to say the truth I haven't tried his method yet and compared it, but I think it's "better").

As a good PBR renderer (or any renderer really) should know what kind of errors it's committing, I implemented an area light integrator and used it to verify a few ideas, including Epic's:

Blue dots: ground truth. Meshed plot: Epic's

It turns out the representative point solution does not do a great job at "preserving the shape" of the underlying BRDF, and it (quite understandably) just "caps" it with a spherical arc. Also note that at more grazing angles the ground truth is quite different and its cap starts to have a slanted angle as well.

Normalization is also interesting, here I actually thought Epic's method would fare worse (as it seemed to be just an heuristic with not enough justification behind it), but it's actually quite close to ground truth. Always check...

The smaller, gray mesh is the ground truth, the non-gridded light blue surface is Brian's normalization for the representative point solution, the gridded light blue one is my own version.

It is possible to do better, with varying degrees of complexity. At a given point things start to be needlessly complicated so, when you start looking at data try not to lose track of what your artists want and what makes a perceivable difference.

Don't do my mistake of staying too long in Mathematica to find a perfect solution, without verifying in actual shaders that you've passed the point where it matters...

Arguably for example, for small lights, the "roughness modification" solution is actually better (don't you love when the old OpenGL fixed-pipeline model, which had a specular power modifier in the lights, is more right than people thought for years?), Brian notes in his writeup that for small lights that is indeed a good method, but you might want to think twice considering if you need big lights or just lights big enough to "correct" for a roughness factor that otherwise would end up wrong in the materials.

One of my own analytic area approximations

Some other food for thought:

• How much does hemispherical shadowing (area light dipping under the normal hemisphere visibility) does matter?
• Should more of your sky be represented analytically? Note that it is true that the sun itself has a small arc when seen from the earth, but with the scattering that happens in the sky a larger area could be represented as "area light". The advantages are that it would work better with the BRDF and have more "correct" shadowing... 

Controlling light

Qui habet aures audiendi, audiat

Gregory Crewdson - behind the scenes

• Lighting, science and magic
• Hot shoe diaries
• Brief encounters

Tuning: Two-axis controllers, OSC and multitouch.

So far, and even in the near future I'd say, game visuals, even with physically-based (or inspired... or "reasonable") shading, have been a matter of tweaking and tuning, both by the artists and by the programmers, functions and parameters. Thus, the speed and intuitiveness at which this tuning, iteration happens, has a crucial, direct impact on the end results.

Now, if you've been following this blog for a bit, you might have noticed that I'm a strong proponent of live-coding, hot-swapping and all manners of ways to transform our art into a direct feedback mode of manipulation.

So I was very curious when I had, two weeks ago, the chance to attend talks by two of the forefront proponents and innovators in this field, the designer-coder-instructor-inventor Bret Victor and Pixar's and demoscene's manipulator of functions extraordinaire, Inigo Quilez.

I won't go into the details of these, but one thing struck me that I thought was worth further research and sharing: IQ's use of OSC to connect hardware midi (well, OSC) controllers to shader parameters in his demo/livecoding/vj system. Of course, such things make perfect sense when you're VJing in a club, and I've seen even in our industry some experiments with midi devices to control a variety of things in a game, honestly my experience relegated them as being most often nothing but a gimmick. Inigo, begin the smart guy he is, also came to a similar conclusion, with an exception though, when it comes to tune two or more correlated parameters, they have their space.

Of course! All of a sudden I feel so stupid having had to explain so many times all kind of multiply/add controls to artists! MADD is the shader coder best friend, it's cheap, it's powerful, it ends up everywhere. But for the artist, even when you "convert" it into the often more friendly add-multiply (just postprocess the data out of the tuning tool), and using the right bounds, constraints and so on, it often comes to be a pain to manage. I've always had these controls bound to two separate sliders, which is a really stupid idea, on a non-multitouch device (or without physical independent sliders).

Of course, you could just use a two-axis pad and control it with the mouse, the solution is quite obvious, honestly, that's why you should have a human interface designer in the team, not only for the game UI, but for tools, for in-game data display and so on. But I digress...

I went on and did some research on OSC and devices. There are too many things! I'd say, a good starting point is to take a look at a client for multitouch devices (iSomething, Android or whatever). These are nice to play with because they are easily configurable and support bi-directional communications (which some hardware devices implement, often at a premium price).

Even more interestingly, some applications, like the open-source Mrmr and Control (whose development, unfortunately, don't seem to be very active nowadays) come with support for dynamic configuration and scripting, so your game or tool can send messages to change the layout of the controller, which basically ends up to be a remote, programmable GUI - per se an awesome thing to have. Control is essentially a HTML5/JS GUI connected to OSC, with OSC able to feedback into the JS to manipulate the GUI remotely!

Image from the TouchOSC website

Now of course, all of this could be easily done even in whatever custom tuning environment you already have, in the end, nothing of this is rocket science, remote displays for GUIs existed twenty years ago, and probably you already have a decent collection of widgets, maybe some good colour pickers, some axis/angle selectors, sliders and so on.

The game-changer here in my mind is the multitouch support though, which enables a category of manipulators impossible on a PC alone.

Also once you support OSC, you automatically will be able to use any kind of OSC device, driver, or aggregator. A joystick as input? A webcam? Sound? Wiimote? A sequencer (see this, it's pretty crazy)? Custom hardware? You name an input method and there is probably a Midi or Osc interface for it, and pretty much any programming language will have bindings, and most libraries for creative coding, like the multiplatform OpenFrameworks.

Hardware is surprisingly not too expensive either, you can get something basic like a Korg Nanocontrol for sixty bucks, and even fancier interfaces like the QuNeo, retail for around two hundred. A fun world.

A hunch

Have you noticed how nice realtime raytracing looks? I was just watching this recent video by Sam Lapere done with Jakko Bikker's famous Brigade renderer. Yes it's noisy, and the lighting is simple, and the models are not complex. But it has a given quality to it, for sure, it feels natural and well lit.

I have a hunch.

It's not (mostly) because of the accuracy in the visibility solution. Nor it is because of the accuracy (which, at least in that scene, does not even seems to be a factor) of the BRDF and material representation. I think that with our skimpy realtime GPU rasterization hacks we are technically capable of producing materials and occlusions of a good enough quality.

I suspect that where games often fall is on finding the right balance. Raytracing does not need this balancing at all, diffuse, ambient specular, they all bounce around as a single entity, and light simply is. In the realtime rasterization world, this unity does not exist, we have components and we tune them and shape them. We start with a bit of diffuse and subtract shadows and add ambient and subtract its occlusion and sprinkle specular on top. Somehow... And sometimes this complex mix is just right, most often, plain wrong.

It's artist's fault? Is it too complex to get right? I think, not, it should not be. In many cases, it's not hard to take references for example, and meaningful ones, measures and measures that split a real world scene into components, devise experiments and validate our effects. Know what we are doing...

It's that often us, rendering engineers, work on technical features and not their quality. We do "SSAO" and add some parameters and if the artists say they're happy we move on, this is our relationship with the image, it's a producer-consumer one.

I think this is irresponsible, and we should be a little more responsible than that, work a bit more closely together. Observe, help, understand, light is a technical and artistic matter, and often we underestimate how much complexity and technicality there is into things that are not strictly complex routines in code. If you think about it, until a couple of years ago, we were still doing all wrong math on colors, and the solution is a simple pow(col,2.2) in code, but still, we did it spectacularly wrong, and called ourselves engineers. We should really understand way better what we do, both its physics and the perception of the physics.

Next-gen: Quality vs Quantity

Pixar's Luxo JR, 1986. 

Is this rendered "offline"? Would we mistake this frame as something our realtime engines are capable of? We can go any number of years in the past and look at offline outputs for movies and shorts, up to the very beginnings of computer graphics, and still recognize a gap between videogames and movie productions. How comes?

It's a matter of quality. Siggraph ended last week and Johan Andersson delivered yet again a clear, precise and technically superb presentation on what are the key challanges of realtime rendering. And yet again at number one he identified "cinematic quality" as the priority.

Now of course at a technical level, this implies research on antialiasing solutions (Have you ever noticed how -some games- do look incredible by just bumping up on PC the sampling rate?), spatially and temporally. His presentation is great so if you haven't read it yet, stop reading right now and fetch that first (and be sure also to follow the links and references he included).

But, I would argue there is more to it than our technical limitations, it is an attitude and a way of working. When creating a realtime rendering experience we are always constrained by our hardware and human resources. We have to make compromises, all the time, and one fundamental compromise we make is between quantity and quality. Should we add or refine?

This is indeed a very fundamental tradeoff, we don't encounter it only in rendering but all across our production and in all activities beyond the videogame industry as well. Isn't it the same compromise that lends to code rot and technical debt in our software? I would dare to make a comparison between Apple and other companies, but that's probably not the best and I could end up having to remove too many comments... Leica versus Canon?

Ok, going back on track, are we doing our best job at this compromise? When we add a visual feature to our rendering, how conscious are we of where it falls on the quality versus quantity line? Is quality even really considered as a target, or do we still reason in terms of features on the "back of the box"?

Of course if you ask that question, everybody would indeed say that quality is their focus. Quality sounds good as a word, it's the same as "workflows" and "tools" and "iteration", everybody wants to have these things, especially if the come for free. We have a "great" team, so quality is a given, right?

Have you ever finished a task for a rendering feature, and it works in game. And it's considered "done"? Or gets tested by an artist, it doesn't crash, it's understood, done? When does quality ever enter this process? I won't rant on otherwise I'll spend half of the post just talking about similar things and companies that value their tools by assigning the most junior programmers to the task and such horrors that really deserve their own space another time.

Do we really know what matters anyways? It is easier for everyone really to go for the quantity, add another feature, it looks like progress, it's easy to track on the schedule, it's easy to show in a trailer, caption and tell the world, we spent money in this new thing, regardless of how it feels. Next-generation HDR and bloom, rings a bell? Crank the parameters to a million and show your work...

Great rendering technology is not supposed to be seen, without trying to be romantic, it should be "felt". The moment you're capable of identifiying rendering techniques in an image, and not layers visual elements in the image (i.e. volume, texture, silouhette, shadows) we already have failed somehow.

If you can't do it well, don't do it at all. I think about Toy Story and how Pixar chose not to make a story with organic creatures, believing they could not yet deliver that level of complexity. And then of Crysis 2, a great game, with stable graphics (one of the first things I noticed, almost defect free), that I bought and finished.

I make this example often really, because it's such a popular game that most especially renderers have played, which does something peculiar, it fades your LODs pretty aggressively, especially it seems on small detail objects, like rocks on the ground which dissolve into nothing after a few meters. And to me that's surprising, in such a well crafted game, why does it has to show me that rock and then fade it? I can see it, I can see the trick, and I don't want to. I can live without the rock, I would not complain about it not being there, but now I can see it dissolving, I can see the render tech.

The truth is, from my experience at least, we are still far from understanding what makes great realtime visuals. Not only how to manage them, how to produce them and how technically how to solve these problems but also exactly what matters and what doesn't. So you see (and I'm making that up, right now I'm not thinking at one specific title) games with yellow skin highlights and glowing cyan sky that spent time into a dubious water simulation at a scale where it won't anyways be sold as right.

And then when it's all done and badly, we layer on top thick pass of bloom (possibly temporally unstable) and vignette and tinting and lensflares that would make Instagram blush, and we call it done.

In part it's a lack of education, we're a naive, young industry that only now is transitioning into a mature craft with fierce competition and we often just lack the right skills in a team. In part, it's truly a dark art with a lot of unknowns, far from being a science.

Do I really need that? Could I bake that other? What approximation of this is the most reasonable? If you're lucky, you work closely with artists, and great artists and art directors, with the right workflows and somehow magic happens. And some companies do seem to deliver this magic consistently. But as an industry? We're far. Both in practice and in theory, as I wrote some time ago.

There are people saying that rendering is less relevant, or even "solved" (which is crazy, as it's far to be solved even offline, even in theory). That behavior and motion are the next big challenges (and undeniably, they are becoming crucial, and if you have some time read this three, in increasing order of complexity: motion graphs, motion fields and their successor) but we are still far not only from photorealism but from proper understanding and application of the techniques we have, and from proper understanding of how to manage the creative process and how to achieve predictable levels of quality. We probably have more techniques and technical knowledge than experience and "visual taste".

I truly suspect that if we were to start an engine from the basics, forward rendering and mostly baked, a sprinkle of stable CSM shadows and a directional light, and stretch it to quality, we would already have enough tasks to figure out inputs (Textures? What resolution, what filtering. Compressed how? Normals? What are these? Specular? Occlusions?), materials (Specular and prefiltering? Fresnel, how? With Blinn? Diffuse, how? What matters, what does not work...), outputs (Antialiasing, HDR, Tonemapping*, Exposure... there are books written on tone mapping alone) and what matters were (Perception, A/B testing, biometrics? Eye tracking? LODs how etc...) to keep a rendering team busy for a couple of game cycles, write some research papers in the process and create the best visual experience of any generation...

-

*Note: It took way too long and a few articles in books to get this whole gamma/degamma thing to a "mainstream" level. Let's not wait another ten years to get the idea that HDR needs to be PROPERLY tonemapped to make any sense. And by HDR I don't mean an HDR framebuffer in post, even if you write just to a LDR 8bit target, you can still do it in the shader, right? Right? Ok. If you don't know what I'm talking about, go and read and USE this.

Photoshop scripting - Cleartype for images

Left: bilinear, Right: bilinear with "cleartype"
note- the effect is configured for a "landscape" RGB pattern LCD

I always wanted to learn how to script Photoshop (what I learned is that it's a pain and the documentation sucks...), so yesterday I started googling and created a little script to emulate cleartype on images. Here is the source (it assumes that a rgb image is open in PS):

// 3x "cleartype" shrink script

var doc = app.activeDocument;

var docWidth = doc.width.as("px");

var docHeight = doc.height.as("px");

doc.flatten();

// let's go linear RGB

doc.bitsPerChannel = BitsPerChannelType.SIXTEEN;

doc.changeMode(ChangeMode.RGB);

// now that's a bit tricky... we have to go through an action, which has binary data... which I'm not sure it will be cross-platform

// it works on Photoshop CS3 on Win7...

function cTID(s) { return app.charIDToTypeID(s); };

function sTID(s) { return app.stringIDToTypeID(s); }; 

var desc6 = new ActionDescriptor();

var ref5 = new ActionReference();

ref5.putEnumerated( cTID('Dcmn'), cTID('Ordn'), cTID('Trgt') );

desc6.putReference( cTID('null'), ref5 );

desc6.putData( cTID('T   '), String.fromCharCode( 0, 0, 1, 236, 65, 68, 66, 69, 2, 16, 0, 0, 109, 110, 116, 114, 82, 71, 66, 32, 88, 89, 90, 32, 7, 219, 0, 10, 0, 22, 0, 19, 

0, 25, 0, 58, 97, 99, 115, 112, 65, 80, 80, 76, 0, 0, 0, 0, 110, 111, 110, 101, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 

0, 0, 0, 1, 0, 0, 246, 214, 0, 1, 0, 0, 0, 0, 211, 44, 65, 68, 66, 69, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 

0, 0, 0, 9, 99, 112, 114, 116, 0, 0, 0, 240, 0, 0, 0, 50, 100, 101, 115, 99, 0, 0, 1, 36, 0, 0, 0, 101, 119, 116, 112, 116, 

0, 0, 1, 140, 0, 0, 0, 20, 114, 88, 89, 90, 0, 0, 1, 160, 0, 0, 0, 20, 103, 88, 89, 90, 0, 0, 1, 180, 0, 0, 0, 20, 

98, 88, 89, 90, 0, 0, 1, 200, 0, 0, 0, 20, 114, 84, 82, 67, 0, 0, 1, 220, 0, 0, 0, 14, 103, 84, 82, 67, 0, 0, 1, 220, 

0, 0, 0, 14, 98, 84, 82, 67, 0, 0, 1, 220, 0, 0, 0, 14, 116, 101, 120, 116, 0, 0, 0, 0, 67, 111, 112, 121, 114, 105, 103, 104, 

116, 32, 50, 48, 49, 49, 32, 65, 100, 111, 98, 101, 32, 83, 121, 115, 116, 101, 109, 115, 32, 73, 110, 99, 111, 114, 112, 111, 114, 97, 116, 101, 

100, 0, 0, 0, 100, 101, 115, 99, 0, 0, 0, 0, 0, 0, 0, 11, 67, 117, 115, 116, 111, 109, 32, 82, 71, 66, 0, 0, 0, 0, 0, 0, 

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 235, 194, 0, 1, 0, 0, 0, 1, 65, 50, 

88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 97, 15, 0, 0, 36, 77, 255, 255, 255, 232, 88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 103, 37, 

0, 0, 220, 208, 0, 0, 5, 29, 88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 46, 162, 255, 255, 254, 227, 0, 0, 206, 39, 99, 117, 114, 118, 

0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0 ) );

desc6.putEnumerated( cTID('Inte'), cTID('Inte'), cTID('Clrm') );

desc6.putBoolean( cTID('MpBl'), true );

desc6.putBoolean( cTID('Dthr'), false );

desc6.putInteger( cTID('sdwM'), 2 );

executeAction( sTID('convertToProfile'), desc6, DialogModes.NO );

doc.backgroundLayer.applyGaussianBlur(0.75); // limit the frequency a bit to avoid too many fringes

doc.resizeImage(

UnitValue(docWidth, "px"),

UnitValue(docHeight / 3,"px"), 

null, 

ResampleMethod.BILINEAR // To-do: box filter (mosaic + nearest)

);

var unitValue = UnitValue(1, "px");

// RGB pattern, note that the nearest resize will take the center pixel, that's why red shifts by one and not zero

var redLayer = doc.backgroundLayer.duplicate();

redLayer.applyOffset(unitValue, 0, OffsetUndefinedAreas.WRAPAROUND);

var greenLayer = doc.backgroundLayer.duplicate();

greenLayer.applyOffset(-unitValue, 0, OffsetUndefinedAreas.WRAPAROUND);

var blueLayer = doc.backgroundLayer.duplicate();

blueLayer.applyOffset(-unitValue*2, 0, OffsetUndefinedAreas.WRAPAROUND);

doc.resizeImage( // Resize to "select" the RGB columns in the various layers

UnitValue(docWidth / 3, "px"), 

UnitValue(docHeight / 3,"px"), 

null, 

ResampleMethod.NEARESTNEIGHBOR

);

//var col = new SolidColor(); col.rgb.hexValue = "FF0000"; redLayer.photoFilter(col, 100, false);

redLayer.mixChannels ([[100,0,0,0],[0,0,0,0],[0,0,0,0]], false);

greenLayer.mixChannels ([[0,0,0,0],[0,100,0,0],[0,0,0,0]], false);

blueLayer.mixChannels ([[0,0,0,0],[0,0,0,0],[0,0,100,0]], false);

redLayer.blendMode = BlendMode.LINEARDODGE; // add!

greenLayer.blendMode = BlendMode.LINEARDODGE; // add!

// blue is the base layer

doc.flatten();

// let's go to 8bit sRGB

doc.convertProfile ("sRGB IEC61966-2.1", Intent.PERCEPTUAL, true, true);

doc.bitsPerChannel = BitsPerChannelType.EIGHT;