Normalmaps everywhere/2

First followup to this: http://c0de517e.blogspot.com/2012/02/normalmaps-everywhere.html

Nothing smart really, it's just a small Mathematica playground I made to visualize and experiment with normalmaps, mipmaps and phong, I'm posting it in case you want to play too!

http://www.scribd.com/doc/80425295/Normalmaps-mipmaps-Mathematica-playground

Just to be clear, this is something to have fun and experiment, of course if all I wanted to do was to fit Phong (or any other rotation-invariant BRDF or part of them) I could have done it better in many other ways (first of all, by considering a single parameter that is the dot between the normal and the reflection vector/halfvector instead of working on the hemisphere). Hopefully I'll have something more "serious" coming out on the topic soo.. eventually :)

Normalmaps everywhere

You are modeling a photorealistic face. Your artists want to be able to have detail, detail detail. Skin pores. Sweat droplets. Thin wrinkles. What do you do?

If your answer is to give them high-res normalmaps and some cool, sharp specular, think twice. 

Normalmaps are not about thin, very high frequency detail! Especially NOT when rendered by our current antialiasing techniques.

Let's think about a high fidelity model for example, millions of triangles. It's too expensive to render it in realtime, but we can imagine it looking nice. It will have aliasing issues on regions of high frequency detail, like the lashes on a human face, so it requires many samples to avoid that artifact. We bake normalmaps on a low-poly base mesh, and all our problems disappeared. Right? Now it's fast and it doesn't shimmer. Cool. How? How comes that that aliasing magically disappeared? Isn't it a bit fishy?

Normalmaps: a few pixels wide detail

Normalmaps suffer from two main issues.

The first one is that they are not really that good at capturing surface variations. Imagine some thin steps, or square ridges on a flat surface. The normals in tangent space will be all "up" but hey, we had some dramatic discontinuities (steps) where are they? Of course, you lost them in the resolution of your normalmap. Now, steps are not representable no matter what obviously (discontinuity = infinite frequency = no resolution will be enough) but you might think, ok, I create a normalmap with a high resolution, create a bit of bevel around these edges so the normalmap captures it et voilà, problem solved. Right?

This introduces us to the second problem, which puts the nail on the coffin. Normalmaps don't antialias well. They are textures, so on the GPU they are antialiased by some sort of pre-filtering (i.e. mipmaps). We will get always a normal per sample (and MSAA does not give you more shading samples, only depth/geometry ones), in the end (unless we do crazy stuff in the shader that I've never seen done but maybe it would be a cool idea to try...) thus it will take little to zoom out enough to have our normals disappear and become flat again due to the averaging done by filtering. It takes actually _really_ little.

You might want to think about this a bit, and I could make lots of examples, but in the end normalmaps won't cut it, by their nature of being a derivative surface measure they take quite a bit of space to encode details, and can reproduce them faithfully only if there is no minification, so if a feature takes five texels on a normalmap, you might get good results if on-screen that feature still takes five pixels, when you start zooming out it will fade to flat normals again.

Unfortunately some times artists fail to recognize that, and they "cheat" themselves into thinking that certain features are wider than they are, because if they model them realistically they won't show. Thus, we get faces with huge pores and thick lashes or facial hair. 

Or models which are covered with detail in the three-five pixel frequency range but that does not exhibit any finer one geometry-wise (other than the geometry edges) and creates a very weird look where the frequencies do not match between colourmaps, normalmaps and geometry, almost like a badly cast plastic toy whose geometry could not hold well the detail.

It also doesn't help that this issue is largely not present in offline rendering, where proper supersampling is able to resolve subpixel detail correctly from normalmaps, thus there is some mismatch in the experience between the same technique used with different rendering algorithms.

A normalmapping example... from google images (sorry)

So what? 

How can we solve this? By now some ideas should be circulating around your head. Prefiltering versus postfiltering (supersampling). Averaging some quantities and then performing some operations (lighting) versus performing operations on the individual quantities and then averaging? These problems should be familiar, one day I'll write something about it, but always remember than a given rendering technique is not so much useful per se as it is as a part of a "generic" idea that we can apply... For example, a similar-sounding problem could be shadow filtering... and VSM...

But let's step back again a second. The problem with mipmapping basically means that far enough, normalmaps will always suck right? And what we want really is to find a mean to capture the average of the illumination all these bunch of tiny normals that we can't express in a given miplevel would have created. Some sort of "baking"...

Mhm but this averaging illumination... Isn't that what a material, a shader, and a BDRF exactly do? You know, the whole "there is no diffuse material" but really it's just that if you zoom close enough, a lot of tiny fragments in the material are all rough and oriented in different directions and thus scatter light everywhere, in a "diffuse" way... We ignore these geometrical details because they are way too small to capture and we created a statistical around them! Sounds promising. What could we use?

Well, lots of things, but two of them are _really common_ and underused, and you have them already (most probably) so why not...

Occlusion maps: pixel-sized detail

One great way to encode geometric detail, maybe surprisingly, is to use occlusion. The sharper, the smaller a depression is, the more occluded it will be.

And occlusion is really easy to encode. And it mipmaps "right", because it's something that we apply _after_ lighting, it's not a measure that we need to compute lighting. Thus the average in a mipmap will still make sense, as we don't need to transform that by a function that we can't distribute over the mipmap averaging...

Digital Emily demonstrating again specular occlusion

Also, chances are that a lot of your detail will come from the specular, as the specular more high-frequency than diffuse. And chances are that you already have a specular map around... Profit!

Ok but what if we have geometric details which cause sharp lighting discontinuities but not occlusion related. For example, round rivets on a metal tank's body. Or transparent sweat beads on a man's forehead. What could we do then?

Exponent maps: specular detail to the subpixel

The two examples I just made are convenient. Round, highly specular shapes. What does that do, lighting wise?

Photographers are known to use the reflection of the lights in the eyes to help them reverse engineering the lighting in a photo. These round shapes will then do something similar, capture the light from any direction.

And as they are highly specular, they will create a strong reflection if there is any strong light anywhere in their field of view, in the field of view of the hemisphere. This strong reflection, even if it's small, will still be significantly visible even if we look at the feature far enough not to be able to resolve, in our samples, the spherical shape itself.

So that is interesting. In a way this surface which has a high phong exponent and creates sharp highlights, when seen from far away acts as a surface with a much lower exponent, that shines if a light is "in view" from the surface and not only at very narrow viewing angle. Is this surprising even though? Really not right? We already know that in reality our materials are made by purely specular, tiny fragments, that average together to create what we model with Phong exponents of whatever other equation.

Sweat on a face though is usually better modeled, for the typical viewing distances found in videogames, as streaks of lower exponents in the material "gloss" map, instead of using normalmaps. This also completes the answer to the original question, at the beginning of the post...

Conclusion

So that's it, right? Tricks for artists to express detail, done.

Well, sort-of. Because what we said so far kinda works only at a given scale (the occlusion does not). But what if I need to express a material at all scales. What if I want to zoom close enough to see the sweat droplets, and then pan backwards? I would need a model where up close the sweat layer uses normalmaps and a high specular exponent, but far away the normals get "encoded" in the BRDF by lowering the gloss exponent instead...

Well. Yes. We can do exactly that! Luckily, we have mipmapping which lets us encode different properties of a material at different scales. We don't have to straight average from the top miplevel in order to generate mipmaps... We can get creative!

And indeed, we are getting creative. Lean and Clean, and more to the point, Clean translated into exponent maps. Or transitioning from a model to an entirely different one, for oceans or trees...


Now if you were following, you should also understand why I mentioned VSM and shadows... One idea could be to be able to extract variance from the hardware filtering operation (which is a general idea which is _always_ cool, I'll show that in another post... soon) and use that variance somehow to understand how big is the normal bundle we have to express. 


While I was chatting about all this with a coworker, he brought to my attention the work that  COD:BlackOps did (I guess I was distracted during Siggraph...), that is exactly along these lines (kudos to them, I love COD rendering)... They directly use this variance to modify the gloss amount, and to bake that in mipmaps, so we don't have to compute anything at runtime, only during mipmap generation we take the variance of the normalmaps in a given mip texel and use that to modify the gloss... Simple implementation, great idea!

But it's also a matter of educating artists, and I was really happy to see that one of our artists here proposed to drop normalmaps now that our game is rendering from a more open viewpoint, because they were not able to provide good detail (either too noisy if too little filtering or too flat), and resorted to specular maps instead for the purpose.

You might even think that you could take all your maps, and a BRDF, and compute maps and mipmaps for the same BRDF or another BRDF re-fitted to best match the lighting from the original at a given scale... And this is really what I was working on... but this article is already too big and my Mathematica example is too stupid, so I'll probably post that in a follow-up article. Stay tuned...

Sneak peek

P.S. From twitter people reminded me of these other references, which I admit I knew about but didn't read yet (shame on me!): http://www.cs.columbia.edu/cg/normalmap/index.html, http://developer.nvidia.com/content/mipmapping-normal-maps, http://blog.selfshadow.com/2011/07/22/specular-showdown/
Also an addendum, when thinking about aliasing... what about specular aliasing due to geometric detail? I actually started looking into that before thinking about this. Differential operators in the shaders could be used to evaluate curvature for example, but really I found it to be hard to have problems because of geometric normals even for the dense models I was using, you really need very high exponents, so I dropped that and went to analyze normalmaps instead.

Interview mistakes

Interviewing candidates is undoubtedly a tricky and important task for a software company who is seeking for new talents. Books have been written on the topic and while I can't call myself an expert, I do have been through the process enough times, on both ends, to have at least a good opinion of it.

There is a lot to talk about and things are different depending on if you're looking for a junior or a senior, also, a lot has been said already by people which are more competent than I am, so I would like here to focus on my personal experience as an applicant to senior/lead rendering roles, and the mistakes I see more often in the process. It's going to have some rant-ish undertones :)

In my opinion, two are the main problems to take care of:

 

Don't waste time.

Who would you rather employ? Someone who's working, has lots of competing offers, and little free time, or someone who things spending a days on your coding test is the most exciting thing happened to him that week? You want to hire the busy people. 

If you applicant has three companies which are interested in interviewing him (and even in this economy, you know well that for seniors it will happen), who do you think he will try first? The one that lets him a technical interview the day after, or the one which asks for a long test for pre-screening?

Of course your engineers are busy too and pre-screening is effective in reducing the time wasted, but be reasonable. And differentiate between juniors or newly graduates which would probably have more time on their hands and fail more often, and seniors with years of experience. A one hour test is ok, a day long one, not so much, more than that and you'll start risking to be ignored or considered last.

This though does not apply to pre-screening in general, it applies to everything really. Get to the point. Yes, warm up questions might be useful if your applicant seems nervous, but there is really no need to start with things which won't get you any useful information, if you want to chat start with more colloquial questions instead of going straight to equations or code, and bring these to the table after the more loose conversation on a given topic.

A lot of the time waste though comes from not creating different interview paths for the different engineering positions, which also brings me to the next issue:

Show some effort.

Who would you rather employ? Someone who is in love with your company or someone who wants proof that you're worth his time? An interview is usually the first point of contact between the applicant and your company. And you should assume that he is there all the time analyzing you, he's evaluating you at least as much as you are evaluating him.

I never, so far, applied to a company for which I was absolutely persuaded I wanted to work with them when I applied. There are some games that I really would have loved to make, but I find impossible from the products a given company does to understand how working for them will be, and I suspect a lot of games I love are not made in environments I would have loved to be part of. 

There are certain persons who fascinate me and who I would like to work with, but still, every time I interview I'm interviewing the company at the same time.

Put some effort, come prepared, don't waste my time. I have been asked questions which made me seriously doubt the kind of job they were interviewing me for (like once, when my whiteboard math question was about computing the distance between two points...), I've seen questionnaires with errors in them, and have been asked the same "tricky" algorithms over and over again. Nowadays, if someone asks me about polymorphism in C++ and the "virtual" keyword, it's an immediate flag that I raise.

Virtual functions? Distance between a point ad a line? Dot product? But, I hear you say, you would be surprised to know how many technical directors can't code. No I won't, but are you seriously going to think "oh, finally, a lead that knows virtual, let's hire him". 

What are you asking? Is it relevant even or are you just wasting time? Seriously you can't think of a simple function to write, or problem to solve which tests all of these things and more?

Be original! Reversing the words in a string? The building and two eggs "google" problem? Finding the missing number in a sequence? Do your homework, are you going to test if your candidate studies the list of the ten most common interviewing questions, or if he has a working brain?

Even simple variants will do better. The distance between a line and a circle instead of a point. Same thing but it avoids the mechanical "go to my dictionary of ready-made answers I've studied the night before". 

 

There are plenty of simple problems you can invent which can expand in all sorts of interesting directions, craft a few, unique for your company, for each engineering level, and maybe change them every few years.

P.S. So are there companies which did it "right"? My best experience so far was with Sony Santa Monica. Original, effective and required little time, showed effort and respect, which is not surprising knowing how smart Christer Ericson is... Crytek was also good I have to say, I rate it second because the process required way too much time, but talking with Ury Zhilinsky was awesome. The worst ones? Probably left me so unimpressed that I won't remember the companies.

Replacing google share: zootool

There are quite a few bookmarking services on the web today, and I never felt the need to find one until Google shut down its handy reader share feature. So, kinda randomly I picked Zootool and I've been working on sharing some links for a few weeks, so now there is a bit of content and I can publish its feed. Let me know if there are problems, otherwise I'll stick to this one to share my finds in the "rendering et al." category, at least until Reeder does not add support for Google+ :)

http://zootool.com/feeds/user/c0de517e/type:all

Prototyping frameworks (rendering)

Every now and then, I look around for prototyping frameworks for my rendering work. I always end up with very little but maybe I'm too picky or too lazy. Here are some I've found:

Stuff I actually use:

• FXComposer, both 2.5 and 1.8 (Nvidia does not host it anymore, so you have to google it) as they both crash/have problems in different ways. In particular, 2.5 seems to have problems clearing rendering targets (I use a pass with ztest always to clear), 1.8 crashes in some situations. 1.8 is also nicer for a programmer, but 2.5 is fairly usable. I use SAS and NVidia has some documentation (finally!) about it, to script render passes. In theory both also support proper scripting, but the documentation is thin. A few times when I wanted to look inside FXC 2.5 I used something like ILSpy or .net reflector to delve in the undocumented parts (that's to say, almost everything).
• Wolfram's Mathematica. I wrote a couple of articles on this blog about it, it's great and I love it, I love the language, it's not what you would expect if you're a mathematician but for a programmer is pretty neat (well at least, if you like lisp-ish things, which you should, syntax apart)
• Python/IPython (I like the Anaconda distribution) is a good alternative to Mathematica. I still use Mathematica most of the times and I'm not a Python expert, but I've done a few experiments with it.
• SlimDX or SharpDX, to tell you the truth I mixed them up a few times, the names are similar. Bottom line, a DX wrapper for C#, I love C#. SharpDevelop if I don't have an updated visual studio which supports the last .net framework.
• Processing. I wrote an article on the blog about using it with Eclipse for live-coding, it's neat, it's simple, it has a ton of libraries at this point, even to do 3d stuff and shaders but I use it mostly for 2d prototypes.
• ShaderToy. There are a ton of offline programs that offer similar functionality, even on iPad, wherever, it's very popular. But. Having it online is nifty for some quick tests. Unfortunately crashes often on certain browsers/computers (the most common issue is for large shaders to take too long to compile, making WebGL think something's wrong). There are also lots of alternatives (kickJS editor, Shdr, GLSL playground and SelfShadow's playground), some more powerful (WebGl playground which also supports three.js), but ShaderToy is the most popular.
• 3D Studio Max. It has an horrible support for shaders (at least it used to, and I suspect not much has changed since) and I never loved it (I love Maya even less though), but I used to know it (six years ago or so) and know maxscript, so I ended up prototyping in Max a few things. It can be handy because you can obviously manipulate meshes all the ways you want and define vertex streams and visually paint attributes on meshes. You can't really control the rendering passes though, so doing non-surface shaders or stuff other than the most basic post-effects is hard. Nowadays I don't use it much if at all.
• Pettineo's framework. Comes with all his sample projects and it's a great, simple, well written C++/DX11 framework, very easy to toy with. I have my own fork with some improvements.
• Jorge Jimenez demo framework - as Jorge is a coworker of mine, I have access to his latest version

Seem promising:

• PhyreEngine, if you have access to the Sony stuff... Might be a bit overkill as it's a fully fledged engine, so the learning curve is not so steep per se but there are tons of examples.
• Microsoft/DirectX MiniEngine. Quite nice! Also NVidia made Falcor, a "research" framework, but currently it's only OpenGL which is fairly sad (even if understandable as lots of HW extensions come out for OGL first...)
• Bart Wronski's C# framework. A solid alternative to MJP's, with the added bonus of being C# code.
• Karadzic's BGFX wraps Dx9, 11, 12, OpenGL, GL|ES and Vulkan! It's a bit higher-level than any of these APIs, providing a draw-centric model where draws are sorted on a per-draw key. Neat, even if I don't necessarily care much about being cross-platform while prototyping.
• ReedBeta's DX11 framework.
• Threejs.

Some other alternatives:

• Erik-Faye Lund published the sources of his "very last engine ever" which is used in a bunch of great demos (as in demoscene) I didn't have the time to look into it much yet, but the name sounds great!
• Hyeroglyph 3, it's the 3d engine that "ships" with the Practical Rendering and Computation with DirectX11 book (which is nice). It still has a bit more things that I'd like to (more of an engine than a framework) but it's nice. 
• Matt Fisher BaseCode could be handy for some kind of experiments.
• Cinder looks still a bit young, it has many nice things but it lacks some other which I would consider "basics". I feel the same about openFrameworks and to me Cinder looks nicer. Plus I don't love C++ that much, and Cinder depends on Boost which is a huge turn off :)
• Humus Framework 3. This is great, it's simpler than a full fledged engine, it's easy to read and it has tons of examples and Humus is notorious for his graphic demos, which all come with sourcecode and were made with his framework!
• Intel's Nulstein.
• VVVV. It's a node-based graphic thingie. Which would seem like the least suitable thing for rendering prototypes, but it supports shaders, and it supports "code" nodes where you can write C#, so it might be worth a try...
• OpenCL Studio, I used to use this for experimentation, but it seems abandoned, sadly.