Search this blog

03 June, 2014

Rate my API

Metal, Mantle, OpenGL's ADZO, GL|ES, DirectX 12... Not to mention the "secret" console ones. It's good to be a graphics API these days... And everybody is talking about them.
As I love to be "on trend" now you get my take on all this from hopefully a slightly different perspective.

 To be honest, I initially wrote an article as a rant in reaction to the excellent post on modern OpenGL by Aras (Unity 3d) but then after Metal and some twitter chats I became persuaded I should write something a bit more "serious". Or at least, try...

- When is a graphics API sexy?

Various smart people are talking with nice detail about the technical merits of certain API design decision (e.g. Ryg and Timothy's exchanges on OpenGL: original, reply, re:re: and another one) so I won't add to that right now. I want instead to cast these discussion in a different and to me more relevant point of view. What do we really want from a API (or really any piece of software)? 

First and foremost will consider to adopt a technology if it's useful. It might seem obvious but apparently it's not. How many times have you seem projects that don't really work, yet spend time on aesthetic improvements?

Ease of use, documentation, great design, simplicity. All these attributes are completely irrelevant if the software doesn't do some compelling work. We can learn undocumented stuff, we can write our own tools, we are engineers and if there is something we need and there is a road open to obtaining it, we can achieve what we need. Of course we'd rather prefer not to endure pain, but pain is better than just not being able to do what we need to. Easy is better than hard, but hard is better than impossible.

Of course after we have something that does something we could be interested in, then if we'll adopt it or not depends on how much we want it divided how hard it is to achieve it. Cost/Benefit, unsurprisingly. To recap we want an API that is:
  • Working. Is actually implemented somewhere, the implementation actually works. If it's written on paper but it's not reliably deployed, we can safely ignore its existence. This is actually part of "useful" or of the benefits, but it's important enough I'd like to remark it here.
  • Useful. Does something that we need, in a market we're interested in. If I'm a AAA company and you make a great API that enables incredible graphics on a device that sold ten thousands units, that's not useful. If you provide a big speed improvement on a platform that is not performance bounds on my products, that's not so useful either. And so on.
  • Easy. Do I need to change my entire engine or workflow to adopt this API? That's the most pressing question. Then it comes documentation and support. Then tools. Then in general how nice the API design is. APIs usually work in a realm that is well-separated from the rest of the software, if your API requires to sacrifice a (virtual) goat each time I have to call it, it's probably still not going to make all my project bad, it's not going to "spread". If the bad API design "spreads" to the engine or the entire software then that's changing the workflow, so it goes back to the first, most important attribute.

Now we can go through a few graphics API on the table these days and see how they fare (in my humble opinion) according to this (obvious but sometimes forgotten) metric.

- OpenGL and AZDO

OpenGL has a long history, once upon a time was winning the graphics API war, started to lose ground and by the time DirectX9 was around, pretty much all games switched (a good history lesson was posted a while ago on stackoverflow).
That didn't stop the downward spiral, to the point that around the time DirectX11 came (2008, shipped with Windows 7 in 2009) even multi-plattform CG software (Maya, Max and the likes) moved to DirectX on Windows as the preferred frontend.
OpenGL  took years to catch up with a variety of patches to DirectX11 (g-truc reviews are awesome) and even longer to see robust implementation of these concepts. Still today the driver quality and the number of extensions supported varies wildly across vendors and OS (some examples here), ironically (and to make things worse) the platform where OpenGL has the best drivers across vendors today is Windows (that though doesn't even ship by default with OpenGL drivers but only an ancient OpenGL 1.1 to Dx layer) while OSX which is the best use-case for OpenGL in many ways, has drivers that tragically lag behind (but at least they are guaranteed to be updated with the OS!).

But, for all the faults it has, today OpenGL is offering something very worth considering, which is what cool people call AZDO (instance rendering on steroids): a way to reduce draw-call overhead by orders of magnitude by shifting the responsibility of working with resources from the CPU, generating commands that set said resources into the command buffer, to the GPU, that in this model follows a few indirections starting from a single pointer to tables of resources in memory.

To a degree AZDO is more a solution "around" OpenGL, rather than fixing OpenGL by creating an api that allows fast multithreaded command buffer generation, it provides a way to draw with minimal API/driver intervention.
In a way is a feat of engineering genius, instead of waiting for OpenGL to evolve its multithreading model it found a minimal set of extensions to work around it, on the other hand this probably will further delay the multithreading changes...

Results seem great, the downside of this approach is that all other modern competitors (DirectX12, Mantle, XBox One and PS4 libGNM) allow both to reduce CPU work by offloading state binding to GPU indirection and support fast CPU command buffer generation via multithreading and lower-level concepts, which map to more "conventional" engine pipelines a bit more easily. There is also a question about if the more indirect approach is always the fastest (i.e. when dealing with draws that generate little GPU work) but that's yet up to debate (as AZDO is very new and I'm not aware of comparisons pitting it against the other approach).

For AAA games. Today for most companies this means consoles first, Windows second, anything else is much less important. For these games having more performance on a platform that is not the primary authoring one and that is not often a performance bottleneck, at the cost of significant engine changes doesn't seem attractive at all (and with no debug tools, little documentation and so on...), especially considering that DirectX12 is coming, an alternative that promises to be as good but easier, better supported and that will also target Xbox One, thus covering two of the three target platforms.

A notable exception though are free-to-play games hugely popular in Asia that are not only usually Windows exclusive, but where Windows XP is still very relevant, which means no DirectX11 and even less DirectX12. For these games I guess OpenGL could be a great option today.
Note also that AZDO is currently not fully supported on Intel hardware (no bindless, MDO software emulated) so you'll probably need a fallback renderer as well, as Intel hardware is quite interesting for games at the lower end.

For applications. Most CGI applications are the worst-case scenarios for GPU efficiency, they tend to do lots of draws with very little actual work (wireframe drawing, little culling) and in not very optimized ways as well due to having to work with editable, unoptimized data and often also carrying legacy code or code not thought to achieve the best GPU performance. 
Also, shipping on multiple platforms is the norm while working across multiple vendors is less of a concern, NVidia has the golden share among CG studios and Intel is completely out of the picture, even only NVidia/Linux is probably a compelling enough target to consider "modern OpenGL there" and even more as Windows would benefit as well.
These things considered I would expect modern OpenGL to be something most applications will move towards, even if it might be a significant effort to do so.

Some more links:

 - Mantle

AMD's Mantle is an clear example of a nice, good, easy API (exaggerated, but interesting praise here) that fails (in my opinion) to be really useful for shipped titles. On the technical level there's nothing to complain, it seems very reasonable and well done. 

For AAA games. Today Mantle works only on Windows with AMD hardware. That's a bit little, then again especially when DirectX12 is coming and AZDO is an alternative too. While it's most probably easier to deploy than AZDO (and I bet AMD is going to be willing to help, even if right now there might be no tools and so on), is also much less useful. Worse even if you consider that even on AMD hardware only certain CPU/GPU combinations are CPU limited.
It simply covers too little ground, I hoped at the beginning that AMD would come out sooner and with a PS4 layer as well, thus getting deployed by many projects that were looking for an easier way to target PS4 than figuring out libGNM. It didn't happen and that I think is the end of it. Some people were thinking it could have been a new cross-vendor standard, but it will -never- happen.
They did though score with Frostbite's support which pretty much means all EA games. But I would be very surprised if they didn't have to pay for that, and wonder how long it will last (as it's still a cost to support it, as it is supporting any platform)...

For applications. It's a bit more interesting there, as if you remove the consoles from your target then you're increasing the surface occupied by Windows. Also it's not unreasonable to think that Mantle could be ported on Linux. Unfortunately though NVidia is more popular than AMD for CG studios and that pretty much kills it.

For the people. There is something thought that needs to be praised a lot: AMD also has lots of great, public documentation about the working of its GPUs (Intel is not bad as well, NVidia is absolutely terrible, a sad joke) and tools that show the actual GPU shader working (i.e. shader disassembly) which is really great as it allows everybody to talk and share their findings without fearing NDAs.
This creates a positive ecosystem where everybody can work "close to the metal" and Mantle is part of that. Historically it just happens that the more people are able to hack, the most amazing things get created. See what happened after twenty years of C64 hacking (some examples here).
I expect all graphic researchers to focus on GCN from now on.

- DirectX12

It's hard to criticize DirectX11, especially if you consider that it was presented in 2008 and what was the state of the other APIs at that point. It changed everything, mapping better to modern GPU concepts, introduced Tessellation and Compute Shaders, looks great and easy, is reasonably well documented and supported, and it's very successful.

Arguably DirectX9 had better tools  (VSGD is horrible AND they killed Pix that was actually working fine), but that's hardly a fault of 11 and rather due to the loss of interest in PC gaming, nowadays things are getting much better. Consider that only now we're starting really to play with Compute Shaders for example, because next-gen consoles arrived, but we had them for five years now! It was so ahead of time that it needed only rather minor updates in 11.1 and 11.2.

The only, big issue with 11 is that Microsoft wants to make things simpler than they really should be, for no great reason. So 11 shipped with certain "contracts" in its multithreading model that don't seem really useful or needed but hugely impacted the performance of multithreaded drivers to the point where multithreading is useful only if your application and not the driver is the bottleneck. 
If your code is fast enough, multithreaded Dx11 will actually be slower than single-threaded, which is clearly an issue. I suspect it could still be technically possibly to carve "fast paths" for applications swearing not to exercise the API in certain ways but probably it was simply not important enough for the PC gaming market and now 12 is coming, probably just in time...

For everything Microsoft. DirectX won on windows and it also ships on Microsoft consoles. I can't comment much on 12 and it's not finished yet. Hardly it will be displaced on Windows though, especially for games.

- Metal

Metal is Apple's Mantle. On my very personal biased poll from the reactions I've read on my twitter feed, it has not been received with the same enthusiasm as AMD's initiative. Some explained to me it's because Mantle promised to be a multi-vendor API while Metal didn't. Oh Apple, outclassed at marketing from AMD, you don't know how to appeal the engineers, next time say it's designed to be open...

I've also seen many people complaining this is foul play designed only to create vendor lock-in, a mere marketing move. I don't agree, and if you think it's only marketing then you should prove that's possible to write an equally fast driver in today's OpenGL|ES.
I believe that's not technically possible and I believe OpenGL|ES is plagued by many of the same defects of desktop OpenGL, only much worse as it has no AZDO and it ships on platforms that are very resource constrained, so where performance and efficiency matters even more!
It would have probably been possible to carve fast-paths and patch ES with proprietary extensions that would have been a bit more friendly to the ecosystem (extension often get incorporated into the standard down the line), but if it reaches the point where most of the rendering would have gone through extensions what's the point, really?

Actually this might be for the best even for the overall ecosystem, as it's a bigger kick-in-the-nuts than everything else could have been, and when many vendors on Android are shipping drivers that are just the -worst- software ever and Khronos shows to be slow to evolve and ridden with politics, a hard kick is what's most needed.
It's very new as we speak and I haven't had an in depth look into it, so I might edit this section later on.

For games. iOS has still the golden share of mobile gaming, with many more exclusives and games shipping "first" of that system than the competitor, but the gap is not huge. Also, most games are still 2d and not too demanding on the hardware, so for a lot of people a degree of portability will matter more than a magnitude improvement in drawcall performance. 
But, for the games that do care about performance, Metal is just great, iOS is big enough that even if your game is not exclusive, it's very reasonable to think about spending money to implement unique features to make your game nicer on it. 
It's true that Metal won't be available on older Apple hardware but Apple has always succeeded in giving people reasons to update both their software and their hardware, so that's not probably a big concern.

 - Conclusions

 Learn AZDO, play with Mantle, ship with DirectX.

 If you're doing an indie title do use a rendering library or engine (I keep pointing at https://github.com/bkaradzic/bgfx but it's just an example) so you'll still ship with the best API for each platform and with the least amount of headaches. If you really love toying with the graphics API directly then I guess a flavor of OpenGL that is supported across platforms could be nice (3.3 if you care about Intel/Linux right now).

 If a market is interesting enough for a given application and the vendors there decide on their own API, like it's happening for Metal and happened for DirectX, I'd welcome that.

 The problem with many of the APIs we're seeing is not that they divide the market, but that they try to do so in segments that are too small and uninteresting to specifically target. If for example Linux decided on its own 3d API for games I doubt that would be at all interesting...
If AMD shipped Mantle on consoles and PC then it could have been big enough of a segment to target, PC-only is not. If NVidia GameWorks offered a compelling solution on consoles, guess what, it would see a bigger adoption as well, while right now I suspect it will be used only on projects where NVidia is directly involved.

 Most projects already have to ship with an abstraction layer of sorts, many of these are available, in practice the idea of using OpenGL directly to ship products across platforms doesn't exist (except for very small projects and some research code).
It's always best to have to write (or use via third-party libraries) lower-level code on things that we understand that have to fight with very opaque, wildly different implementations of a supposedly standard API. 

 In fact I bet that practically no (a very tiny number) gamedev knows even the basics of what a driver does and why certain API decisions led to slow CPU performance. Also the number of people not using third-party game engines especially for indie work is dwindling.

 In theory a single API is better, in practice, today, it isn't and that's why the emergence of these low-level libraries is not just a marketing plot but actually a reasonable technical solution.
29 comments:
Etichette:

17 May, 2014

Photographic inspiration

If materials, meshes, textures and lighting were "solved", what would be all the focus of (realistic) environment visuals? 

Architecture, photography, scenography. And the technology to serve these better.

Should we emulate cinema and still photography or does photography in a virtual world mean something different? Is the interactivity a possibility to create techniques that we don't even have in CG movies?

I don't know, but it's wise to know the past while thinking of the future.

When it comes to environments, I love the American cityscapes depicted by the early adopters of color photography. It's a feeling that is hard to describe, estrangement maybe, probably an effect that is amplified on me as I grew up in the super dense Southern Italian cities, where nothing has an end and people build all in the little space available, from the sea up to the Vesuvius.

 Who are your favourite photographers?

Trent Parke
Richard Misrach
Gregory Crewdson
Todd Hido
Joel Meyerowitz
William Eggleston
Stephen Shore
No comments:
Etichette:

26 April, 2014

Smoothen your functions

Do you have an "if", "step" or such? Replace with a saturate(multiply-add(x)).
Do you have a mad-saturate? Replace with a smoothstep.
Do you have a smoothstep? Replace with smootherstep...

Ok, kidding, but sort-of, I actually do often ed up replacing ramps (saturate/mad... the fairy dust of shading, I love sprinkling mads in shader code) instead of steps, I remember years ago turning a pretty much by-the-book crysis 1-style SSAO into a much better SSAO by just "feathering" the hard in/out tests (which is kinda what line sampling SSAO does btw).

If you think about it, it's a bit of "code smell". What shading functions should be discontinuous? True, most lighting has a max or saturate right? But why? Really we're considering infinitesimal lights, for physically realistic lights we would have an area of emission, and that area would be fractionally shadowed by a surface, so even there, the shadowing function wouldn't just be a step of the dot product. This might not be evident on diffuse, but already when you're trying to use half-angle based specular attention has to be taken when handling transition to the "nightside".

And of course even when reasonable, any "step" function (well -any- function!) in a shader should be anti-aliased... And of course everybody knows what's the convolution of a step with a box (pixel footprint) is... Texturing and Modeling, a Procedural Approach is the canonical text for this, but it's funny, googling around one of the first hits is this documentation page of Renderman on antialiasing, whose slides are horribly aliased. The OpenGL Orange Book also has examples, and I really want to mention this IQ's article on ray differentials even if it doesn't do analytic convolution...

Many times the continuity of derivatives is not that important (visible), that's why we can use saturated ramps (discontinuous in the first derivative) or saturated smoothsteps (discontinuous in the second), with the big exception of manipulating inputs to specular shading. In that case, even second-derivative discontinuities can very clearly show, thus the need of the famous "smootherstep".

Anyhow. I usually have a bunch of functions around to help with ramps, triangle ramps, smoothsteps and so on, most of them are trivial and can be derived on paper in a second or so. Lately I had to use a few I didn't know before, so I'll be writing them down here.

Yes, all this introduction was useless. :)

- Smooth Min/Max

log(pow(pow(exp(x),s) + pow(exp(y),s),1/s))

This will result is a smooth "min" between x and y for negative values of s (which controls the smoothness of the transition), "max" for positive values.

For s=-1 this results in the "smoothest" min:

log(exp(x+y)/(exp(x)+exp(y))

If you know that x,y are always positive a simpler formulation can be employed, as we don't need to go through the exponential mapping:

pow(pow(x,s) + pow(y,s),1/s)


Note also that if you need a soft minimum of more than two values, your expressions simplify, e.g. pow(pow(pow(pow(x,s) + pow(y,s),1/s),s)  + pow(z,s),1/s) = pow(pow(x,s) + pow(y,s)  + pow(z,s) ... ,1/s).

 Note also the link between softmax and norm-infinity.

 - A few notes on smoothsteps

Deriving smoothstep and smootherstep is trivial, just create a polynomial of the right degree (cubic or quintic) and impose f(0)=0, f(1)=1 and f'(0)=0, f'(1)=0 (and the same for f'' in case of smootherstep), solve and voila'.


Once you do that, it's equally trivial to start toying around and derive polynomials with other properties. E.g. imposing derivatives only at one extreme:


You can have a "smoothstep" with non-zero derivatives at the extremes:


Or a quartic that shifts the midpoint:


It would seem that the more "properties" you need to have the higher degree polynomial you need to craft. Until you remember that you can do everything piecewise...
Which is basically making small, specialized splines. For example, a quadric smoothstep can look like this:


This is helpful also because there are certain tradeoffs based on applications, especially as having continuous derivatives don't mean automatically that it will be nice looking...
You can make functions that impose more and more derivatives (and do you know that smoothsteps can be chained? smoothstep(smoothstep(x))...) but that doesn't mean the derivatives will "behave", as they can vary wildly in the domain and result in visible "wobbling" in shading.


Another thing that you might not have noticed is how close smoothstep is to a (shifted) cosine, I didn't before a coworker or mine, the all-knowing Paul Edelstein, mentioned it. Probably not too useful, but never know, in certain situations it might be applicable and cheaper.


- Sigmoid functions

Another class of functions that are widely useful are sigmoids, "s shaped functions"

Smooth Sigmoid: x/pow((pow(abs(x),s)+1),1/s)
Logistic: 1/(1+exp(-x))

Sigmoids are similar to smoothsteps, but usually reach zero derivatives at infinity instead at 0,1 endpoints.


They make nice "replacements" for "step" as they approach nicely their limits as they go to infinity:


But also for saturated ramps, especially the smooth sigmoid as it has f'(0)=1 as we have shown before.


Another sigmoid is the Gompertz function, which has nice and clear parameters:

asymptote*exp(-displacement*exp(-rate*x))

Beware though, it's not symmetric around its midpoint:


 There are a ton more, but I'd say not as generic. If you look at the various tonemapping curves, most of them are sigmoids, but most of them are in exponential space and not symmetric.
In fact at a given point I made tonemapping curves out of sigmoids, piecewise sigmoids or other weird things glued together :)

- Bias and Gain (thanks to Steve Worley for reminding me of these)

 Bias pow(x,(-log2(a))
Gain if x < 0.5 then 0.5*bias(2*x, a) else 1-0.5*bias(2-2*x, a) 

Schlick's Bias x/((1/a-2)*(1-x)+1)
Schlick's Gain if x < 0.5 then SBias(2*x,a)*0.5 else 1-0.5*SBias(2-2*x,a))

 Bias is just a power (-log2(a) only maps 0...1 to the power), and Gain maps one power next to a mirrored copy around the midpoint, the easiest way you can construct a piecewise sigmoid (without imposing conditions on the derivatives and so on).

 Schlick's versions were published in Graphics Gems IV, and are not only an optimization of the original Bias/Gain formulas (credited to Perlin's Hypertexture paper), but are symmetric over the diagonal, which is a nifty property (it also means that for parameter a the inverse curve is given by the same formula with 1-a)

- Smooth Abs

 Obviously if you have any "smoothstep" you can shift it around zero to create a "smoothsign" and multiply by the original value to get a smoothed absolute. The rational polynomial sigmoid works quite well for that:
SmoothAbsZero d*x*x/sqrt(1+d*d*x*x)

If you don't need to reach zero at x=0 then you can simply add an epsilon to the square root of the square of your input, yielding this
SmoothAbs sqrt(x*x+e)


And that's all what I have for now, if you encountered other nifty functions for modelling and tinkering with procedurals and so on, let me know in the comments! 
I'm always looking for nifty functions that can be useful for sculpting mathematical shapes :)

 - Bonus example: Soft conditional assignment


 Some links:
8 comments:
Etichette:

08 April, 2014

How to make a rendering engine

Today I was chatting on twitter about an engine and some smart guys posted a few links, that I want to record here for posterity. Or really to have a page I can point to every time someone uses a scene graph.

Remember kids, adding pointer indirections in your rendering loops makes kitten sad. More seriously, if in DirectX 11 and lower your rendering code performance is not bound by the GPU driver then probably your code sucks. On a related note, you should find that multithreaded command buffers in DX11 make your code slower, not faster (as they can be used to improve the engine parallelism but they are currently only slower for the driver to use, and your bottleneck should be the driver).


Links below are all about the idea of ditching state machines for rendering, encoding all state for each draw and having fixed strings of bits as the encoding. I won't describe the concept here, just check out these references:

Some notes / FAQs answers. Because every time I write something about these system people start asking the same things... I think because so many books still talk about "immediate" versus "retained" 3d graphic APIs and the "retained" is usually some kind of scenegraph... Also scenegraphs are soooo OOP and books love OOP.
  • Bits in the keys are usually either indices in arrays of grouped state (e.g. camera/viewport/rendertarget state, texture set, etc...) or direct pointers to the underlying 3d API data structures
    • So we are following pointers anyways, aren't we? Yes of course, but the magic is in the sort, it not only will help minimize state changes but also guarantees that all accesses in the arrays are (as-)linear(-as possible)!
    • Of course if you for example sort strictly over depth (not in depth chunks), then you have to accept to jump between materials at each draw, and the accesses over these might very well be random.
      • If that's the case try to avoid indirections for these and store the relevant data bits directly in the draw structure.
      • Another solution for this example case is to sort the material data in a way that is roughy depth coherent, i.e. all materials in a room are stored near each other. In theory you could also dynamically sort and back-patch the pointers to the material data in the game code, but we're getting too complex now...
    • The same can't be guaranteed for resource pointers (GL, DX...), even if the pointers will be linearly ordered they might be far away in memory, that's unavoidable. On consoles you have control on where resources are allocated even for GPU stuff so you can pack them together, but even more importantly you can directly store the pointers that the GPU needs w/o intermediate CPU data structures
  • You don't need to have a single array of keys and sort it!
    • Use buckets, i.e. some bits of the key index which bucket to use. Bucketing per rendertarget/pass is wise
    • "Buckets", a.k.a. separate lists. In other words don't be shy to have a list per subsystem, nobody says there should be one solution for all the draws in your engine.
      • This is usually a good idea also because draw-emitting jobs can and should be sequenced by pass, e.g. in a deferred renderer maybe we want first a rough depth-prepass, then g-buffer, then shadows... These can be pulled in pass-order from the visibility system
      • Doing the emission per pass means we can kick the GPU as soon as the first pass is done. Actually if we don't care for perfect sorting, and we really care about kicking draws as soon as possible, we can even divide each pass in segments and kick draws as soon as the first segment is done.
      • I shouldn't say it but just in case... These systems allow to generate draws in parallel, obviously, and also to sort in parallel and to generate GPU commands in parallel, quite easily. Just keep lists per thread, sort per thread, then merge them all (the only sync point) then split in chunks and per thread create GPU command lists (if you have an API where these are fast...)
  • You don't need to use the same encoding for all keys!
    • Some bits can decide what the other bits mean. Typically per rendertarget/pass you need to do very different things, e.g. a shadowmap render pass doesn't need to care about materials but might want to use some more bits as a z-key for depth sorting
    • Similarly, you can and should have specialized decoding loops
  • Not all the bits in the key need to be used for sorting
    • Bits of state that directly map to the GPU and don't incur in overheads from setting them, should not be part of the sorting, they will just slow it down.
  • Culling: make the keys be part of the visibility system
    • When a bounding primitive is finally deemed to be visible, it should add all the keys related to drawing its contents
    • At that point you want also to patch in the bits related to the projected depth, for depth sorting
  • Hierarchical transforms
    • Many scenegraphs are used as a transformation hierarchy. It's silly, on most engines a tiny fraction of objects need that, mostly the animation/skinning system for its bones. Bones do express a graph, but it's not enough of a reason to base your -entire- rendering system on it.
  • Group state that is (almost) always set together in the same bits
    • E.G. instead of having separate bits (referring to separate state structures) for viewport, rendertarget, viewworldprojection constant data and so on, merge all that in a single state structure.
  • Won't I need other rendering "commands" in my list? Clears? Buffer copies? Async CPU jobs waits? Compute shaders? Async compute shaders...
    • All of these can be part of "on bind" properties of certain parts of the state. E.G. when the bits pointing to the rendertarget/pass change we look up in that state structure to see if the newly set rendertarget have to be cleared
    • In practice as you should "bucket" your keys into different arrays processed by different decode loops, these decode loops will know what to do (e.g. the shadowmap decode will make sure the CPU skinning jobs are finished before trying to draw and so on)
  • Are there other ways?
    • Yes but this is a very good starting point...
    • Depends on the game. A system like this is good when you don't know what draws you'll have, typically because they come from a visibility system which can't spit them in the right order and/or because of parallel processing.
    • Games/systems where you can easily generate GPU commands in the right order and you exactly know which state changes are needed, obviously can sidestep all this architecture. E.G. Fifa, being a soccer game, doesn't need to do much visibility and knows exactly how each player is made in terms of materials, thus the code can be written to exactly process things in the right order... Something like this would be reasonable for Frostbite, but you won't use Frostbite for Fifa...
2 comments:
Etichette:

26 February, 2014

Valve VR. I want to believe.

Premise: Spoiler alert, I guess. Albeit it might not matter, it's worth noting that if you're going to experience either Valve's or Rift's Crystal Cove demo anytime soon (GDC perhaps) you could want to consider approaching it without any preconceived notion that this or other articles might give. Also, lengthy as usual.

Introduction
Valve fanboy, I am not. In fact I might say I still hold a grudge against it, I want Valve to make games, not stickers to slap on PCs in a marketing stunt with the hope of moving some units of a platform in general decline. I understand that Steam makes money, but on a personal level I can't really care...

So this morning, after waking up way too early for my habits I arrive at Valve's building I can't help muttering "so cheap" as I look for the street number and make sure that I'm in the right place. Of course it's silly of me as money is not the reason why they didn't bother to put a logo on the facade nor the name on the directory, but anyhow I digress... The "I" in the titular "I want to believe" here refers not to my faith, but to how my brain reacted to the system.

Now, it's hard to put in words an experience, especially something novel as this is, as there aren't easy analogies to make and frames of reference in most people's experiences to anchor to. I'll try my best to explain how it works, or how I think it works.

Visuals are a tricky beast. Rendering engineers are often deeply embedded in very technical concerns, but at the end of the day, what really matters is psychology. What happens, when stars align and things go right (because, mind you, we're really far from making this a science and the little science there is most of the times is still very far from mingling with entertainment) is that we create a visual experience that somehow tickles the right neurons in our brain to "evoke" a given atmosphere, sensations, feelings that we learned in real life and are "replayed" by these stimuli. 
Mostly for me that happened with environments, I guess because we're really discriminating when it comes to characters: Call of Duty, Red Dead Redemption, Kentucky Route Zero are all great examples.

When we try to achieve this via photorealism we hope that by simulating what's perceptually important, what we can notice in a scene, the light, the materials, the shapes and so on, we reach a point where our brain accepts the image we generated as a reproduction of reality, like a photo. And we hope that our artists can use this tool that gets us close to reality to more easily be able to fool our brains into producing emotions, because we're nearer to the paths that normally fire when we experience things in the real world.

Inside Virtual Reality
A good VR experience completely sidesteps all this, Abrash says it right, when things align in VR you get presence and it's an infinitely more powerful tool. Presence is the sensation that you are there; That virtual wall is at ten meters from you. And it is really unquestionable. Realism doesn't matter anymore.
The VR prototype suffered from all kinds of defects: it's clearly "low resolution", a lot of demos didn't have lighting, most did do only lightmaps or diffuse, not specular, most had no textures, I could see banding and even some aliasing, I could spot errors in the lightmaps, even quite clearly the ghosts from the OLEDs and so on and on. A nightmare for a rendering engineer, on a 2d screen you would have said the worst visuals ever.
Yet you were fooled into thinking you were there, not through realism but through the immersion that is possible with the low-latency, full head tracking (rotation AND position) stereo sauce Valve has implemented. I suspect there are a million things to get just right, we discussed how OLEDs provided enough dynamic range that you didn't question the scene much for example and they have a catalog of things that are crucial and things you can ignore.

The most succinct way to describe this is that in all medias before this, at best you had the impression of looking at a greatly detailed, technologically advanced reproduction of reality (think of the best immersive stereo projection movie you've ever seen for example). Here even when you're looking at really basic renderings you think you are in some sort of weird, wrong alternate world, similar if you wish to certain installations, rooms that play with light and shapes to create some very unusual experiences.

The demo environments Valve created (or actually I should say, I witnessed) were quite tame. Clearly they didn't want to push it to avoid certain people reacting negatively to the experience. Most of the time the scene was static, not in the sense of devoid of animation but as in a fixed room you could move in but that wasn't moving relative to you. Things never got scary and I didn't interact in any way with the simulation (even if I, in numerous times, instinctively went reaching with my hands to objects and avoided objects getting to near me), yet there were some intense moments.

I guess a few people now described a scene where you're in the simplest room possible, no shading, no lighting, yet you start on a small ledge and you can't avoid feeling vertigo and have to actively force yourself to step into the "void". You know it's not real, at an intellectual level. You have all the possible visual hints saying it's not real, yet your brain tells you otherwise.
Another, switching for the first time to a scene with some animated robots, at the moment of the switch I had a second of high alertness, as your primitive brain steps in and rushes to prepare for a suspicious activity.
The weirdest sensation was at the end, flying (very slowly, as apparently motion creates quite easily discomfort) through CDAK. There the visuals were distorted enough (see the youtube video linked) that I didn't feel as much presence (so there are some extreme cases where visuals can break it), yet when some of the weird blue objects passed through me I had, again for a split second, a sensation that I could only later rationalize as reminiscent somehow of being in a sea, I guess because that's the only thing I have in my experience of going through something like that.

Practicalities
When does presence break? Visuals can be pushed quite far before they break presence. Mind you, rendering will be a huge issue there and good rendering I am sure does make a difference to remove the idea that you are immersed in something quite odd, but again it simple visuals don't break immersion. I would also have loved to see a scene with and without various rendering effects (visual hints) but alas, no such luck.
Even the low resolution and blur and ghosts are to me defects of my vision, like seeing through glasses or through a dirty motorbike helmet, not a problem of the "reality" of the scene. Impossible behaviours do. In one of the scenes for example there were some industrial machines at work behind a glass wall. Poking my head through the wall into the room breaks it. 
It's not like suddenly losing stereo, as in one of the cross-eyed stereograms where you focus on the page and lose the effect. The closest analogy I can think of is Indiana Jones' leap of faith, and you might have experienced something similar with some visual tricks in theme parks: you realize it's an illusion.

There are a myriad of ways of doing something that breaks the presence, many things that are totally acceptable on a normal display are intolerable in VR. You might know that you can't really use normal-maps or any other non-stereoscopic hints of detail for example, but you are also much more aware of errors in animation, physics, not to mention characters (which weren't demoed at all).
And it's good if the consequence of an error is only bringing back to the idea you're in a VR helmet, the bad cases are when certain hints are very strong, but certain others are completely wrong or missing, like falling without acceleration, wind and so on, as these can cause discomfort.

Conclusion
In conclusion, it was better than I expected, as I expected all the visual issues to have a bigger impact. I think it can be used to create incredible, amazing experiences, that will feel like nothing ever felt before. And it obviously has a lot of applications outside entertainment as well.
I think and hope all this research will also be useful for traditional image synthesis, as for the first time we really have to systematically study perception and how our brain works, and not just be lucky with it. Also certain technological advances, for example in low-latency rendering system, will directly apply to traditional games as well.

I also think that it will be still for a long while a very niche product, or if it will succeed it will be due to a killer app that doesn't look in any shape or form like a traditional game, as if for certain technological issues we can clearly see a roadmap (weight, tracking, resolution, lag and so on), for certain others we don't have any idea yet, mostly controls but also how to deal with all the situations where we our brain is accustomed to have more sensorial hints than just what the eyes tell.
Even tiny things, like just the fact that with position tracking we can compenetrate with everything is quite an issue to solve. Fast movement is hard as well, it exacerbates the technical issues (lag, refresh rates and so on) to a degree that even "cockpit" games are hard (not to mention the lack of acceleration), even worse if you have to move your body in any athletic way as it's easy to get the VR system out of the optimal alignment it needs for crisp vision.

I don't think games can be "ported", a FPS in VR will be much more of a gimmick than e.g. FPS with virtual joysticks on an iPhone. We will need radically new stuff, low movement (for now at least, later on maybe some cockpit games can work well enough for the masses), novel ways of interaction (gaze for example can work decently, wands do work great... kinect-like stuff is very laggy and thus limited to only gesture recognition, not direct manipulation right now), new experiences...

It will probably be for a few early adopters, but I'm quite persuaded to be among them, just to be able to create weird environments that feel real.

 P.S. I saw the number "3" multiple times in Valve offices. You certainly know what that means...

 Update: Sony's Morpheus prototype is worse than Oculus DK2 as far as I can tell, and Oculus DK2 is still quite a bit behind Valve's demoroom. It need much more resolution.
No comments:
Etichette:
Subscribe to: Posts (Atom)