Search this blog

26 February, 2014

Valve VR. I want to believe.

Premise: Spoiler alert, I guess. Albeit it might not matter, it's worth noting that if you're going to experience either Valve's or Rift's Crystal Cove demo anytime soon (GDC perhaps) you could want to consider approaching it without any preconceived notion that this or other articles might give. Also, lengthy as usual.

Valve fanboy, I am not. In fact I might say I still hold a grudge against it, I want Valve to make games, not stickers to slap on PCs in a marketing stunt with the hope of moving some units of a platform in general decline. I understand that Steam makes money, but on a personal level I can't really care...

So this morning, after waking up way too early for my habits I arrive at Valve's building I can't help muttering "so cheap" as I look for the street number and make sure that I'm in the right place. Of course it's silly of me as money is not the reason why they didn't bother to put a logo on the facade nor the name on the directory, but anyhow I digress... The "I" in the titular "I want to believe" here refers not to my faith, but to how my brain reacted to the system.

Now, it's hard to put in words an experience, especially something novel as this is, as there aren't easy analogies to make and frames of reference in most people's experiences to anchor to. I'll try my best to explain how it works, or how I think it works.

Visuals are a tricky beast. Rendering engineers are often deeply embedded in very technical concerns, but at the end of the day, what really matters is psychology. What happens, when stars align and things go right (because, mind you, we're really far from making this a science and the little science there is most of the times is still very far from mingling with entertainment) is that we create a visual experience that somehow tickles the right neurons in our brain to "evoke" a given atmosphere, sensations, feelings that we learned in real life and are "replayed" by these stimuli. 
Mostly for me that happened with environments, I guess because we're really discriminating when it comes to characters: Call of Duty, Red Dead Redemption, Kentucky Route Zero are all great examples.

When we try to achieve this via photorealism we hope that by simulating what's perceptually important, what we can notice in a scene, the light, the materials, the shapes and so on, we reach a point where our brain accepts the image we generated as a reproduction of reality, like a photo. And we hope that our artists can use this tool that gets us close to reality to more easily be able to fool our brains into producing emotions, because we're nearer to the paths that normally fire when we experience things in the real world.

Inside Virtual Reality
A good VR experience completely sidesteps all this, Abrash says it right, when things align in VR you get presence and it's an infinitely more powerful tool. Presence is the sensation that you are there; That virtual wall is at ten meters from you. And it is really unquestionable. Realism doesn't matter anymore.
The VR prototype suffered from all kinds of defects: it's clearly "low resolution", a lot of demos didn't have lighting, most did do only lightmaps or diffuse, not specular, most had no textures, I could see banding and even some aliasing, I could spot errors in the lightmaps, even quite clearly the ghosts from the OLEDs and so on and on. A nightmare for a rendering engineer, on a 2d screen you would have said the worst visuals ever.
Yet you were fooled into thinking you were there, not through realism but through the immersion that is possible with the low-latency, full head tracking (rotation AND position) stereo sauce Valve has implemented. I suspect there are a million things to get just right, we discussed how OLEDs provided enough dynamic range that you didn't question the scene much for example and they have a catalog of things that are crucial and things you can ignore.

The most succinct way to describe this is that in all medias before this, at best you had the impression of looking at a greatly detailed, technologically advanced reproduction of reality (think of the best immersive stereo projection movie you've ever seen for example). Here even when you're looking at really basic renderings you think you are in some sort of weird, wrong alternate world, similar if you wish to certain installations, rooms that play with light and shapes to create some very unusual experiences.

The demo environments Valve created (or actually I should say, I witnessed) were quite tame. Clearly they didn't want to push it to avoid certain people reacting negatively to the experience. Most of the time the scene was static, not in the sense of devoid of animation but as in a fixed room you could move in but that wasn't moving relative to you. Things never got scary and I didn't interact in any way with the simulation (even if I, in numerous times, instinctively went reaching with my hands to objects and avoided objects getting to near me), yet there were some intense moments.

I guess a few people now described a scene where you're in the simplest room possible, no shading, no lighting, yet you start on a small ledge and you can't avoid feeling vertigo and have to actively force yourself to step into the "void". You know it's not real, at an intellectual level. You have all the possible visual hints saying it's not real, yet your brain tells you otherwise.
Another, switching for the first time to a scene with some animated robots, at the moment of the switch I had a second of high alertness, as your primitive brain steps in and rushes to prepare for a suspicious activity.
The weirdest sensation was at the end, flying (very slowly, as apparently motion creates quite easily discomfort) through CDAK. There the visuals were distorted enough (see the youtube video linked) that I didn't feel as much presence (so there are some extreme cases where visuals can break it), yet when some of the weird blue objects passed through me I had, again for a split second, a sensation that I could only later rationalize as reminiscent somehow of being in a sea, I guess because that's the only thing I have in my experience of going through something like that.

When does presence break? Visuals can be pushed quite far before they break presence. Mind you, rendering will be a huge issue there and good rendering I am sure does make a difference to remove the idea that you are immersed in something quite odd, but again it simple visuals don't break immersion. I would also have loved to see a scene with and without various rendering effects (visual hints) but alas, no such luck.
Even the low resolution and blur and ghosts are to me defects of my vision, like seeing through glasses or through a dirty motorbike helmet, not a problem of the "reality" of the scene. Impossible behaviours do. In one of the scenes for example there were some industrial machines at work behind a glass wall. Poking my head through the wall into the room breaks it. 
It's not like suddenly losing stereo, as in one of the cross-eyed stereograms where you focus on the page and lose the effect. The closest analogy I can think of is Indiana Jones' leap of faith, and you might have experienced something similar with some visual tricks in theme parks: you realize it's an illusion.

There are a myriad of ways of doing something that breaks the presence, many things that are totally acceptable on a normal display are intolerable in VR. You might know that you can't really use normal-maps or any other non-stereoscopic hints of detail for example, but you are also much more aware of errors in animation, physics, not to mention characters (which weren't demoed at all).
And it's good if the consequence of an error is only bringing back to the idea you're in a VR helmet, the bad cases are when certain hints are very strong, but certain others are completely wrong or missing, like falling without acceleration, wind and so on, as these can cause discomfort.

In conclusion, it was better than I expected, as I expected all the visual issues to have a bigger impact. I think it can be used to create incredible, amazing experiences, that will feel like nothing ever felt before. And it obviously has a lot of applications outside entertainment as well.
I think and hope all this research will also be useful for traditional image synthesis, as for the first time we really have to systematically study perception and how our brain works, and not just be lucky with it. Also certain technological advances, for example in low-latency rendering system, will directly apply to traditional games as well.

I also think that it will be still for a long while a very niche product, or if it will succeed it will be due to a killer app that doesn't look in any shape or form like a traditional game, as if for certain technological issues we can clearly see a roadmap (weight, tracking, resolution, lag and so on), for certain others we don't have any idea yet, mostly controls but also how to deal with all the situations where we our brain is accustomed to have more sensorial hints than just what the eyes tell.
Even tiny things, like just the fact that with position tracking we can compenetrate with everything is quite an issue to solve. Fast movement is hard as well, it exacerbates the technical issues (lag, refresh rates and so on) to a degree that even "cockpit" games are hard (not to mention the lack of acceleration), even worse if you have to move your body in any athletic way as it's easy to get the VR system out of the optimal alignment it needs for crisp vision.

I don't think games can be "ported", a FPS in VR will be much more of a gimmick than e.g. FPS with virtual joysticks on an iPhone. We will need radically new stuff, low movement (for now at least, later on maybe some cockpit games can work well enough for the masses), novel ways of interaction (gaze for example can work decently, wands do work great... kinect-like stuff is very laggy and thus limited to only gesture recognition, not direct manipulation right now), new experiences...

It will probably be for a few early adopters, but I'm quite persuaded to be among them, just to be able to create weird environments that feel real.

P.S. I saw the number "3" multiple times in Valve offices. You certainly know what that means...

Update: Sony's Morpheus prototype is worse than Oculus DK2 as far as I can tell, and Oculus DK2 is still quite a bit behind Valve's demoroom. It need much more resolution.

10 February, 2014

Design Pattern: The Push Updater

Just kidding.

Let me tell you a story instead. Near the end of Space Marines we were, as it often happens at these stages, scrambling to find the extra few frames per second needed for shipping (I maintain that a project should be shippable from start to the end, or close, but I digress).

Now, as I did such a good job (...) at optimizing the GPU side of things it came to a point where the CPU was our bottleneck, and on rendering we mostly were bound by the numbers of material changes we could do per frame (quite a common scenario). Turns out that we couldn't afford our texture indirections, that is, to traverse per each material the pointers that led to our texture classes which in turn contained pointers to the actual hardware textures (or well, graphics API textures). Bad.

Most of the trashing happened in distant objects, so one of the solutions we tried was to collapse the number of materials needed in LODs. We thought of a couple of ways in which artists could specify replacement materials for given ones in an object, collapsing the number of unique materials needed to draw an LOD (thus making the process easier than manual editing of the LODs). Unfortunately it would have required some changes in the editor which were too late to do. We tried something alongside of the idea of baking textures to vertex colors in the distance (a very wise thing actually) but again time was too short for that. Multithreading the draw thread was too risky as well (and on consoles we were already saturating the memory BW so we wouldn't get any better by doing that).

In the end we managed to find a decent balance sorting stuff smartly, asking for a lot of elbow grease from the artists (sorry) and doing lots of other rearrangements of the data structures (splitting by access, removing some handles and so on), we ended up shipping a solid 30fps and were quite happy.

A decent trick is to cache the frequently-accessed parts of an indirection, together with a global counter that signals when any of the objects of that kind changed and a local copy of the counter. If you saw no changes (local copy of the counter equals global) you can avoid following the indirection and use the local cache instead... This can get more complex by having a small number of counters instead of just one, hashing the objects into buckets somehow, or keeping a full record of the changes that happened and have a way for an object to see if its indirection was in the list of changed things... We did some smart stuff, but that was still a sore point. Indirections. These bastards.

So, wrapping up the project we went to the drawing board and started tinkering and writing down plans for the next revision of the engine. We estimated that without these indirections we could push 40% more objects each frame. But why do you have these to begin with? Well, the bottom line is that you often want to update some data that is shared among objects. In case of the textures the indirection served our reference counting system which was used to load and unload them, hot-swapping during development and streaming in-game.

Here comes the "push pattern" to the rescue. The idea is simple, instead of going through an indirection to fetch the updated data, create an object (you can call it UpdateManager and create it with a Factory and maybe template it with some policies, if that's what turns you on) that will store the locations of all the copies of a piece of data (sort of like a database version of a garbage collector), so every time you need to make a copy or destroy a copy you register this fact. Now if create/destroy/updates are infrequent compared to accesses, having copies all around instead of indirections will significantly speed up the runtime, while you can still do global updates via the manager by poking the new data in all the registered locations.

A nifty thing is that the manager could even sort the updates to be propagated by memory location, thus pushing many updates at once with potentially less misses. This is basically what we do in some subsystems in an implicit way. Think about culling for example, if you have some bounding volumes which contain an array of pointers to objects, and as these bounding volumes are found visible you append the pointers to a visible object list, you're "pushing" an (implicit) message that said objects were found visibile...

06 February, 2014

OT: The Sony RX1 (versus Fuji X100s)

I recently bought an used (new is way too expensive) Sony RX1, time for another photographic off-topic. I'll keep it short, follow the links if you want to know more.

Throughout this I'll share some meaningless images of my first week with the RX1
- Mhmmm, GOOD

First and foremost, the lens. Or if you prefer, lens/sensor combination, because these days lenses mean nothing if not seen on a specific sensor. Case in point, all modern mirrorless cameras from the Sony NEX to Micro 4/3 to the recent Sony A7 fullframe in theory can mount a lot of extemely expensive glass from Leica and Zeiss, in practice nothing by the Leica M seems to perform well with these, especially if you go 35mm or wider. 
Yes you can correct some of the defects (and really the Leica M does apply some correction in-camera, that's also why it works better) but honestly, if you're attaching a thousands-of-dollars lens to a body, you don't want anything less than perfection.

So, how does the Sonnar do? One word: wonderful. It's The reason to buy this camera. Easily the best 35mm I've personally used, film or digital, and among the best lenses I've ever seen. 
It's a "people" lens, optimized near and wide open, doesn't care about avoiding distortion (and it shouldn't, these things are easy to correct digitally and lens optimization should focus on other issues nowadays) from f2 to f4 you basically gain only less vignetting. Some people say it's somewhat worse at infinity, I didn't pixel-peep that but I can believe it, still it's an excellent performer, with a dreamy, smooth quality.

Loves people
Second, the handling and build quality. It's very, very good. Still not perfect because you know, no way a modern camera ships with all the controls done right, so you still need two presses for manual focus assist/zoom and you don't have a direct ISO wheel nor an absolute shutter speed wheel (i.e. with markings), but the aperture ring is excellent (fly-by-wire but it feels mechanical, third-stop clicks) and the manual focus one is very good as well. Also, plenty, plenty of customizable buttons. Good rear LCD quality. Well done.

Last but not least, the sensor. Nothing really to say, it's a 24 megapixels full-frame and it delivers exactly what you expect. Very low noise, lots of details, it's quite normal for such a sensor but in a "compact" camera it's not something you see often (in fact, this is the smallest full-frame camera you can buy, period).

No light? Not a problem
- Eeeew, BAD

It's a toy. A wonderful, mesmerizing toy, but not a "professional" camera. At best a second body to have around just for its lens/sensor combo. That doesn't mean you can't take amazing photos with it in any possible condition, it's just not done in a way where I would depend on it if it was critical, if a missed shot meant failure. Why?

I want a "Monochrom" version of this camera!
First, it's lacks a viewfinder! Oh buy you can buy an external one... Let's see, you have two bad options there. 
The first is to go with an EVF, which not only is an EVF (duh) but also attaches to the flash hot-shoe, leaving you without the option of attaching an external flash.
The second is to buy an OVF, but even Sony's own is just a pure optical VF, with no indicators, not even focus confirmation. Which means it's utterly useless, because there are only two scenarios, really. Either you're in a sunny day, and then you can/have to use fast shutter and the three-contact-points given by a viewfinder are not important (certainly not enough to lose all indicators!) or you're at night, then you would want to grip the camera better, but focus is critical, you're probably wide-open and can't even trust the autofocus... So yes, no indicators means no photo.

All in all the EVF is still an important accessory to buy, even with its defects, it's your best bet, and it helps if you want to be able to review images in bright light. I really hope Sony will make an OVF with some electronics in it... but I know it won't happen, maybe when they do an RX2...

Second, it's slow. Especially the autofocus. Not the slowest ever but among the slowest, the good is that even if it's slow it seems to truly not lie. If it's green, it's in focus. Autofocus systems that lie, saying it's in focus while it's not, are worse than anything else, but this especially in lowlight is problematic, it requires area with lots of contrast, more than what you are used to use, and if you're photographing a moving subject it can be frustrating. Also, very bright regions seems to be able to confuse it (I guess because the out of focus highlight still register as contrast, but it's only a guess)
Luckily it's quite easy to toggle AF/MF by assigning that to one of the buttons which makes the camera much more usable: just AF then lock and do MF adjustments as needed.


The way I see it, there are only two alternatives in its class, or maybe even just one if we wanted to be totally fair.

The X100s is prime competition. A bit unfair because the X100s will fit in a coat pocket while the RX1 doesn't, so it's in a different size range (useful ranges are: fits in pants, fits in coat, everything else) but still it holds its own and in practice many people will wonder what to do between the X100s and the RX1.

Having used both of them I can say, the X100s hybrid viewfinder is the reason to buy that camera, the RX1 lens is the reason to buy this. Interestingly if you look on dpreview (my first stop for camera reviews) you could think that browsing the studio shots there's not much difference between the X100s lens/sensor and the RX1, or anyways it's really hard to tell while pixel-peeping. Well, it's wrong, in real-life the Sony camera is on a entirely different planet.

The RX1 also feels better built and has a less erratic AF, but the X100s AF is quite a bit faster... The RX1 has probably better peaking (due to a better LCD really) but the X100s has the excellent virtual split image aid... All in all? I prefer the RX1, because even if the X100s is almost in every way better to shoot with (and mostly because of the incredible viewfinder, really almost everything else is worse, handling-wise) when you get the image the Sony image quality shines quite a bit ahead of the Fuji.

The second, more fair competition is Sony's own A7 and A7r... but I didn't try these so I can't say much. For now, they have no lenses really, just a zoom that seems quite mediocre and a 35mm f2.8 which seems good but not as excellent as the RX1 f2, and really these differences are "worth" (or maybe I should say cost...) thousands. Yes you can buy adaptors, which are worthless, as they make the camera/lens combination bigger (and you don't want that in a camera that's supposed to be compact!) and often perform not-so-great.

Also the A7 focal plane shutter won't flash-sync as fast as the RX1 and won't be as quiet and vibration-less. In fact the A7r shutter seems to be suffering severe vibration issues and it's quite on the loud side (even without the mirror!) while the A7 seems better in that regard.
For now the A7/A7r are toys with no real advantage over the RX1, but if (and it's a big if, as that was one of the really sore points of the Nex line) Sony makes some truly great compact lenses for them, then the situation will change significantly.

P.S. It's actually good that the X100, X100s, RX1 and A7/r are "flawed gems". They come used for much cheaper than new even if they are new-ish cameras, as some people can't deal with the flaws and some others will enjoy the discount... Compare this with well-established DSLRs, I can't think of any reason for example to change my 5d-II, in fact even if it's a much older camera, and new went for around the same price, today you can find the RX1 selling for around the same price as the 5d!