Digital Dragons 2020

Slides from my two presentations at this year's Digital Dragons. Enjoy!

Open Problems in Real-Time Rendering

A.k.a. Are we solving the wrong problems today? We have been looking a lot at certain bits and pieces of math, but are we looking at the -right- pieces? A reminder that math and physics don't matter, if they don't solve actual problems for people - artists, or players.

Note: Naty Hoffman in his presentation at Siggraph 2020 physically based shading course makes some similar (better!) remarks while he shows how we don't know Fresnel very well... Must read!

Rendering the Metaverse across Space and Time.

An introduction to Roblox and how its design and mission shapes the work we do to render shared worlds on almost any device, any graphics API - and how we Roblox has been achieving that for more than a decade now.

What I learned at Activision.

Today is my last day at Activision.

Not quite the end of an era, but my six-year stint at Activision|Blizzard|King has been by far the longest I’ve so far worked for a given company, and I wanted to write something about it.

I don’t usually do things like this, but long gone are the times where I pretended this blog could stay anonymous. Also, I do think that we should really talk more about our experiences with teams and companies, be more open. I’ve never done that on this blog (albeit I’ve always been happy to chat about anything in person), so let me quickly fix that.

This has been my, to be honest quite lucky, video-game career trajectory:

- Milestone (Italy, racing games). The indie company. Here, we were a family. And as most families, often loud and dysfunctional, sometimes fighting, but in the end, for me, it was always fair and always fun. We were pioneers, not because we were necessarily doing state-of-the-art things, but because nobody around us knew better, we had to figure out everything on our own. Eventually, that became a limiting factor for my own growth, but it was great to start there.

- Electronic Arts (Canada, Fight Night team). My team at EAC was probably the best example of a well-organized game studio. Everything was neat, productions went smoothly, and we created some quite kick-ass graphics as well. EA, and probably even more so EA Sports is truly a game developer. And by that I mean that it takes part in the game development, we had shared guidelines, procedures, technologies, resources. Of course, each team could still have plenty of degrees of freedom to custom-fit the EA way to their specific reality, but you always felt part of a bigger ecosystem and had access to this gigantic wealth of knowledge and people across the globe. Fun times!

- Relic (Space Marines, mostly). Relic had much more of the “indie” feel of my Milestone days. Not quite the same, we were a big team in an even bigger studio, part of a publicly traded company, but we were also exploring uncharted territory (for us), with very smart people and lots of last-minute hacking. I’m proud of what we achieved, it was fun and I loved the spirit we had in our rendering/optimization corner of the office. We did perhaps chew a bit more than we could though, doing something unprecedented for the studio. THQ was also failing fast, which didn’t help.

- Capcom (Vancouver). This studio is now closed. It was an unlucky choice for me, our project was riddled with all kinds of problems, in all possible dimensions. Eventually, I was laid off, then the project was canceled, and a few years later, the studio went down. It was a very stressful time, day after day I was quite unhappy. Still, I have to say I met some excellent people and I’m glad that I now know them!

- And now, Activision.

It’s the people.

One of the company mottos is “it’s the people”. I didn’t particularly think that these “values”, that all companies put forward, were particularly received in Activision’s case, at least when it came to Central Technology I always felt we didn’t pay much attention. But for me, it was the people, first and foremost.

Activision is the place where I found the smartest people around me, by far. And I’ve worked with very smart people, in great companies, but nothing touches this.

Now, I have to say, I have also a unique, very biased vantage point. Being a technical director in central technology means to interface mostly with the most senior technical people and the studio leadership. I was not in production and not working with a single studio. A different ballgame.

But still, I can’t even make list here! Ok, to give you a picture. ACME: Michael, Wade, Paul, Aki, I don’t think I can in a few words express the brilliance of these individuals. Paul’s technical abilities are unbeatable. He knows everything and can do anything. Wade is worse, infuriatingly smart, and any time I find the tiniest flaw in Michael’s life I have a sigh of relief (my girlfriend says he’s my work husband, even if I do have several man crushes to be honest, yes, there is a list). Aki started as an intern and was recently hired full time. I think he is already a better programmer than I am, and I definitely do not suffer from impostor’s syndrome...

Peter-Pike Sloan’s research team is the best R&D team I’ve ever seen, with people like Michal Iwanicki, Josiah Manson, Ari... But then again, I literally can’t make a list and I’m talking only about the rendering people, no actually, the rendering people I know best, even! My home team at Radical, which was a great studio in its own right, has been fantastic, Josh, Ryan, Tom, Peter, Andrew. CTN and that shadertoy genius of Paul Malin. And then the game teams. I mean, you can’t beat Drobot, can you? Jorge Jimenez! Dimitar Lazarov, Danny Chan, these are people who every project decide to completely change how Call of Duty renders things. Because why not, right? And of course, the people above me, Christer, whom I admired way before he landed at Activision but also Dave Cowling who hired me, and Andy our CTO who came to speak to us a year before he got hired, and even back then I thought he was an incredibly smart guy. And that's to speak just of the people in the Call of Duty orbit...

Truth is, if you know, you already know. If you don’t, it’s hard for me to tell you, so let’s just cut it short. Amazing people.

Not just smart.

Ok, so we have smart people, blah blah. What gives, are you just showing off or is there a point to this? There is actually.

And the lesson is not even “how to hire smart people” or that you should hire smart people. To be honest, I don’t think we have a way, even if Christer (and myself even) did put a lot of time and effort thinking about the process, you can only affect some multipliers I think. Mostly, teams seem to me to build up from the gravitational pull, around a given company culture. Once you have a given number of people that value given things, they tend to grab other people who also share the same core values, even if you want to have different points of view.

No, what Activision really taught me is that smart people don’t really matter, per se. Brilliance alone doesn’t mean that good things will happen to your products, actually, it can be dangerous because it can correlate with ego, especially if you haven’t reached the highest level of enlightenment yet.

What’s interesting about this particular bunch of smart people, is that they are also what actors call “grounded”. There is little bullshit going around. Tech is not made for tech’s sake. We don’t even have an engine, in a time where even if you really just have a game, one game, one codebase, you would call silly codenames each library and each little bit of tech, and maybe put some big splash screens before your main titles. Made with.. xyz.

This is actually a lesson that originates from the early days of Call of Duty at IW. From what I’m told, IW never named their engine, because any time you name something it becomes a bit more of a thing, and you start working for it. And they were a studio making games, the game is the thing. Nothing else.

This meshes just so well with the Activision way. So many people taught me so many lessons here, but the kind of fil rouge I found is this attention to what matters, when especially for us, rendering people, is so annoyingly easy to get distracted, quite literally, by the cool shiny thing that’s in the hype at the moment.

You might even call it ROI, I sometimes do even if perhaps I should not, it sounds “bad”. But you have to be aware of what matters, what are the trade-offs, what should you spend your time on. Which doesn’t mean you are a drone doing complex math in Excel, quite the contrary, there is definitely space for doing things because you want to, and you like to. 

Thing is, we cannot really compute the ROI of our tech stuff, especially for a thing that is so far from the product sales as rendering code is. But, we can be aware of these things. Even just thinking about them a bit makes a big difference.

Peter-Pike could be another example. I said I’ve never seen a better R&D team. Does it mean that there aren’t other teams that do research as good, or maybe even “better” sometimes? Of course, there are! The difference though is impact. Not only almost all our R&D work shipped on the given year’s title, but it’s also focused on what matters. Either help ship the title, technology that is needed to do what they want to do, or help the teams work better, technology to help productivity. Often both.

And this then again reflects also in the people. Yes, Peter-Pike is an accomplished academic, and his team does real research, meaning, things we don’t know how or even if we will solve. But he’s also incredibly pragmatic. He is hands-on coding all the times. More than I am. Better than I am!

Activision and Electronic Arts...

...couldn’t be more different, contrary to the popular belief that lumps all the three-four big publishers into the same AAA bucket.

And that is also what was interesting and life-changing. You see, I truly love them both. They are great in their own right and the results speak for themselves. They both have great people, I don’t even need to tell you that.

But they work so differently.

EA as I said, feels like a big game developer, a community. Sharing is one of the key values, finding the best ways of doing things, leveraging their size. It sounds very reasonable, and it is.

But Activision is almost the opposite. It feels like a publisher, who owns a number of internal studios. The studios, of course, are accountable to the publisher, but they are independent, that is key, core value. Central technology is not there to tell people how to do things, but as a publisher-side resource to help if possible. The teams are incredibly strong on their own, even in terms of R&D. And, at least from my vantage point, they get to call all their own shots. Which is again very reasonable, if you tell people they are accountable they should definitely have the freedom of choice too.

And yet again. Valuing independence versus valuing a shared community, the opposite viewpoints end up in practice creating results that are not that dissimilar. How comes?

If you’ve been doing this for a while, it won’t be even surprising. The key is that we don’t really know what’s best. Companies and teams even technologies and products organize around given values, a cultural environment that was created probably when there were almost a handful of employees. And once you have these the truth is you can most often structure everything else around in a way that makes sense, that works. 

We see this so often in code too. A handful of key decisions are made because of legacy or opinions. We’re going to be a deferred renderer. Well, now certain things are harder, others are easier. Certain stuff needs to happen early in the frame, other late, and you have certain bottlenecks and idle times, and you work from there, find smart ways to put work where the GPU is free, and shift techniques to remove bottlenecks and so on.

Which doesn't mean everything is ever perfect, mind you. We always have pain points and room for improvement, and different strategies yield different issues. To a degree, this is even lucky!  I don't make games, I help people and technology. The day Unity will solve all technical and organizational problems for game making is the day I'll be out of a job. At least in this industry...

Ok, then why?

If you love all these companies. Why leave? What are you not telling...
Well. I’m not too bright. And instead of keeping my great job, sometimes I venture into the unknown. But that’s a tale for another day...

Activision @ Siggraph 2016

Siggraph 2016 recently ended, as always it was inspiring and lots of great discussions were had. Activision presence was quite strong this year after we shipped Black Ops 3.

If you missed any of our presentations, this post might help. I'll try to keep this updated with links relative to our presentations, as soon as they come online.

Note: for convenience, I've written a summary of the techniques in the text below but keep in mind that, as always, this blog is truly made of personal opinions and observations - which come from my limited, R&D centric point of view.

- Volumetric Global Illumination at Treyarch

Call of Duty has always been a lightmapped title, but lightmaps come with a quite large set of issues: large baking times, complexity in representing runtime materials and effects, lighting discontinuities with dynamic objects and the inability of "kit-bash" geometric detail (compenetrating meshes).

Static and dynamic objects, particles, volumetrics:
all lit with the same illumination data!

It's incredible that after all this time, there still isn't in real-time rendering much research on ways of solving the problem of baked irradiance. Naive lightmaps are not enough to generate an image that does look coherent, without leaks and discontinuities.

This new development at Treyarch tackles all these issues at once with a new runtime representation of baked irradiance that works seamlessly with deferred shading and is tightly coupled with prefiltered irradiance cubemaps and new, state of the art, heuristics for parallax-corrected reflections and a baking system that cut down iteration times substantially for artists without having to resort to renderfarms.

Baking via prefiltered cubemaps

Now all our art production happens in a fully WYSIWYG editor, and all the lighting is unified, regardless of the object type (dynamic, static, skinned, particles, volumetrics...).

This is achieved by employing hardware-filtered volumetric textures as the only representation of baked irradiance.

WYSIWYG editor with tools to quickly place volumes in the scene

Artists place irradiance volumes quickly in a level, and author clipping convex volumes to avoid light leaks (note that these have to be authored anyways for reflection probes, so it's not really extra work, and our editor supports very fast workflows for placing volumes and planes), and the authored volumes are usually robust to iterations in the level geometry.

As part of this system, Josiah Manson also further developed his fast GGX pre-integration scheme, now presented at EGSR 2016.

- Sparse Shadow Trees

Kevin Myers presented in the "dark hides the light" session our new compression technology for baked shadowmaps. 

Call of Duty had a system for caching shadowmaps since Ghosts, as the new generation of consoles saw a bigger increase in memory than in computation, caching techniques have seen a resurgence. 

This work is a quite radical improvement of our shadowmap caching technology, and allows to fully pre-bake a shadowmaps for entire levels, achieving thousand-fold compression ratios.

In comparison with precomputed voxelized shadows this technique allows for easily recovering the shadowmap depth, which makes it easier to integrate with other effects (i.e. volumetric lighting). It's also fast to traverse. The closest relative to SSTs are the Compressed Multiresolution Hierarchies of Scandolo et al., but our solution was developed independently in parallel, so the data encoding employed is not the same.

To my knowledge the first use of compressed shadowmaps in game production. It also enabled savings in the g-buffer, as we can use the SST to quickly shadow far objects, instead of having to bake a lightmap space occlusion map.

- Efficient GPU Rendering of Subdivision Surfaces  

This is the continuation of the subdivision surface research started by Wade on Call of Duty Ghosts, (which to my knowledge is the first videogame to make extensive use of Catmull-Clark subdivision surfaces in real-time).

It's a quite neat technique, which sidesteps a limitation in current hardware tessellation pipelines by passing a variable number of control points to the hull shader via L2 read/writes.

During the development of this technique, Wade also made a nifty thread tracer for NVidia GPUs, which helped debugging issues with work being (erroneously) serialized on the GPU.

This is the only presentation at Siggraph 2016 that is not directly linked with a shipped Call of Duty title.

- Open Problems: Geometry and Material Representation in Games

Natasha and Wade gave a compelling presentation at the open problems in computer graphics course, surveying artists in many companies, even going outside the videogame companies circle we work in.

This is one of my favorite courses at Siggraph, and the only presentation from Activision that I haven't reviewed beforehand, so it was truly interesting for me to watch it live.

- Filmic SMAA

Jorge Jimenez presented a new version of his SMAA antialiasing technique, greatly improving both performance and quality (sharpness and stability) with a plethora of tweaks and innovations. The slides also summarize relevant previous published techniques and their image quality tradeoffs.

One of MANY improvements presented

In my opinion, with Jorge's Filmic SMAA, the quest for antialiasing techniques is largely over, and we can say that temporal reprojection has "won".

Not only temporal reprojection often achieves better quality than MSAA when using comparable performance budgets, but it's being easier to integrate into deferred renderers and nowadays it's becoming unavoidable anyways because so many other effects can benefit from the ability to perform temporal supersampling (e.g. shadows, reflections, ambient occlusion...).

Hybrid techniques are still very interesting, and there are probably still improvements possible (and probably Jorge will come yet again on stage in the future to show something amazing in that space), but in general the interest on MSAA reconstruction techniques for edge antialiasing has diminished from being a must-have and a pain point for deferred rendering, to a much less important solution.

MSAA is still a great feature to have in hardware though as it allows to subsample or supersample certain effects and render passes (e.g. particles), but it should be more thought as a way to do mixed resolution than strictly for edge antialiasing.

- Practical Real-Time Strategies for Accurate Indirect Occlusion

Jorge, Xian, Adrian and myself worked on extending rendering ambient occlusion state of the art by crafting techniques that are based on modeling the (ray traced) ground truth solution for diffuse and specular occlusion.

Fast, accurate ambient occlusion...

We derived closed-form analytic solutions when possible, and when such models could not be found for ampler generalizations of the problems at hand, we extended the analytic solutions by fitting "residual" functions to the ground truth data, or by employing look-up tables.

...and specular occlusion.

GTAO has already been used in production, as a drop-in replacement of the previous state-of-the-art technique we were employing (HemiAO), yielding better image quality (actually, even better than HBAO, which is a very popular high-quality solution) in the same performance budget (0.5ms).

Goodbye Zootool, welcome Twitter

I used to have the habit of sharing interesting articles, originally via google reader share, then I moved to ZooTool when the latter killed the share functionality. The good about this is that you could get a RSS feed and everybody was happy...

Nowadays, RSS seem not to be that hot anymore, I'm tired of ZooTool (I originally chose it because it was the only one integrated with my iPad RSS reader... reeder. Have since moved to mr.Reader if you care) so, I'm killing that (effectively I did a long time ago as if you were following that, you would have noticed I haven't shared anything in a while).

Also, once upon a time I used to keep this blog "anonymous" because I thought it would give me more freedom of commenting on games, graphics and so on without having people associate me with a given company. Nowadays I guess most people know whose blog this is an certainly google does... Also I end up being more "controversial" on outlets that are completely not anonymous, so what's the point...

Bottom line, if you don't yet and you want to, follow me on Twitter where I will share interesting stuff I find during the day... My handle is "kenpex".

And to make this post a bit less useless... let's add a couple of blurred instagram pictures of what I've been working on in the past few weeks...

Job free(ish)

http://www.gamasutra.com/view/news/174374/Dead_Rising_2_developer_suffers_layoffs.php
I won't comment much on this, Capcom Vancouver is expanding and its ways of dealing with projects and technology are peculiar, some people found themselves to be not ideal fits. I did my best to honestly guide my area, as usual, in a way that I believe was for the best. It has been a very intense, stressful experience and that is why the blog also did't see many updates.
I doubt that I will be able to stay in Vancouver, considering the way the game industry is going here (I'm guessing this city became too expensive for what it offers to big game studios).
I will probably have some more time to update this blog. Stay tuned.

Update - I'll join a team until December for a project I'm really excited about, focusing on topics that are very dear to me. Then, I don't know yet... Thanks to everybody for the support and opportunities offered.

Leaving Relic

So, time has come again, and today was my last day at Relic Entertainment. As I did when I left EA, I want to write a bit about my experience with the Relicans and why I left (but not where I'm going, it's not a big secret but I always keep that from the public of this blog to avoid spam in the comments).

Relic and EA(C) are almost polar opposites. The latter latter being the world biggest videogame studio, shipping iterations every year or less (Fifa...), relies on a very refined (even if still suprisingly flexible) production methodology, while Relic still feels like an Indie studio even after the acquisition by THQ. Relic works by distilling talent into games, it's really about people over processes.

One of the kitchens, with free pop. Donuts on Friday, I'm too fat...

And oh boy if they do have plenty of talent... It was truly a pleasure working among them!

By the way, I don't mean that the folks at EA are not bright, quite the opposite, it takes a lot of genius to make NHL, Fifa, Fight Night and all the other franchises there at the studio, but it's a rather different situation, it's a matter of priorities I guess. Working at Relic feels less constrained, sometimes scaringly so. It's a more hands on approach where great hackers can make a bigger difference, and they do.

Space Marine

I should have started my job at Relic on a title which is still unannounced today, but as it often happens things change from the interview to your first day in the office and I was immediately reassigned to Space Marines, and I stayed there until it shipped. And with my help it become the third worst game (in terms of metacritic) of Relic's history (actually, the worst game if we look at the PS3 metacritic only)...

What happened? 

I wouldn't know, really. I'm a gamer and a technician. I enjoy games and do everything that is required to make them visually beautiful, and that's all. I've seen too many people who are not experienced in my own craft saying so much... inaccurate things (and I was among them ten-ish years ago) about it that I don't really love talking about fields which are not my own.

But... If I have to tell you my own perspective... Nothing really happened. It went they way it probably had to. It was the company's first big console title, it was the company's first third-persons action shooter, the first multiplatform game. It started from zero and it was pitted against one of the biggest franchises ever (and I'm guessing here but I'd bet... spending less money to make it, and much less to market it for sure). Moreover, it intended to be a Hollywood production made by people who were really well known for their indie-art titles.

There were mistakes, sure. There was plenty of overtime.... Could it really have ended much differently? With the same people, the same culture, the same resources and amount of support I would say not. 

An area were part of the SM team was that has not been reassigned yet

You can see Relic behind Space Marine... I'm not a fan of the genre but I can tell that the core gameplay is incredibly well done (and remember, company's first shooter!). I had fun with the multiplayer, and I don't play multiplayer! And the rendering is much more accomplished than it seems, and its consistency really surprised me. The amount of features these geniuses managed to cram in the development cycle of something that basically started from scratch is amazing. What it lacks its the production, is the depth and breadth.

The table-tennis room. No day went by without a 2vs2 match

Anyhow it doesn't really matter, and surely what I write on the argument does not matter. I'm sure that Relic will make new and really great games, the ones I saw in development are... exciting, really. 

New, great games will come.

Unfortunately, they are not the kind of projects that I'm personally, for my career, needing, and there was nothing I or them could do about that obviously. We just become not matched for each other, and it sucks, but it happens. Goodbye and best luck to all Relicans and be proud of Space Marine!

Bye...

Replacing google share: zootool

There are quite a few bookmarking services on the web today, and I never felt the need to find one until Google shut down its handy reader share feature. So, kinda randomly I picked Zootool and I've been working on sharing some links for a few weeks, so now there is a bit of content and I can publish its feed. Let me know if there are problems, otherwise I'll stick to this one to share my finds in the "rendering et al." category, at least until Reeder does not add support for Google+ :)

http://zootool.com/feeds/user/c0de517e/type:all

Google reader share

So, some of you have noticed that my google reader share is dead (not updated anymore). That's not me being lazy, but google being a bit evil and trying to shove google+ down our throats by killing the old facilities instead of integrating plus into the existing stuff. So, there is no reader share anymore, and my iPad newsreader (reeder), which was responsible of most of the share posts, does not support google+ yet. Stay tuned.