tag:blogger.com,1999:blog-6950833531562942289.post6952902666023112143..comments2024-03-25T03:36:48.099-07:00Comments on C0DE517E: Commenting on graphical shader systemsDEADC0DEhttp://www.blogger.com/profile/01477408942876127202noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-6950833531562942289.post-82417458607120589782008-12-02T09:50:00.000-08:002008-12-02T09:50:00.000-08:00Hi, this subject is very touchy but I though I'...Hi, this subject is very touchy but I though I'd share my own experience. To give a bit of context on my opinion here, I have been a game developper for 8 years now and I have seen the tech evolve. I have been the rendering lead engineer on a triple-A PS3/Xbox360 game that shipped earlier this year using the Unreal engine. So, to satisfy Yours Truly's requirement, I know all about Unreal's implementation of the node graph shader base system. All the flaws of this system are very well pointed out by DeadC0de. I personnally have suffered greatly because of this system (well, our use of it actually). In our last project, we had around 4000 different materials (imagine that!), created by some 20-30different artists, each with a set of shaders for depth pass, emissive pass, light&shadow passes and counting all the vertex factories, resulting in, say, 100 real shaders per material. The result of this is most of those materials are sub-optimal performance wise and also memory wize. Optimizing those for PS3 was a real challenge. We managed to optimize some of them in critical areas, but this cost us many precious months of profiling and optimization (really negating all advantages of the system). The game slipped 3 months, not only because of this, but in part because of this. And did we even get better looking shaders out of it? I don't think so. No artist was able to bring the best out of the system. They felt empowered and all, but in the end, their shaders where no better looking than simpler ones. <BR/><BR/>So, all this to say, in real life, with the current situation, a node graph based shader system for a game is a very very dangerous thing. YOU CAN'T LET IT GET OUT OF HAND. Rules have to be enforced (and when you do that, it means your system is flawed!). I am currently working on the sequel to the game, still using Unreal engine. This time around, we have 2experienced shader artists that create the parent materials (equivalent to uber shaders) using the material instance system in Unreal. Artists are not allowed to create materials anymore (!!). They must use one of the parent material in a fixed set and instance it, effectively creating an ubershader with parameters and check boxes system. This way, we have fixed the uncontrolled proliferation of materials/shaders. Also, optimizing the parent material will optimize all the instanced materials, which is pretty neat. The parent materials do not require engineering support to be done, technical artists can do it and verify performance on the consoles using performance tools. <BR/><BR/>In summary, this actually boils down to a ubershader approach (implemented by technical shader artists instead of fully fledge engineers) and proves, to me at least (through those past 3 years of struggle), that a node-graph based shader system cannot work in real life game production situation. It really boils down to the points Deadcode (and Christer) are making. So far, the uber-shader (or uber-materials) approach has worked really well. We have cleared pre-pro and are now in full production and the number of shaders is quite stable. So I would suggest, to anyone using Unreal, to use the material instance system. I think it is the best hybrid approach where you can still expose the freedom of node based system to experienced technical shader artists and wrap it in an uber-shader system (which is the layer normal artists will use).<BR/><BR/>J-SJ-Shttps://www.blogger.com/profile/00751573558103213471noreply@blogger.comtag:blogger.com,1999:blog-6950833531562942289.post-79546958941714891112008-08-10T15:26:00.000-07:002008-08-10T15:26:00.000-07:00I guess you're right, if you constrain the graph s...I guess you're right, if you constrain the graph system enough you can achieve a good balance.<BR/><BR/>But then the advantages of having a graph-based system become thinner, if a programmer is still involved for any non-trivial change. It starts to look more like a switch based system...<BR/><BR/>And still you loose semantic, and still you don't have an easy way to control packing and all the tiny details that can make a huge difference (i.e. declaring a constant in a way or in another, that could save a register and have the shader to be scheduled on more threads...).<BR/><BR/>If you restrict the changes you can make to the number of used textures well, my #define based shader system supported up to four texture layers and artists could choose any blend-mode across them plus having a constant multiplier/opacity parameter, they had a lot of flexibility in that respect... And yes, if they needed more they had to ask me, so I could go there, see what they wanted, think about the possible solutions, discuss with them about alternatives etc...<BR/><BR/>This is something that I don't think it's good because it restricts artist's domain, I think it's good because it improves the overall game quality!<BR/><BR/>Lastly, of course when I write a post, and I think the same is also for Christer, saying "X is bad" it doesn't mean that is completely unusable, I know that there are people that use that to a good degree, but still I hope that readers can see my point...DEADC0DEhttps://www.blogger.com/profile/01477408942876127202noreply@blogger.comtag:blogger.com,1999:blog-6950833531562942289.post-66094784244473060582008-08-10T03:45:00.000-07:002008-08-10T03:45:00.000-07:00First, I did not call your post naive. You obvious...First, I did not call your post naive. You obviously know what you are talking about and many of your arguments are sound. <BR/><BR/>What I called naive was the implementation of graph shader system you assume and discuss. For example, if you look at the unreal shader system you will see that the BRDF is burried under the graph system - artists have little control over that. <BR/><BR/>Further you argue optimisation is key, and I agree, but again you just assume a graph based system is going to be implemented naively. In practice about 10% of the code of shader is generated from a graph system and the rest 90% comes from the shader framework back end, lighting/BRDF/ etc. Even those 10% are actually written by programmers although the links are provided from the graph. In reality the code generated from a graph system is written by a shader programmer and can be just as hand optimised to be fast.<BR/><BR/>Some graph based shader systems can provide more semantics than handmade ones and most oftenly the process is more automated than with manual ones. (this of course is based on my experiece with naive manual shader implementations ;) <BR/><BR/>The key advantage of graph shader system is maintenance - i.e. the artists dont have to ask a programmer every time they want to make a minor change to the shaders. For major changes (including BRDF changes) the programmers still need to be involved.<BR/><BR/>Finally, I completely agree that the data-driving everything results in more overhead for programmers - data driven bugs are more difficult to find and fix. So, my point is, choose very carefully what system you make data driven and what part of it you expose to the artists. For your argument you choose to believe people would expose everything in the shader to a graph based system - in reality many people, who implement such systems, know that would be wrong and choose carefully what is exposed and how. <BR/><BR/>I hope you see my point: game engine shader graph system != DCC tools shader graph system.FieldsOfCarphttps://www.blogger.com/profile/17720373803891645622noreply@blogger.comtag:blogger.com,1999:blog-6950833531562942289.post-50355245848789139452008-08-09T14:33:00.000-07:002008-08-09T14:33:00.000-07:00Yours Truly, I would like to believe you, but you ...Yours Truly, I would like to believe you, but you don't provide any evidence about my arguments being flawed. So I don't know which part of my post you see as "naive".<BR/><BR/>That shaders are performance critical? Well that's a fact really, 90% of the GPU execution time is in the shaders, the Larrabee paper provides some intresting details about that.<BR/><BR/>That hand-optimizing such shaders provides a better performance than tool-generated ones? Well, in my experience that's really true. Even if shader code is easy to machine-optimize, still there are plenty of things that can't be automatically done like changes in the packing of the constants and textures, algorithmical improvements, load balancing between vertices and pixels, between number of threads and number of instructions, coordinate system changes etc... I've often seen 2-3x improvements in performance of shader code after hand-optimizations.<BR/><BR/>Maybe you argue that it's wrong that graph-based systems provide less semantic information than switch-based ones, than are still worse than hand-picked ones? And that this lack of sematic makes late, global changes more difficoult? That also seems to me pretty evident...<BR/><BR/>I pushed my opinions further than Christer did, and actually said that graph-based systems are bad in theory too, as they encourage the wild use and mix of empirical BRDF models... In my personal experience that's true, and it really makes the difference between CG-look and realism (and between having 100 parameters and 10, too)...<BR/><BR/>Last but not least, even if technical artists with advanced graph based systems could really do a fine work, and in many cases they do, I hate that approach, it's pushing artists and coders far away each other.<BR/><BR/>Those advanced tools not only have a number of drawbacks, not only require a lot of care to be used properly, but I've seen them in manay many cases be a serious productivity problem.<BR/><BR/>In my company we have a super-advanced animation system, that's totally cool. It comes with a huge C# tool to let technical animators author logic, connecting animations together, emitting and receiving signals and stuff like that. The idea was always the same, let artists modify the game with no coders effort.<BR/><BR/>The result is, even if we have very skilled animators, and animation is the biggest art department here, that artists make a lot of "bugs", coders are still required to fix those, but they have to use tools instead of code, and they hate it, they don't want to learn such tools, plus making code become slower, because each change in the animation system now have also to update the tools and involves touching a huge framework...<BR/><BR/>Even if your experience is different, I still think that those kind of systems are not the way to go, the way to go is having coders working tightly together with artists, cross-breeding ideas, using science and creativity, and that we should try to make this kind of paired work more efficient, not decouple it. That means that coders should be enabled to code faster, and artists to tune the parameters of the code easily. That means that we need fast iteration, small, agile frameworks, easy to be changed, ideally live-coding.<BR/><BR/>I worked for some time in the tools, and I'm pretty skilled now with 3dsMax scripting and stuff like that. I've almost always regret my decisions of building huge, generic tools. Most of the time is so much easier to work with the artists and write a small script tailored at a given need. We are too optimistic when we have to evaluate reusability... well but now I'm writing too much, and I don't want to digress from my main point.DEADC0DEhttps://www.blogger.com/profile/01477408942876127202noreply@blogger.comtag:blogger.com,1999:blog-6950833531562942289.post-28495602280837506382008-08-09T04:15:00.000-07:002008-08-09T04:15:00.000-07:00It is clear from your post that you (both you and ...It is clear from your post that you (both you and Christer) make wild assumptions how such systems are implemented for games and then present these assumptions as flaws. I would recommend investigating/or working with good implementations of a graph shader system in a game engine before arguing against or for them. <BR/><BR/>Believe me, many of your arguments are flawed simply because you assume a naive implmenetation.FieldsOfCarphttps://www.blogger.com/profile/17720373803891645622noreply@blogger.com