Sunday, 9 March 2014

Weekend Research

Work

Did a little research over the weekend on the subject of lighting models in relation to performance. There is an argument for implementing something like a cluster deferred lighting renderer which has the potential to reduce the 'lighting/shadow cost' which was the route cause of the present performance issues. Alas such an implementation (done correctly) will require an upgrade to DirectX11 and a new deferred rendering pipeline which combined would take a few months of dedicated coding. That is, nothing else would get done while this work happened, and the kicker would be that some users will not even notice the difference, aside from some frame rate changes and the benefit of adding thousands of lights without a performance hit.  It is a thankless task that has more long-term benefits than short term goodies.

I expect everyone wants performance yesterday so will not be willing to sanction a six month sabbatical while Lee buggers off to re-write the entire graphics engine. To that end, the smart course is to finish the optimization work on the DX9 engine and get it as fast as it needs to go, so that when we do upgrade to DX11, we still have a very good fall-back for those users still using Windows XP and Vista (DX11 won't work on those OS platforms I think). Moreover, DX11.2 only works with Windows 8.2. See the pattern ;)

Anyhoo, the reason for my quick weekend blog is to write down a small idea I had about the shadow system I am working on. Right now the fast entity shader (LOWEST) does not use the dynamic terrain shadow texture due to the relatively low resolution texture and the lack of any meta data in the shadow texture to work out whether to shade entities at higher elevations (i.e. the roof gets a shadow when it should not).

My simple (and fastish) idea is to feed in the texture holding the height map data of the terrain, which will give each XZ coordinate a world space height position. I then write how 'deep' the shadow pixel in the dynamic terrain shadow texture is instead of just black/white. From these two pieces of information, I can work out whether a single world space position of the entity is in or out of the shadow being cast.  I would have to increase the texture size of the dynamic shadow texture to get a better finish, and there is a concern that the extra per-pixel calculations and texture read might create some drag factor in terms of performance, but the theory is sound in my mind. It's not a lot of work and it would mean entities get 'almost' true shadowing, just as terrain and grass currently receives.

I have all day Monday to work on this, plus the other ideas I have, and the big job of getting it all on an Ultrabook as my meeting is over 100 miles away from my main machine. It's a good test however as GDC is even further and this trial run will be very revealing.

Play

And now, I will forget all that stuff and see if I can boot up Thief and continue my pilfering in the dark and rain soaked streets of what looks like London. It's possible they put Big Ben in there for the 'pending' UK tax breaks. It will be interesting to see UK developed titles in the next few years coming out with all manner of Britishness crow-barred in. The next time you play a fast paced zombie-horror blood-splat gore-fest shooter, and have to consume 'cream teas and buttered scones while affecting a cockney accent' to restore your health, you can blame the politicians of Europe! Interesting times!!

Friday, 7 March 2014

End Of A Good Performance Week

Work

As the weekend approaches, I wanted to end early (midnight) today but it's now gone 4AM. Apart from making the vegetation and entity shader follow in the footsteps of the new shadow system, I've spent probably far too much time tweaking the lighting and pixel effects of the LOWEST shaders to get as close to the HIGHEST ones without adding to the performance hit. It's educational, but it's slow work.

Here is the before shot with everything set to HIGHEST and using the expensive fragment shaders:


Here is the same shot but using my new LOWEST shaders:


The terrain and grass are rendering shadows, and the entity approximates a shadow effect (but I want to do more here somehow/somewhere). Both are rendering all four cascades and the fifth dynamic terrain shadow texture but the LOWEST has a few more tricks in that I can completely switch off the cascades and only draw to the fifth texture when something moves. I did some tests prior to these shows, and I could get another 40 fps by switching them off without loosing my shadows.

The clock has beaten me (once again), but I have had lots of extra ideas on top of what I have now including the addition of meta data into the dynamic terrain shadow texture (DTST) to store information about the shadow being cast (very similar to deferred rendering but with local render targets). This extra info would allow me to shade entities 'above' the floor surface such as tables and things under canopy. I had it 'mostly' working without this, but the tops of entity roofing got shaded too which was a bit displeasing.

I also thought of reading the maximum texture size allowed on the card and then create the DTST to that size, giving my shadows greater resolution. My GeForce 9600 GT can create textures 8192x8192 large, which will increase my shadow resolution by a factor of four. For terrain and grass it is not too noticeable (but enough), but I really need a higher resolution for the entities!  It may be straying into visuals vs performance though, and there is much to do yet on the performance side (despite the early good results).

I have still to research the static vs dynamic DTST idea to avoid rendering ANY static entities after the initial blast, and using a different texture format for the DTST to reduce the memory it takes (16MB right now, 262MB if I use a 8192x8192 texture). Ouch. With an 8-bit format, this would drop to a more friendly 65MB. AND I want to see how much I can move some of the pixel shader work into the vertex shader to increase calculation efficiency. So many ideas, too little time!

Before I turn-in, I will leave you with a video monologue I made this afternoon as I was attempting to explain the new shadow system. I think it merely serves to confuse everyone, but it's material you might like:


Have a good weekend, and if I get up in time, I might have one too.  I just realized "Thief" (reboot) has been released on Steam and I had it pre-ordered, so I think I will play a few hours of my all time favorite franchise as a little treat for getting some serious performance work done this week.

Thursday, 6 March 2014

Long Day - Early Night - I Wish

Work

As I was forced out of bed at an ungodly time (NOON), my eyes are telling me that 1AM is the time to stop working.  I can't burn the candle at both ends like I used to (at least not the candle I'm currently burning).  Progress has been steady all day, but in the last hour it has taken on a slightly sour note. My main task of creating a faster shadow system is fine, but when I promoted the code to the main editor, the engine crashed when you save levels. You could not make this stuff up!  And worse, it's one of those D3D9.DLL crashes that leave zero clue as to the root cause. Friday will certainly involve lots of undo tweaks to see when the crash stops - grrr.

Anyhoo, the boon of the day was to be a new dynamic terrain texture generator using quads and cascade detail, but it would have taken days and there was no guarantee it would give me a substantial increase once the new code and it's performance hit was taken into account.

I decided instead to target something I KNEW would give me a boost, which was eliminating the expensive terrain shader without loosing my shadows. The idea I had was to collect the shadow information in a large dynamic terrain texture image directly from the cascade shadow map information and then use that data at the more extreme distances. It would effectively replace 90% of the expense in the terrain shader with a simple texture read. Everything has gone smoothly for the most part, but getting the shadows to line up with the terrain heights is proving a 3D headache.  Not being able to save my test levels is the icing on the cake of this headache :)


To cheer you up, I can report that the old terrain shader would run at 53 fps with shadows on. The new 'flaky visual' version can run at 131 fps with shadows on. The shadows are more blocky and out of place right now, both to be corrected, but the performance boost is undeniable.  Let me repeat, the shadow slider was not switched off, it's still rendering 4 cascades and I still get 131 :)

The image above shows the system as it was being built. The blue square is the new dynamic terrain shadow texture and the black square inside it is the current largest shadow cascade area in the camera view. The white dots are the shadows cast from the buildings. As you move around the scene, this blue texture is updated with the shadows from the cascade and slowly builds a picture of relevant shadows that the lower-powered shadow renderer uses. Once I have everything straightened up and looking pretty again, I will write something which will populate this dynamic texture at the start of the level so even distant shadows will be rendered (as you would expect).  I have some other ideas about using 'multiple' dynamic textures for higher quality long-term storage of terrain shadows but I want to get everything back together first.

So in conclusion, despite the engine having wires hanging out of it and crashing, I think I am on the coat tails of a serious performance improvement and if I can get the visuals comparable to the expensive per pixel fragment shader version, we'll be laughing.

Wednesday, 5 March 2014

Strange Frame Rate Fruit

Work

A rather mixed bag today. Once I had dispensed with the first five hours of the day on menial stuff and nonsense, I began the adventure of performance seeking in earnest. My plan was to re-introduce the quad system, but tie it to the individual static objects in the scene, so in addition to their LOD transitions they would have quad's at the furthest range. This was accomplished relatively quickly, but the next part of my evening would be filled with non-quad related musings.

It turns out that after adding the quad system, and as a quick test replacing ALL static objects with their quad equivalents, and ensuring that the quad textures or quad vertex buffers where not being locked, I only gained 4 fps for my trouble on a GeForce 9600 GT using the 'run to the river' level with everything set to Low. That's right, I went from 141 fps with real objects to 145 fps with quad replacers. You could have knocked me down with a feather. My knight in shining quad armor turned out to be a total faker. It was remarkable in that the 141 fps was pushing 134K polygons with 225 draw calls and the 145 fps was pushing 86K polygons and 132 draw calls. I achieved my goal of halving the draw calls but I did not get my reward of more frame rates. I should also mention that I switched off my built-in occlusion system for this test, initially as a way to un-bias my results but in fact when I did remove my performance helping system I went from 66 fps to 141 fps. Sometimes the universe likes to have a huge belly laugh at my expense!

Getting 163 fps on a GeForce 9600 GT - It IS possible but at too High A Price

Undeterred, I decided to abandon logic and look for something that would give me some more performance. Having done everything right by adding occlusion, quad rendering and other object thinning methods and not get a prize in performance, I decided to spend an hour running a battery of daft tests until I saw a big jump in performance.  I finally found one such spike, which happened when I moved the camera to look at the sky but not so far that the terrain and ground objects became invisible. I noticed that the more terrain was rendered, the more frame rate drain occurred.  I then replaced the terrain shader with a single color draw and the frame rate went through the roof.  It seems the single biggest performance killer is my terrain shader, which has the job of painting most of the 100K-200K polygons in a typical scene.

The good news is that I have recruited someone (Dave The Ravey) to help me reduce how much terrain is rendered in the first instance, but now I know the shader is a crucial bottleneck to fast gaming, this is the focus of my performance hunt on Thursday. Unfortunately, I have to take a phone call during daylight hours so have to cut short my development tonight and resume when I wake.  My initial thoughts are that rendering every terrain pixel with my intense terrain shader is just daft, and that if I can render a cascade of terrain textures on the GPU and then feed those textures to the terrain rendering, it will all but eliminate the drain which can take a frame rate from over 300 fps down to 110 fps.  Naturally, whatever I use to build the dynamic terrain texture will cost something, and balance the scale a little, but having experienced almost no drop in performance when rendering LOTS of quad textures, I think I can get away with it.  There are some big holes in this approach however, such as no dynamic shadow information getting to the terrain shader, but I think I can re-channel the shadow to the dynamic terrain texture and get the shadows back.  Using this new technique, I will also be able to introduce 'texture splatting' as a freebie feature, allowing more than four textures per terrain :)  I won't be doing anything on this feature until performance is solved, but it will be a great bonus if this new technique works as I expect it might.

I think you can start to appreciate why some game developers decide to buy a middle-ware engine for $250,000 and skip the whole process of figuring out the best way to do everything :)  I certainly can, but I won't be beaten!!

Tuesday, 4 March 2014

Crash Alley Tuesday

Work

Managed to delegate some more, reduce my inbox count and get an internal version out the door for more testing and veg video production, but the main headline story is the sequence of inexplicable crashes I have had to wade through. You all know the frustration as developers of not getting to the task you want to start because of a handful of totally mysterious bugs that need fixing first. Well that was my day. Even after 30 years of coding I can still be tripped up by a crash who's cause is the random corruption of some data at some point in time, and the crash event sheds no light on either.

Add a new flag in the meantime called "dividetexturesize=X" which scales down the textures loaded into the engine. I will have it default to 2 so that lower end machines are not choking their GPU video memory with 2048x2048 textures, and those with higher end cards can simply adjust this value to 1 for the highest resolution textures. In the future I might introduce some modes which do not reduce very small textures as they have much less impact on overall video memory usage.

I've just had a bath, and waiting for a Eureka moment on the crash issues, but none came. I have thus decided to approach the problem methodically. I will totally analyse the area of memory that the crash occurs on, and record as much as possible the before, during and after states, and setup monitors of the data around the memory block in case something else changes it.  It's long-winded, but it eliminates the need for intuitive guesswork, and if I can finish off today (by 3AM) with a fix of this most illusive bug, I will be a happy chappy.

My plan (for Wednesday now), is to skip the LOAD OBJECT memory work (which would only yield a small saving overall) and go for the huge performance gain of making all distant objects QUAD buffers, and then extend it to shadows, reflection and light ray cameras. It will reduce draw calls by MORE than half for the same visual, so it's well worth it and early tests show a marked improvement in FPS, even at lower levels.  My run-to-the-river currently runs at 10fps with everything on and 15fps with some conservative reductions. The game plays well at 30-35fps so that will be my initial aim on the GeForce 9600 GT card I am currently using.  I also plan to recruit one of my coders to help me with some terrain work which should vastly reduce the memory and performance footprint too.

Monday, 3 March 2014

Monday Already

Work

Having crunched code over the weekend, it feels odd that Monday is my first day back.  Despite the strange feeling, I set to work obliterating my backlog of emails, sorting out the work for the next two weeks and getting some key issues resolved in the engine.  I have sent code off to Simon to add shadows to the Construction Kit and a new version to Rick so he can make some nice videos about the new Vegetation Pack.

Still the best news is the massive memory savings I made by loading the HUGE textures used by the engine into GPU Video Memory. The consequence is that I can make seriously large levels now and the system memory creeps up very slowly compared to the old version. I have ideas to make even more savings too, but as much as I want to chase that particular tiger down, the elephant in the room (performance) remains the highest on my personal snag list. I will be running a demo of Reloaded on an Ultrabook in exactly two weeks time in the biggest developer conference on the planet, and I don't want egg on my face. Yes I will be saving face, but you will be getting some serious performance boon as a result so I think we both win.

I also got an email asking for clarity on the license terms of media provided in Reloaded, and the legality of releasing it as part of the standalone build. The official TGC EULA for this product allows you to distribute the assets (encrypted or otherwise) providing it remains part of the standalone demo you built. If anyone remembers the culture of sharing FPSC Classic games, same deal. We do specifically exclude the extraction of those assets for use elsewhere, either as part of a library or other game creator, but within the context of using the assets to show off your Reloaded creations you are quite safe.  It's also great to see the maturity of the community to ask such a question as it demonstrates your respect for copyright material and the terms of use applied to digital media.

Saturday, 1 March 2014

A Change Is As Good As A Rest

Work

It seems my buss-mans holiday paid off this time. Within two hours of hitting the code again, I discovered that although the LOAD OBJECT command was gobbling a little more memory than it should, the real villain of the piece was LOAD IMAGE which had been placing MOST of the texture content in system memory as a managed backup.  My building ate 23MB in the old Image DLL, but with some extra code to ensure certain commands could still lock a video memory texture, my new Image DLL loaded the same building in at 9MB. Multiply this effect by an entire level and you will start to appreciate how this single tweak has improved your lives in Reloaded land :)

My investigations also saw the LOAD OBJECT memory usage double when the LOAD EFFECT applied it's magic to it, so there is something very suspicious happening there too.  Going to stop my weekend work for how, and celebrate my code win with some good, and Sunday/Monday I will chase down the reason for the shader taking my 3MB model up to 6MB (and then find out why my model was 3MB in the first place given that the vertex data cannot be more than 1MB).

Play

In order to think laterally, you need to approach a problem for an entirely new direction. Leaving your code for a week, then returning to it is one way to do that, and leaving it to talk geek for a week is probably the best way to prepare your brain for the comeback.