Third Party Speed Reports and Waste Twist

Been an interesting day, starting with some great FPS reports from the field. I sent a build of The Escape demo to some internal team members and the report from one machine shows the V1.0085 version running at 32 fps and the new version running at over 90 fps, so we've definitely made a substantial improvement in performance (for some levels).  

Aside from building versions for internal usage, I have spent most of the day writing a small prototype which allows me to manually adjust the weapon direction of the character at various waste twist angles. It's not as straight forward as rotating the X or Y axis as the spine of the character is angled in such a way that such simple interpolation fails miserably. The curve is not linear too, which meant I needed to write something that allowed me to manually tweak the spine angles for each of the angles the character could twist, both left and right. The engine now has 36 entries based on how much the character twists around to shoot while strafing the target, and then I smooth the transition between the two closest sites.  The result is pretty good with a more human, less robotic twist to the character.  The good news is that writing the waste tweak editor and adding it into the game did not take ALL day, which means I have a few hours left to work on some small artifacts that have creeped in while I was distracted with performance.

My plan is to start the first of my major tasks on Friday, which according to my most current schedule is restoring dynamic shadows to the PRE-BAKE system for things like crates and characters (initially using the new glass terrain geometry to retain the performance we have now).  It will be part of my daily paranoia that each functional item I put back in does not adversely affect our new found performance metrics!  Until Friday...have a nice evening!

Meeting Day

A successful meeting day today with both visual and performance objectives reached, and a new set of tasks for the next 3-4 weeks agreed.  The good news is that I have another week to work on performance and remaining visual touches so hopefully I can get some more out of the engine before I move onto the bulk of the next month of work which is what I am calling the 'third pillar'.

My ideas was that three pillars hold up a good product, the performance, the visuals and the functionality. Visuals and performance have had their weeks in the sun, and good things came from it. The last pillar, that of functionality, will focus on adding those features to the software that we felt are critical, and agreed an order in which they should be done. I cannot remember the order exactly due to two pints of Guinness, but it includes extra terrain editing controls, multiple levels, improved explosions, melee combat, the material system, importing custom characters into the AI system (for Zombie fun) and a bunch of IDE clean-up tasks. The result is a piece of software that will closely resemble it's final shape on Steam (minus the CONKIT and MP).

I prepared some side by side shots for the meeting but never used them, so I am posting them here instead so the effort in making them was not wasted.

Hopefully I don't need to explain which side is which, but you will be pleased to learn you get this extra visual eye candy in addition to a jump from 40 fps to 60 fps (which is a sizable jump at these speeds).

I also took the demo to the Ultrabook to profile with Intel Frame Analyser last night but after about five hours, I discovered that it would crash the app unless my app used only a little amount of system memory. If I used a lot, so did the frame analyser, and it would bomb out.  My new plan is to analyse something like Get To The River level which uses a lower system memory amount and I can study the GPU side of things and understand what in the scene is consuming SO much processing.

My task for this evening will be to continue pushing the odd wires back into the engine so it does nothing naughty under regular use, such as not removing a lightmapped object when one is deleted in the F9 edit mode.  Before I crack on with the next series of major tasks, getting what I have done so far squared away with nice commented code and a clean desk is a good way to prepare for the ordeal to come.  I also have the 'multi-core lightmap crash' issue to investigate as well, which will need to be sorted in order to accelerate the feature of lightmapping the test game on the fly.  Looking good so far though, and I am very happy with the work from the last two weeks.

Another One Bytes The Dust

Today was officially "make the demo work nice" day, but I decided to do one last performance tweak before I started on the main purpose of the day, which was to replace all the GetParameterByName function calls with pre-searched handles for all shader constant calls. Turns out this little pearl moved by frame rate from 78fps to 90fps. Another small win for the performance camp!

My main mission was to get the lightmapping baked and working in a fast new version of The Escape demo. Of course it's never plain sailing and I am getting double-object flicker and completely black objects appearing where there should be nicely baked visuals.  Determined to stay here until a solution has been found, but with a little luck and plenty of determination, I will get there.

Big meeting day tomorrow so once the demo is up and running, going to sort out the rest of the bit tasks, clean the office a little and try to get an early night. This plan all depends on some fast work now..so I think it's time for less blog and more code!  Will report on the meeting highlights on Wednesday.

Amazing Community Video Of Reloaded 1.0085

Had to start my blog immediately after seeing this video from RETRO GAME BLOKE, proving once more that the community can often find ways to do things that the guys who wrote the engine thought was impossible.  In his video, you will see demonstrations of a third person body that animates as the player moves, and the ability to pick up and drop items. Both of these features are not native to the engine, they have been scripted with the built-in LUA commands and has left the lead programmer of the Reloaded project wandering how the 'smeg' it was done:

https://www.youtube.com/watch?v=xLGbVhBXr1M&feature=youtu.be

Not to mention the dynamic in-game scaling!  I am officially gob-smacked, and it's great to know the performance issue is not slowing down the community when it comes to creative ideas!

As mentioned last week my main thrust of performance work ended on Friday and the two days this week before my meeting on Wednesday is about cleaning up and putting the wires back in the box ready for a demo to the internal folk. First news is that The Escape used to run at 35-40fps on my machine, and the same settings on my machine now run at 65-90fps.  It was no one thing that fixed it, and you can re-read the blog posts from last week to get clues what was improved, but my personal milestone of playing the whole game and staying over 60 fps has been achieved (phew).  I then decided to have one more day on performance before the clean-up, which I estimate should not take more than one day if I am focused.  This meant I could tackle one or two more ideas I had, including a look at the cost of shader effect swaps, texture reference costs, the true cost of batching and general behavior of the engine.

An Intel engineer put me onto a tool called GPUView which I assume has been used for YEARS in the industry but news to me (naturally). It clearly shows the work the CPU and GPU are doing in broad strokes (actually it's shown in hideous detail but it provides some nice charts that can illustrate the activity of the app quite quickly).  It clearly showed that the app is indeed GPU bound (that is, the engine is spending a LOT of time waiting for the GPU to finish what has been sent to it), and also that the CPU spends a good deal of time sitting around waiting for the GPU to flush sufficiently that more render instructions can be sent.  I am still not 100% proficient at reading all this new data, but it's great to have my fingers on the various pulses of the engine now.

The last big 'investigation' on my list (aside from object occlusion and render draw order which did not get actioned) was to find out if the terrain system (which is notoriously slow thanks to a huge shader) could be increased in speed. The above scene in the version from this morning ran at 135fps with all features switched on.  Not bad compared to V1.0085 speeds, but when I switched to a re-written LOWEST shader to use only a single surface texture (no painting, no normals, no triplanar, no vegmap, no shadows), I jumped to 225fps.  It was clear in my cut and paste analysis that each texture reference added to the terrain shader knocked another 20-30fps off my total.  For a proper analysis I would need something like Frame Analyser or NSIGHT now, but it's revealing to know where a huge performance chunk could be found!  The reason I switched to one texture is that I had an idea to implement something called Texture Splatting which would render a local texture of my immediate area to a render target and then use that as my single texture in the shader. My theory is that rendering textured quads was faster than the seven million texture reads I am doing every cycle, which works out at 438 million texture reads per second at 60fps. Pity there is no time left to implement the idea, but it's very tempting to know that implementing it could give me between 40-80fps extra in some situations.

My actual tasks for today include a final test of the batching system, correction of a crash that only happens after 2 hours of lightmap baking (a horrible one to reproduce) and restoring the dynamic shadows for the editor and higher shader modes.  It's plenty to be getting on with, and is a needed step to ensure the engine stays in one piece.  It's all to easy to break everything to get more speed, but it has to be tempered with ensuring the foundations are still in tact, and that everything that worked before, works still.  Hopefully I can argue for more performance work at the meeting on Wednesday, but with my goal of 60 fps achieved in The Escape demo, and pre-baked lighting in place to improve the visuals of the scene, I rather suspect the vote will be to move to the 'third pillar' which we have dubbed 'functionality'. That is, stuff like more LUA commands, grenades, better explosions, a fully working entity properties panel, finalized widget graphics, the list, as you can imagine, goes on :)

Final Week Day On Performance

Well I've had a week, and I've crushed some bottlenecks and got my demo from 35 fps up past 60 fps which was my goal.  Today is about putting everything back together now so the software has no wires hanging out before the weekend. That means adding a PRE-BAKE vs REALTIME slider panel drop-down so we can choose between the old way and the new way.  Making sure the editor and test game can work as it did before, and also that I can lightmap any time I want when I press F1, that the levels save the lightmapping work and reload it perfectly every time, that it can handle The Escape demo without crashing (all be it on a single thread which causes test game to freeze while the lightmapping is in progress), plus fix an issue that has just emerged in the IDE which is now slowly than it used to be, so that needs investigating.

Also been experimenting with quality settings, with the above showing a quality setting of 2.0 and the one below showing the setting at 0.5.

The effects are so subtle I decided to change the default lightmapper to 1.0 for the moment and come back to the degree of effect the quality value provides.

I have also added a new PRE-BAKE vs REALTIME slider panel drop down box which lets me bounce from these two modes, and I invite you to guess which one is which between the one above and the one below.

Perhaps a little unfair as I still need to line up the intensities of the scene, but I am planning something very specific in this respect mid-October.  Still plenty of performance work I 'COULD' do to the engine, but my week is up and I will be making a case at the next strategy meeting to do some MORE performance work, especially on the GPU side using the GPA Frame Analyser, and perhaps a little NSIGHT as well.  My task list still shows render ordering and object occlusion work as priority items, but it looks like I have run out of time this week. I still have two more days next week before the meeting so more progress on performance still to come.  Pretty happy with this weeks work however, from all the team, and quite buzzed to repeat this next week too!  

Until then, enjoy your weekend and if the sun is shining, try to pry yourself away from the keyboard and go for a nice walk.  At least that's my advice to me, whether I take it or not is another matter.

Another Fine Day

Not only is it shining outside, but it's shining inside too thanks to some cooperate static batching code I wrote not too long ago.  With some other minor lightmapping task tweaks out the way I plumbed on to get common local static objects to group together before the lightmapping step. Worked out much better than I had hoped, and the upshot is that draw call count has been reduced.  You will notice in the following screenshots that although the draw call count went down, so did the FPS score. This is due to the slightly more expensive static shader I am using to lightmap the objects, and the addition of the shadows on the terrain which comprise of four terrain shaped objects.  Once I have optimized the static shader a bit more, I should be able to gain this back and of course over a larger scene with more common static objects I can glue together the benefit increases exponentially over the pretty flat cost of adding the terrain shadow geometry.

You will notice this scene has no shadows and each of those small plants is a separate entity thus a separate draw call.

After the lightmapping process, we have batched some objects together and now we have saved 72 draw calls (really 76 but we added four terrain shadow objects for the shadow casting over the floor).  Almost halved the draw call count, and for scenes which use lots of very small and similar entities, this saving will become quickly apparent.

Of course it's all for naught without a noticeable increase in frame rate, and my mission Friday will be to apply this new batching technique to The Escape level, which is a real game level with real challenges when it comes to what renders are important and which ones are redundant.

There are still some artifacts to deal with such as the brush trig plants casting a rectangular shadow and the glass terrain technique showing a faint edge where the non-shadow area is not completely white (1.0), but I am happy to leave these smaller issues until the larger issues have been tackled.

My next primary targets for performance improvement are inside the occluder and the render order systems.  They are somewhat related and so I am tackling them together, and providing VTune agrees with me, these will yield plenty more FPS points by the end of Friday with a little luck. Some minor tasks also remain if I get the above finished such as reducing the memory footprint of font images and going through the engine to make very small improvements based don notes I have made as I tackled neighboring areas but put to one side to avoid the distraction.

One last bit of news before I sign off and start my usual 4PM team call.  I discovered that I have not been handling my normal maps perfectly thus far. This article really describes the issue:

http://www.codeproject.com/Articles/10897/NormalMapCompressor-A-tool-to-automatically-compre

The problem is that the normal map formats used so far both internally and externally are all over the place, using uncompressed, DXT1 and DXT5.  Do I change all my normal maps (AND SHADERS) to use this new R->A swizzle to get a better normal map. If I do, does that means everyone in the store needs to change their normal map files, or do we have two sets of files, and two sets of shaders, choices choices!  I personally want the best visuals, and this technique will provide it, but it also means upsetting the status quo and potentially a LOT of re-exporting of normal maps on our official assets and the thousand plus entities already on the store.  Interesting times!

C64 Tunes Heaven

Discovered my liking for old C64 SID music again, and been playing them in loops on YouTube :)  Very retro and food for my 8-bit brain!  Today has been a mixed bag of work which included some email sorting but mainly focusing on reducing the overhead from the shadow system. As you know my pre-bake is now assuming responsibility for all static shadows so that means I don't need the real-time shadow maker from doing quite so much work, so today has been about reducing that workload until the shadow slider is really just responsible for controlling the pre-bake shadow intensity.  Once I have reached that goal, I can add in the PREBAKE vs REALTIME slider panel drop-down and re-instate the real-time shadow system for those that want it.

A shot of my shadow test prototype without pre-bake

A shot of my shadow test prototype with pre-bake

The above is the level I am using to conduct my shadow testing, that is, one building on the landscape with some barrels and crates.  Nice and quick to load and easy to confirm the existence of shadows and check frame rate.

Here is another shot I took during the call at 4PM, and a suggestion of things to come with the new pre-bake step in the software.  Nice how the light catches the barrel and stretches into the tent.  Once we have interior static lights from lanterns, candles and perhaps a small fire, it should be awesome!

Have another 20 minutes or so to bash the shadow render from it's current 11% down to 1% (after all it's just one building on a simple render target).  I commented out the offending function highlighted by VTUNE and it went from 180fps to 220fps with no visual difference so I need to make a saving here before moving on.

You would think with the progress made my list is being demolished, but every day I have new ideas and thoughts about performance improvements, and add to the list. It's as long as it was when I started, but at least the engine is better for it and hopefully I can make up a day or two as I start to work on the original items such as static batching of lightmapped geometry and finding out why fonts are taking 64MB of video memory and 50MB of system memory :)