Work
If you want good news, best not to read any more of this blog and wait for later in the week :) If you are brave and want to learn about the real world of software development, read on..
I integrated my new occluder into the Reloaded engine (you remember, the one that did amazingly well in my simple one occluder prototype). Only took a few minutes really, and the results where rather poor. My un-occluded scene I created from 50 buildings rendered flat out at 194 fps. This was my benchmark. I then added the code to submit all buildings as both occluders and ocludees and run the occlusion system on the exact same scene view. My frame rate dropped to 101 fps and my polygon count jumped by 90,000 polygons. Oh woot!
I did some investigative tests and it seems when I removed the 'GetRenderTargetData and System Surface Lock' commands, it jumped back up to 186 fps. My conclusion was that the 'slight' GPU stall I was anticipating is in fact a huge stall when you are running at frame rates in the hundreds.
When I put it back in and reduced the depth buffer rendering to one draw call, it was slightly better than the worst score at 129 fps. This means the GPU lock is the biggest spender and the fifty depth scene render calls is the next cost to bear. As I could not avoid the big stall, I worked to combine all the 50 building geometries into one large vertex buffer to do a single draw call but this only yielded a marginally better frame rate of 105 fps.
The good news is that this overhead will not get any bigger as the scene grows in size (with a few optimizations I have in mind). The cost is in the depth scene render and the GPU stall, and those won't get any bigger which means I can throw thousands of objects at it and the cost will be the same (or near as dammit) as ten objects. I cannot rely on that assumption until I have field tested this new occlusion system with some other machines (and other users).
I then created a second level which had 25 barrels hiding behind a tent, and the system correctly occluded all the barrels when I stood behind it, and the draw call count dropped respectively.
The good news ended pretty soon though as the barrels exhibited visual popping because they where trying to occlude themselves (not a good idea to have an object that is both occluder and ocludee), and the last thing I wanted to see what popping (the whole reason this new occlusion system was created).
More Work To Come
If you read this far, I will now treat you to the very good news. I half anticipated all the issues above, and despite the laundry list of woes I am quite pleased with how the HZB is able to work out occlusion and distribute that through the engine. My next plan is to create a 'preferred occluder' system which only selects near 'large' objects for the occluders which will stop that annoying popping and speed up the depth render stage. I can speed up the management of the return results by allocating some fixed memory instead of creating and deleting the memory allocation every cycle and perhaps my MOST AMBITIOUS plan of all, to eliminate the GPU stall.
I've hunted around I could not find any clever white paper which solves this issue, and prefers to put you onto DX10 and DX11 to solve it with the much friendlier stream-out operation. It seems DX9 coders are left to fend for themselves with this problem, and my idea is utterly radical and perhaps just as slow as the current GPU lock. You are welcome to stop me if you think the idea is mad..
Instead of doing a 'GetRenderTargetData' command to get the visibility results back into CPU memory so I can switch objects on and off, I instead redirect the occlusion visibility texture (which contains little 1's and 0's for each object being represented in the scene) and pass it to my entity shader as a new texture. I then use what is called a vertex texture fetch to grab the visibility state from the texture produced by the occlusion system. If the value is 'not visible', I simply adjust the vertex position to 'behind the camera' which will force whole object to skip sending the polygons to the fragment shader. The draw call for the object would still be made, but the shader would quickly reject all its polygons and move on. Not sure if a draw call that renders no polygons is a freebie, or still a performance drain, but that plus the vertex texture fetch are the two problem areas I anticipate. If my fears are unjustified and the performance hit is negligible, I will have created an entirely GPU-only occlusion method in DX9. The reason I am confident is that the current method produces a GPU stall that effectively halves my frame rate, and the benefit of a non-stalling occlusion pipeline will be apparent the moment I finish coding it and run a test (fingers crossed).
It's another 3:30AM in the morning, and no sight of my normal 9-5 day so far, but if we can crack this occlusion question and have it perform splendidly in the main Reloaded engine, I can draw a clean line under it and move on with confidence. No sense moving on until then (unless it starts to gobble up weeks!). As a fallback plan, I emailed a middle-ware company that provides one of the advanced occlusion methods used by Unity (apparently) to ask them for the price for including their tech. No reply yet, but from experience the answer is usually (you cannot use it in a game maker) or a number with many zeros on the end.
Still, the occlusion system seems to be holding up very well, and if the frame rate never drops below 80fps, even with the current stalling system, it is still a benefit over an engine that would otherwise slow down as you start hiding objects around your scene. Plenty more occlusion news to come, watch this space!
Play
Wanted to go to bed three hours ago, but could not let go of this damnable algorithm. It's now 3:15AM, my eyes and head are throbbing, but I finally bashed the HZB occlusion system into a working state and my little box prototype runs with full 'super fast' occlusion.
I created a world with 400 boxes on the floor, and one large central box which I set as an occluder. Without occlusion my frame rate was 160 fps, due to the full textured shader I applied to all of them. When I switched on automatic occlusion, it jumped up to 350 fps with no change in visuals. Of course when I ran up to the wall to force full occlusion, it jumped to 1000 fps. Thanks muchly Mr HZB!
I have yet to see how the system handles more objects and more occluders, but as the system is predominantly a GPU operation scaling up should be almost free on higher end graphics cards. Time will tell if this approach to occlusion works on the lower end cards.
I've probably binned half of Monday with my 'burnout' weekend but I think it was worth it, and now I have the theory running well in a prototype, the next step will be to transfer the few occlusion commands to the main engine and see what kind of 'before' and 'after' frame rates I get. Fingers crossed.
NOTE: I also dealt with a few emails over the weekend too, including one that helped me fix the scaling issue bug causing SCALE entities to lose their scaling when you load them in or save an executable. A five minute fix, but testing with the R4D5 model was lots of fun!!
Play
For my leisurely weekend, I decided to tackle the problem of Occlusion once and for all, and to that end spent the first six hours reading every technique every used in computer games from CPU software rendering through to insanely ambitious prediction systems for reducing hardware query checks. The reason I am not satisfied with the current hardware query system in Reloaded is that the object 'popping' you see when you run around corners is not just 'occasional' but pretty much in your face! No-one has come up with a decent solution to solve the occlusion query popping as it is an inevitable result of having the data for the occlusion cull one frame behind the rest of the visual rendering. Only coherence systems attempt to solve this, but it only amounts to guesswork. I even pioneered some non-white-paper thoughts about creating multiple queries as a product of the player running in several dead-reckoning vectors, but I quickly realized the cost of those queries would be insane, and there are a lot of articles out there which feverishly attempt to reduce query count so why would I deliberate make multiples of them! Here are my brain storming notes from the day:
During my researches, I stumbled onto a technique called Hierarchical Z Buffers (used in several top end games you know) which basically renders a smaller version of the scene into a depth buffer and squashes it down into a real-time mipmap. It is then used in concert with any object you wish to test the occlusion state of using some pretty wicked GPU based test. The wicked part is how they get the DX9 test data back into system memory ;)
I did not wish to go this route originally as adding another pre-render to the scene seemed like spending more GPU time than saving it. It's now nearly 3AM and I want to eat and watch a movie before I sleep, and I am currently at the point where most of the C++ code is in a new set of DBP commands and I just need to establish some occluder candidates, create an object bounds database and then set the object visibility flags. These last three steps should allow me to choose key occluders from any Reloaded scene, then sit back and watch as the GPU performs full occlusion testing with a single draw call and instantly rejects thousands of arbitrary objects from the scene :) I will know more come Monday afternoon, but the early results are exciting!
I have recently discovered how to use Microsoft PIX properly (now replaced with the VS graphical debugger) and have been having a whale of a time watching the scene built up one draw call at a time. I plan to use this tool A LOT now I know where all the buttons are. For those who just fell over in shock, I do admit that I was one of those coders who mostly 'guessed' where the rendering problems where in my engine. The upshot of course is that these days my guesses are pretty accurate!
Work
Sorry for the intermittent and less than relevant blogging, it's pretty disconcerting to know your workplace can disappear at any moment :) Made some progress speeding up the AI system, fixed some erroneous crashing on larger character levels and managed to track a memory leak directly connected to the DarkAI module. The Memory Management Module (MMM) is currently underway thanks to Dave Ravey which means the smart money is for me to switch to QUAD rendering of distant instance objects to get some serious performance back. I can then return to memory issues on Monday with my new 'rough and ready' memory reporting tool and see where the leak is, plug it, and more importantly find out where 700MB is going for a blank level :) Also progress being made with the 400 Reloaded ready models as part of your Gold Pledge so watch this space for news on that, and some nice progress in Construction Kit country (thanks to Simon), so despite the lack of electrons, we move ever forward. As a final note, I have also made contact with a few users who have been experiencing extreme problems with their larger levels, in an attempt to identify, isolate and correct the root cause of their pain. The next update WILL be good, watch this space!
Work
Wednesday afternoon saw fallen trees and powercuts. Thursday looked like we had full recovery, but come 7PM another powercut. It's back up now so I thought I would post a quick blog before it went again. Got some good work done on speeding up the AI part of the engine and firmly focused on making sure some of the more aggressive FPM levels run at a cool 60fps on my machine. This should produce some nice performance improvements for everyone else and ensure we hit the mark when it comes to making user levels playable and fun. I will report more when I can be sure my blog post does not get wiped out mid-type ;)
Work
Here is a snapshot of some of the items on my list that I am working through for you. If you feel I have missed anything critical out, please let me know!
A FEW ITEMS ON MY LIST FOR THE NEXT UPDATE:
Frame rate. Better frame rates if possible.
DAVE to write MEMORY MANAGER and integrate into DBP/Compiler source
Physics. Player physics seem to be all over the place of different PCs.
Shooting. Enemy shooting has to be sorted once and for all.
Full compliance with Win 7 and 8 (UAC)
Item limit when you place a item it increases the memory bar when you delete it its still using the memory
Map blink/screen blink I had this a few time I found what fixes it is when you load the editor wait till you see the blue screen
Jumping/flying the jumping and gravity is spot on the only issue is when the game lags it makes the player seem as if he is flying
Lag/low fps when going into buildings
Enemies Walk Into Water still the seem to fall between the land and water the shoot from under the water
Ai wont die I believe this is down to fps also as when I take out a few AI they are ok
Fps I believe it shouldn't be limited to screen refresh rate it seems to cause more problems
Selecting entities on top of other entities not working (cannot select key sitting on table)
Ability to range select within the editor to move/delete many entities at once
Ability to turn off skyboxes in SETUP.INI
Ability to specify terrain size in SETUP.INI and have terrain created to suit that size
dynamic entities fall through the terrain
In most cases the AI has no idea I am there (see HOOD1 FPM demo)
you can not place a character on top of a building
Introduce new QUAD Reducer (load time only - no modifications afterwards)
Script commands for (only if time moving, visibility, sound loop), animate, spawn
Reduce the amount of memory used in core engine and as levels are added to
Raycast terrain for enemies
Shoot raycast use geometry, not bullet polys
Shoot through transparent objects
Stop enemies from going NEAR water boundary (can be a sharp fall off and enemy drops)
Stop enemies from climbing steep hills
Blood splats on enemies some times show as dust
Stop enemies from running on spot / straf too much
Enemies to partrol if they hear a sound from distance
In combat, remove slow inertia
Collect dropped weapons from dead enemies
Enemies to look around when hearing a shot from distance or close friend killed
Enemy to react a little when shot but not dead
Player water splashes
Big entity splashes
Slider for sky scrolling speed
Reset objects in test mode
Cloud overlap issue raised by Rolfy (email send to Lee)
Old levels did not run due to old.dbo and .bin files
DVADER Issues - placing items on table / in buildings
Shooting enemies does not always hurt/kill them
Play loop sound for atmospheric start - and add script to player start to trigger it
Switch which activates(opens) a door
Stand on pad which activates a stone block to slide to one side (move entity command)
Work
Just got back into the 'hot' seat and going through my emails and post histories, and I am in full agreement with the posts in the forum that the next few releases should be heavy on core stuff. Personally, I would like to focus entirely on core to the exclusion of all else, but development life is rarely that pure
I'm downloading the HOOD demo now, posted on the Reloaded forum, to get a feel for the feedback first hand, and reading the thread you will get no argument from me on the items mentioned. More performance and proper AI seem to be high on your list, and mine too, so I'll be looking to push these as the highest priorities in the development weeks to come. I have also extended the testing period for the next update, so you only get it once a few more eyeballs and a few more days have tested the release. I have a development meeting Wednesday to discuss the exact order of importance on which core issues will be addressed for the next release, and in the meantime I will be knocking on the head the 'easy to fix silly ones' that crept into the beta since the last version.
I would like to post on the forum more, and I do jump in once a week to reply to every post on the first page (when I am not skiing), but I feel going to the forum every day might detract from actual coding and I usually end up repeating myself after a while.
Hopefully this blog provides enough information on the daily stuff and a 'once a week' appearance in the forum will fill in any gaps from me. If you have any recommendations how I could further reduce the apparent communication gap without adversely affecting my coding hours, please get in touch!
I came back from my vacation with a few more ideas how I could get more speed from the engine, so unless I am molested by higher priority items, I will be spending today and Tuesday trying those out (along with fixing those silly items I mentioned).
We also had the foresight to get some parallel developments coded in my absence, so I have returned to a new installer revision which will cleanly uninstall all your installed files from the next version onwards, but leave your user content and levels in tact.
We've also rewritten the LUA scripting engine module, which now gives us full control of the underlying source code, not to mention adding some nifty extras like multi-scope LUA scripting which will eventually allow you to create scripts for HUD and interface displays, plus other independent logic within the game (more on that in the future). Other work has been done for the Construction Kit and Object Importer, but I am strongly inclined to defer these releases as there is an integration cost that would distract me from the singular pursuit of core work.
Time for me to open up the engine now and start working on performance and the sillier core issues in there. If you feel I should be working on something else until Wednesday (as it's very likely I have forgotten stuff since the start of this post), please let me know by posting here so we can get a general overview of how you would prioritize my next two days I will post again after the Wednesday meeting to let you know what priorities were decided (or you can read the blog post on that day as it will be repeated there as well).
For those with some patience left, thanks for sticking with us. For those who just ran out of it, welcome to the world of game engine development and I'll see what I can do to restore your faith. To take the guesswork out of it, please post here by Tuesday PM, one item that you would like to see in the next update and I'll add it to the meeting on Wednesday.
