Spent one hour and a half talking about the lighting question again, all very necessary stuff but it's slow progress making concrete decisions. You could not find a more subjective topic to talk about, and committing ideas to code requires a very clear understanding of what you want to do, versus what do not want to happen. After the call I was thrust into a further three hours of experimentation and analysis to attempt to create a situation where the surfaces of objects can be overexposed without using ambient light to artificially brighten other areas of the scene which became undesirable. The solution was a new slider value called 'Surface Level', and much like 'Ambient Level' controls the intensity of the multiplier within the shader, but this time for direct light. I also added the remaining shader controls to the new static render effect to include shadow intensity and general ambient and surface colors.
While I was coding and testing this, I set my machine off pre-baking The Escape level with ambient occlusion but with a single threaded approach. I did some experimenting last night and discovered that if I completely eliminate the threading code, and run on the main processor thread, I can lightmap the whole scene without any corruption, freezing or crashing. A major clue, as I now know it has something to do with threads competing for use of the same data and getting it royally wrong.
Of course after 2 hours, when the pre-bake was finished, it turns out that by setting the ambient value to 0.0 instead of 0.6, the expensive occlusion effect was lost as inside buildings there is no light to subtract from. Ah well. Returned it to 0.5 and started the build process again. The reason I dropped it was to allow the real shaders to fully control ambience, but as I want to have ambient occlusion mapping even where there is no light sources, I have decided to use 0.5 as the base-line and then deduct this value from the shader so I can effectively have negative light to apply the ambient occlusion effect again.
For today, in the spirit of getting things done, I have decided not to continue hunting for the threading bug and instead get the static shaders finished off and move onto in-game performance and continued visual touches. These items are now more important than making the light mapping process faster, but it's still high on the list, just slightly demoted while I get the engine into a state which can allow me to make some decent screenshots everyone is happy with.
Currently agonizing over creating a new shader (which will be almost identical to the entity_basic.fx) to allow normals, specular, fog, e.t.c. but with the addition of an extra UV data chunk and a texture re-shuffle. Ideally it could all be in one shader but then I would have redundant resources in there on both sides (i.e. secondary UV not required for dynamic entities, occlusion texture not required for static entities). I also like the freedom of being able to tailor the static shader for speed given it's static state. Despite the terrible 'code duplication' it will create I think I will opt for a specific static shader and just cut and paste 94% of the code from entity_basic.fx.
If anyone knows of a good technique to have 'common shader code' which can then be included into the HLSL file, it would make the above concern mute and would significantly clean up my shaders and also reduce the chance of errors creeping in such as typos.
For now I will proceed to create final static shaders, juggle the code to allow the extra textures in there and tie in the HIGHEST to LOWEST settings so they can change the static shader too. That will then set me up nicely to produce some nice shots this evening, and have the engine ready to do some serious performance profiling with Intel VTune (the core duties of my tasks next week). Would have been nice to share a final render of the combined effects of this work, and maybe I will post one this evening if I am not too zonked, but for now here is me diving back into the land of shaders and putting some of the wires back in the box.