Thursday, 2 October 2014

Another Fine Day

Not only is it shining outside, but it's shining inside too thanks to some cooperate static batching code I wrote not too long ago.  With some other minor lightmapping task tweaks out the way I plumbed on to get common local static objects to group together before the lightmapping step. Worked out much better than I had hoped, and the upshot is that draw call count has been reduced.  You will notice in the following screenshots that although the draw call count went down, so did the FPS score. This is due to the slightly more expensive static shader I am using to lightmap the objects, and the addition of the shadows on the terrain which comprise of four terrain shaped objects.  Once I have optimized the static shader a bit more, I should be able to gain this back and of course over a larger scene with more common static objects I can glue together the benefit increases exponentially over the pretty flat cost of adding the terrain shadow geometry.


You will notice this scene has no shadows and each of those small plants is a separate entity thus a separate draw call.


After the lightmapping process, we have batched some objects together and now we have saved 72 draw calls (really 76 but we added four terrain shadow objects for the shadow casting over the floor).  Almost halved the draw call count, and for scenes which use lots of very small and similar entities, this saving will become quickly apparent.

Of course it's all for naught without a noticeable increase in frame rate, and my mission Friday will be to apply this new batching technique to The Escape level, which is a real game level with real challenges when it comes to what renders are important and which ones are redundant.

There are still some artifacts to deal with such as the brush trig plants casting a rectangular shadow and the glass terrain technique showing a faint edge where the non-shadow area is not completely white (1.0), but I am happy to leave these smaller issues until the larger issues have been tackled.

My next primary targets for performance improvement are inside the occluder and the render order systems.  They are somewhat related and so I am tackling them together, and providing VTune agrees with me, these will yield plenty more FPS points by the end of Friday with a little luck. Some minor tasks also remain if I get the above finished such as reducing the memory footprint of font images and going through the engine to make very small improvements based don notes I have made as I tackled neighboring areas but put to one side to avoid the distraction.

One last bit of news before I sign off and start my usual 4PM team call.  I discovered that I have not been handling my normal maps perfectly thus far. This article really describes the issue:

http://www.codeproject.com/Articles/10897/NormalMapCompressor-A-tool-to-automatically-compre

The problem is that the normal map formats used so far both internally and externally are all over the place, using uncompressed, DXT1 and DXT5.  Do I change all my normal maps (AND SHADERS) to use this new R->A swizzle to get a better normal map. If I do, does that means everyone in the store needs to change their normal map files, or do we have two sets of files, and two sets of shaders, choices choices!  I personally want the best visuals, and this technique will provide it, but it also means upsetting the status quo and potentially a LOT of re-exporting of normal maps on our official assets and the thousand plus entities already on the store.  Interesting times!

16 comments:

  1. Better graphic quality should never need to be compromised in favor for already created items in the store. I think artist must accept that FPSCR is still a beta and that things may still change (for the better).

    ReplyDelete
  2. I vote go for the option that gives the best visuals. In the short term it might cause some upset but the goal is to make the ultimate game engine...? Such an ambition must involve some sacrifices.

    ReplyDelete
  3. Don't hold back for zx spectrum users c64 all the way. Time to up the viz. Artists will i am sure relish the chance to improve their creations. Please up the normal maps etc time to get serious about the engine

    ReplyDelete
  4. We must have the better looking visuals please +1

    ReplyDelete
  5. I agree, better visuals. We all know it is in beta and have expected changes. Let us know the final way. I don't mind upgrading the media in the store that I have made already.

    ReplyDelete
  6. You should really get a consensus by putting a poll on the forums. But better visuals +1

    ReplyDelete
  7. Lee, there is a way for the shader to handle BOTH normal map formats in one shader.

    I use this method:

    float4 normalmap = tex2D(NMapSampler,IN.TexCoord.xy); //sample normal map

    //can account for both dxt1 AND dxt5nm here!
    normalmap.x = normalmap.x * normalmap.w;
    normalmap.y = normalmap.y;
    normalmap.z= 1;
    //expand normal map
    normalmap.xyz = normalmap.xyz *2 -1;

    ReplyDelete
  8. Also, I recommend Nvidia texture tools for doing the A->R swizzle. If you have an Nvidia GPU it is super fast since it uses CUDA, great for batch processing.

    Just copy/paste this as a BAT file in a directory with your normal map files:

    for %%X in (*dds) do C:\WINDOWS\system32\cmd.exe /c C:\Progra~1\NVIDIA~1\NVIDIA~1\bin\nvcompress.exe -nomips -bc3n %%X

    ReplyDelete
  9. Better graphic quality should never need to be compromised in favor for already created items in the store.

    Lol, as I thought everyone wants the upgraded visuals. Personally I can't see any difference between them in that article, beyond the really low quality one.
    It sounds good for memory issues sure, but I would seriously consider anything that would make creating media for Reloaded harder than it already is. Even now it is heavily criticized for being way too difficult to add characters in. We are yet to see any 3rd party characters released as yet, and the difficulty to import into Reloaded is the only reason for many artists not releasing some.
    Considering the rather slow sales on the store. I'm sure some people are doing okay, but not everyone has a following form the original FPSC. I am quite sure many artists will not think converting all their normal maps fun, or even worth their time in some cases.
    So, no, if it means converting items over or they don't work anymore then too late I think to start changing the rules. If anything that can break existing objects is done to the engine, you guys must be prepared to fix the issues yourselves I think, or pay someone to do a long boring job lol.
    Before anyone starts saying it was your choice top start making models for the store. Yes you are correct. However I would say that TGC has asked for people to make store items, and keep pushing how many objects they have on the store, in order to make their product look more attractive to potential buyers. To turn around and say, well, they won't work now UNLESS you spend x amount of hours converting all your normal maps to this format. Well, it's plain wrong. Anyone who has put much on the store will know there is a fair amount of work involved to get one item on anyway! Thumbnails, images, descriptions, videos to make possibly, organizing the objects just correctly to work with the store. The more complicated an object the worse it gets. With the guide given on the site fairly laughable compared to what you need for anything beyond a basic non animated non scripted object.
    I'm all for improvements, but being an artist, I do not agree with anyone here, except Mark Blosser who is the only person to make a good suggestion. To simply say artists will have to remake everything is a fairly selfish attitude to take, and not a way to keep them involved at all.
    Again I don't see any visual improvements to jump up and down about on the linked article, just memory gains due to compression. I really think it should be optional, not mandatory. If people producing games want to go through the images and convert after fine. Just don't expect all the artists to jump up and say "Cool! I can spend a day of my life going through all my old normal maps and converting them now!"

    ReplyDelete
    Replies
    1. It's still beta phase - things can and will change. Live with it.
      I think it would be a huge misstake not making the engine the best it could be, just for some artist are crying.
      You see, FPSCR will probably in the future leave DirectX 9 to support newer versions. Will supporting PBR and such... then you will need to make new models as well.
      Trying to stop development is just stupid.

      Delete
  10. As an artist with some items in the store, I can safely say that changing all of the items to adhere to the new format of normal maps will be a hugely tedious task! However, if we are assured that changes made will not have to undergo the rigorous scrutiny we have already endured, initially uploading the assets; I believe it wouldn't be all that difficult.

    I only have 17 models in the store, but, people like WOI, Wolf, Gtox etc, who have well over 50 models, will not be impressed when told they have to amend all of the models.

    ReplyDelete
    Replies
    1. It's still Beta phase.. things can change. Should FPSCR suffer from that a couple of artist already made a lot of entities? No.
      With that kind of arguments FPSCR would never had implemented LUA :)

      Delete
    2. No, Not at all. Lua was introduced somewhat earlier in Reloaded's life cycle. Plus, was a clearly better option than fpi to the nth degree. No offence to Lee, but FPI was about as useful as a chocolate fireguard in comparison to any language I have ever used, ever, even Spectrum BASIC, back in the 80's. In fact Spectrum Basic run rings around it. Whereas this normal map business is not the same at all. It just uses less memory. Also, TGC hasn't been asking for scripts now has it? In fact as scripts and Lua commands are still WIP and still rough, no one has yet to put one on the store. But to change the normal mapping at this late stage is a horrendous idea. Unless as Lee has mentioned below, Reloaded can use either. In which case we have best of both worlds.

      Being in Beta would not be an excuse to change things quite so considerably, so far in, after requesting media off people ages ago. 1000's of store objects, means a lot of conversion work! Which, as I say, is not reasonable for TGC to expect store artists to convert. I'm also sure TGC would not want a situation where media purchased on the new store will not work with Reloaded. It certainly wouldn't reflect well on the product, regardless of being in beta or not.

      Reloaded would certainly suffer if all the current artists hung up their hats and said well, stuff this for a lark, I'm just going to go with other stores. TGC need to keep model import as simple as possible compared to other engines out there. Perhaps they could add some sort of normal map conversion when uploading to the store? So artist's just have to supply standard maps and the store will convert it on the fly?

      Delete
  11. The idea was suggested that the engine can automatically detect an old-style normal map and auto-convert it as a one-time deal which will take the strain away from artists. It also means we can stick with one shader and one way to handle normals (i.e. the best way). Has been added to the list for visuals work.

    ReplyDelete