Thursday 25 April 2013

Thursday Tiptoes

Disclaimer

Please don't take this blog TOO seriously guys and gals!  Think of it as a simple development diary of an ordinary human documenting his journey. Stringing me up on the nearest yardarm might seem like a good idea right now, but if I am not allowing to make mistakes, I won't learn my best lessons! I learn more from a total catastrophe than an easy win.  With a little luck, many of these lessons will find their way into the final product.

Progress

The first order of business was to measure memory usage. Before you can reduce the footprint and solve the huge consumption issue, we first need a good system for measuring process that does not involve toggling to task manager and keeping up with the heap size :)

I have implemented a new command called OBJECT STATISTICS INTEGER which will interrogate a new set of values I am generating internally per object, one of which is the actual memory created when the NEW command is used. It restores it when DELETE is used as well so we get an accurate measure over the scope of a specific command.  I tried to using the various techniques to interrogate the heap directly but it's a mysterious beast that does not lend itself to instant interrogations. The best system is to intercept every instance of memory creation and count it, which is why I have overwridden NEW and DELETE for all Basic3D DLL operations.

It's not perfect as it excludes memory creation from things like creating VBO and texture objects, but now I am dealing with the problem internally I can add code quite easily now and track this.  I also made sure that my additions where using the pCustomData structure within an sObject structure so old plugins will still work with the new DBP DLL should I release it at some point. From the pCustomData pointer, I can pretty much create as much additional property storage as I like without ruining the legacy data structure.

Signing Off

Once I have a reliable way to measure and account for memory usage, the next step will be to retrace my steps and visit those locations where obvious memory usage occurred, and ask the key question 'is this necessary?'.  Also, apologies to readers who don't want all the technical jargon and horror stories, alas this is the real and gritty world of development :)  

I would also like to tip my hat to the honest and gritty comments provided so far regarding the memory issue. It is EVEN MORE clear to me that this is a 'deal breaker' within the FPSC community, and you can consider my efforts redoubled to solve it for ya!  Physics and shaders will have to wait until we have huge levels and an inexhaustible capacity to contain your content.

7 comments:

  1. Hi Lee,

    I feel awful now. Having read the technical jargon which is great to follow and knowing its so easy to critisise which is easy for mere mortals like me. Apologies.

    On an upbeat note I have loved the recent screen shots lighting/shadows which look perfectly top notch to me. Would suit me fine anyway as shown in your shots and over the moon with that.

    :-)

    ReplyDelete
  2. Thanks Peter, but keep up the grilling!! I need it ;)

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. " I also made sure that my additions where using the pCustomData structure within an sObject structure so old plugins will still work with the new DBP DLL should I release it at some point. From the pCustomData pointer, I can pretty much create as much additional property storage as I like without ruining the legacy data structure."

    Yes. Yes, I agree with you. Have you considered reversing the polarity?

    ReplyDelete
  5. Yes that's what I decided to do, if the CustomSize DWORD is over 400000000 (basically a negative number), it tells the object it carries statistical data such as that I might need to hoard in the object. No sense deliberately creating a problem for later :) I am pretty sure no-one will (or can) create custom data for an object that is 4 billion bytes in size!

    ReplyDelete
  6. 3.7GB of custom data per object? Lol haha I doubt it :P

    The only thing that concerns me is your adding more stuff to memory in order to calculate memory usage...is this a good idea?

    And for the record, I love the technical jargon and horror stories. It makes you seem more like a real-world person and programmer ;)

    ReplyDelete
  7. The technical jargon and horror stories are not a problem:). Me personally i am a programmer and i wan to read the latest progress of reloaded. I am very interested in new information from you

    ReplyDelete