Datasets - large, fast but not cheap and absolutely not easy

Todays task: Make the large data set load quickly, render seamlessly and run efficiently.

In the end, there is just only so much you can do.

This excerpt from an email explains the issue that the users were having when "loading the module": (note that it is edited to protect product secrets while context remains)

.... the [largest module source] uses 360MB of RAM to load the [items] (all 465,000 of them at just over 800 bytes each) from the [data source] and render the schematic on the screen. This is on top of the 170MB that the application usually requires.
With the latest changes, 98% of this RAM is either freed and released or reused when a different [module source] is opened but a user still requires 530MB of heap space for this to run .....