Another problem is the performance. If you are dealing with log data in the range of multiple 100MB, adding a new view (which is, essentially, a new tab for the log data with a filter) would be way too slow (as this requires going through the entire list of log packets and checking if the filter allows/denies each packet). There are several other cases like this in the Console.

And there are even more problems, such as updating (we allow adding comments to the log, for instance) or deleting data from the log.

Coincidentally, we are experimenting with the very same idea in a slightly different form. I think the string problem is largely solved by the string pool. The next problem deals with the attached data of log packets (log packets can transmit additional data such as images, hex dumps or database query results etc.). To avoid storing this data in memory, we thought about adding some kind of file-based database support to the Console (for example, with Sqlite). A file database would simplify delete and update operations and should provide a better performance than direct file manipulations.

This is an even older solution, and it can handle data of practically unlimited size. The code required is usually dead simple (replace fields in whatever object structures by methods that fetch the actual data).
As for performance, let the windows file cache do the job for you.

Patrick: Indeed a nice solution. A disadvantage I see is the latency/increased response time of the string pool operation when you are doing your sort/resize/merge operation. However, I’m sure it’s just a theoretical issue and not really noticeable in practice (depending on the amount of managed strings).

For each addition to the pool, we first searched the pool to prevent duplicates. When a string was already present, we returned that string, instead of the one being added (so the reference-count of the former was increased, while the latter was ultimately cleared).

The pool started out as one single array, containing 16 unsorted strings. (Searching in this array required a check at each element, but on average this array was half-full, so it only took 8 compares on average.)
As soon as the unsorted array ran out of space, we sorted these strings, and moved the sorted array aside. (Sorted arrays can be searched more quickly, by using a binary search).

As soon as the unsorted array filled up again, we did something extra : Not only did we sort the 16 strings, but now that we had 2 sorted arrays of equal size, we merge-sorted them into 1 array of size 32, and cleaned out the sorted size-16 array.

When the next fill-up of the unsorted array occurred, we could again move the newly sorted size-16 array aside, as there was no size-16 array in use anymore. But this wasn’t possible on the next fill-up anymore, as we had 2 sorted size-16 arrays and 1 size-32 array… I’m sure you can see a pattern arising here : We just merged all those into one single size-64 array!

This pattern continued for ever larger powers-of-2, meaning that presence-checks became a bit more expensive when the pool grew, as we had to do a binary-search in an ever increasing number of sorted, power-of-2 sized, arrays before we could conclude a string was not yet present. (A hit resulted in an early exit ofcourse.)
It’s not as bad as it sounds thou – I believe we never went past 25 levels or so.

For speed and fragmentation-prevention, our implementation didn’t actually clear out the arrays – we kept all arrays sized to their alotted capacity, but instead used an Empty-flag per array to see which arrays where inactive and as such shouldn’t be searched.

Anyway, as I said – this was years ago. Since then we managed to dispence of the pool altogether, as we off-loaded most of the strings – which is quite the memory-saver

something like this:

procedure InternString(var _s: string);
var
Idx: integer;
begin
if InternedStrings.Find(_s, Idx) then
_s := InternedStrings[Idx]
else
InternedStrings.Insert(Idx, _s);
end;

Where InternedStrings is the global TStringList in question. Not quite rocket science, is it?

A hashing list would probably perform even better.

The problem with this approach is, that it will never free the strings. A solution could be to periodically scan the list and remove every string that has a reference count of 1. Of course, in your case, it is simpler: You can just clear the list when the user closes the log file. Also, when the list gets larger, the performance for inserting a string gets worse.

Btw: I am glad to hear you are working on reducing the memory requirement of the console. That gives me hope that I will some day be able to load my log files without splittig them. They usually are huge. The only real solution for this problem of course is to not load the whole file into memory at all, but I guess you already knew that. I ran into memory constrictions myself just a few weeks ago, it felt like a return to the bad days of DOS programming…

Of course I had never heard about “string interning” back then.

(I must be getting old, because I start sounding like all these old guys when I started into computing: “Been there, done that, nothing new under the sun.”)