Why a Memo1 and not a listbox?
Nothing special. It's just my habit to use TMemo for showing test result. I've tried to modify the code to use TListBox instead, it worked without any issue too.
You said you failed to load 7 millions+ items into a list of string, aren't you?I think I found the culprit of your issue.
Below is the code I used for a test:
procedure TForm1.Button1Click(Sender: TObject);
var
S: string;
i: Integer;
begin
for i := 1 to 5000000 do
begin
S := i.ToString;
Memo1.Append(S);
ListBox1.AddItem(S, nil);
if (i mod 100) <> 0 then Continue;
Application.ProcessMessages;
end;
ShowMessage('Done');
end;
Only on 5 millions items, the Memo1 and ListBox1 already ate about 2 GB of my computer memory and it took more than 5 minutes.
Why it's so stressful compare to the previous 10 millions items massive harddisk operation?
My previous code loads only necessary data into memory. Although it was massive disk operation, the OS and harddisk caching system optimize the processes. It took only 20 seconds to read all the 10 millions records and load some necessary items into memory.
The 5 millions TMemo and TListBox string appending is extremely stressing the hardware.
Because string is expensive 
So I believe that is why you failed. You tried to load all the 7 millions data into a list of string (or TStringList or TListBox). AFAIK, String and TStringList are okay for handling data that are very long but not huge in the count. Because they have to deal with memory allocation, fragmentation, etc.
Correct me if I'm wrong.
You still can load all the data into memory and run smoothly. But you should write your own component, which is more efficient both in memory usage and performance. Maybe you can use BlockRead and TFPList for this purpose. But it will be very hard, I prefer the "load only the needed data" approach.
Recent
