Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile

[NOPIK]

Very long loading time aside - I got an Heap overflow error on application Close(!) in GetControlText(???) in very simple program:

Code: [Select]

procedure TForm1.Button1Click(Sender: TObject);
begin
  if (OpenDialog1.Execute) then
  begin
    Memo1.Visible := false;
    Memo1.Lines.BeginUpdate;
    Memo1.Lines.LoadFromFile(OpenDialog1.FileName);
    Memo1.Lines.EndUpdate;
    Memo1.Visible := true;
  end;
end;
I'm loading test XML file only 97Mb in length, 450K lines (Small for my data and must be no sweat for Win32 Lazarus)

[cdbc]

Hi

Small for my data

It may be small for you, but it ain't for the memo-control. search the net, 'cause I know there are some limit to how much you can stuff into a memo-control on win32.
Regards Benny

[Thaddy]

On windows, compiled for 32 bit,  the Tmemo uses the windows class EDIT and that has by default severe limitations for the size it can handle.
It is limited to 2 GB data and/or 32767 lines and/or available memory, whatever comes first.
There are work-arounds: I think richmemo has the capacity to load larger files.
The windows API class RICHEDIT2 or higher can be created with BOTTOMLESS in CreateParams, which gives you essentially windowed access to very large files using the windows file mapping API. I don't know if this is implemented for e.g. TRichView on Lazarus.
In KOL we have the bottomless property for RichEdit surfaced.
So my advice is to check TRichview first

[Zvoni]

I'm loading test XML file only 97Mb in length, 450K lines (Small for my data and must be no sweat for Win32 Lazarus)

Lines of TMemo is a TStrings: https://lazarus-ccr.sourceforge.io/docs/lcl/stdctrls/tcustommemo.lines.html
TStrings has "Count", which is "integer": https://lazarus-ccr.sourceforge.io/docs/rtl/classes/tstrings.count.html
And Range of "Integer" depends on Mode: https://lazarus-ccr.sourceforge.io/docs/rtl/system/integer.html

The system unit defines Integer as a signed 16-bit integer. But when DELPHI or OBJFPC mode are active, then the objpas unit redefines Integer as a 32-bit integer.

16-Bit signed: --> 32767 max

What mode is in that module?

[Thaddy]

Zvoni, that is not relevant here, the specs are from msdn.
The RICHEDIT2 (or msft)  control has different specs and can handle huge files,
but I don't know if they are surfaced in Lazarus.

[Zvoni]

Zvoni, that is not relevant here, the specs are from msdn.
The RICHEDIT2 (or msft)  control has different specs and can handle huge files,
but I don't know if they are surfaced in Lazarus.

Yeah, just saw it (timestamp)

[NOPIK]

According to Microsoft:
"EM_SETLIMITTEXT message to the edit control. If the buffer exceeds either limit, the system sends the application an EN_ERRSPACE notification code. An application can retrieve the current text limit by sending an EM_GETLIMITTEXT."
Note, that no  errors was returned in my case.
max limit for control is  0x7FFFFFFE and all parameters is uint32.
Of course, all limits for the module set to objpas mode. I used TStrings as test and everything work as intended.
I will try 64bit Lazarus and manually calling WinAPI later.

[Thaddy]

That has the effect as I described:
It is limited to 2 GB data and/or 32767 lines and/or available memory, whatever comes first.
It is a recurring misunderstanding that there are no limits to standard winapi controls on the low-level.
Also note that by default Tmemo becomes very, very slow when approaching the limits.
Usually you won't want that.
It has also nothing to do with TStrings/TStringlist: TStrings is mapped to the content.
Same as e.g. TListbox, Tcombobox and the likes that rely on the windows api controls.

[MarkMLl]

Also note that by default Tmemo becomes very, very slow when approaching the limits.

Hence what I described a week or so ago, where I was taking pains to prevent the amount of logged data displayed in a TMemo/TRichMemo from exceeding a few thousand lines.

I don't know what other participants think, but that really is a very, very large amount of data to put into a graphical control; in fact it reminds me of the problems that the database access facility of many a "productivity suite" had when asked to look at any non-trivial table.

I'd be tempted to look for something which was basically a frontend "window" onto a file, or- even better- a database. However the LCL/VCL's database controls have, TTBOMK, never played particularly nicely with database cursors etc.

MarkMLl

[af0815]

My first idea was - why is someone putting such an amount of data in a visual control ? I see no effort to do this. No person can work with this. 450.000 Lines in a visual control. It looks for me like da misdesigned user interface. Or did i miss something.

 

[MarkMLl]

My first idea was - why is someone putting such an amount of data in a visual control ? I see no effort to do this. No person can work with this. 450.000 Lines in a visual control. It looks for me like da misdesigned user interface. Or did i miss something.

I prefer not to criticise somebody's intent in this kind of situation. For example, this might be a debugging interface looking at raw XML that something else is having problems with.

MarkMLl

[Thaddy]

Or did i miss something.

? I add the question mark for you. Yes, you missed the underlying native control specs.

[jamie]

a VirtualMemo needs to be used here.

So only three pages of info are in memory, it can scroll the file .

[Thaddy]

bottomless turns your control into a virtual memo with a windowed access over a filemapping. so that is basically the same.

[MarkMLl]

a VirtualMemo needs to be used here.

So only three pages of info are in memory, it can scroll the file .

bottomless turns your control into a virtual memo with a windowed access over a filemapping. so that is basically the same.

Pray continue...

MarkMLl