Recent

Author Topic: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile  (Read 2629 times)

NOPIK

  • New Member
  • *
  • Posts: 13
Very long loading time aside - I got an Heap overflow error on application Close(!) in GetControlText(???) in very simple program:
Code: [Select]
procedure TForm1.Button1Click(Sender: TObject);
begin
  if (OpenDialog1.Execute) then
  begin
    Memo1.Visible := false;
    Memo1.Lines.BeginUpdate;
    Memo1.Lines.LoadFromFile(OpenDialog1.FileName);
    Memo1.Lines.EndUpdate;
    Memo1.Visible := true;
  end;
end;
I'm loading test XML file only 97Mb in length, 450K lines (Small for my data and must be no sweat for Win32 Lazarus)

cdbc

  • Hero Member
  • *****
  • Posts: 1649
    • http://www.cdbc.dk
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #1 on: June 12, 2024, 01:52:05 pm »
Hi
Quote
Small for my data
It may be small for you, but it ain't for the memo-control. search the net, 'cause I know there are some limit to how much you can stuff into a memo-control on win32.
Regards Benny
If it ain't broke, don't fix it ;)
PCLinuxOS(rolling release) 64bit -> KDE5 -> FPC 3.2.2 -> Lazarus 2.2.6 up until Jan 2024 from then on it's: KDE5/QT5 -> FPC 3.3.1 -> Lazarus 3.0

Thaddy

  • Hero Member
  • *****
  • Posts: 16160
  • Censorship about opinions does not belong here.
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #2 on: June 12, 2024, 01:56:17 pm »
On windows, compiled for 32 bit,  the Tmemo uses the windows class EDIT and that has by default severe limitations for the size it can handle.
It is limited to 2 GB data and/or 32767 lines and/or available memory, whatever comes first.
There are work-arounds: I think richmemo has the capacity to load larger files.
The windows API class RICHEDIT2 or higher can be created with BOTTOMLESS in CreateParams, which gives you essentially windowed access to very large files using the windows file mapping API. I don't know if this is implemented for e.g. TRichView on Lazarus.
In KOL we have the bottomless property for RichEdit surfaced.
So my advice is to check TRichview first
« Last Edit: June 12, 2024, 02:17:27 pm by Thaddy »
If I smell bad code it usually is bad code and that includes my own code.

Zvoni

  • Hero Member
  • *****
  • Posts: 2738
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #3 on: June 12, 2024, 02:00:32 pm »
I'm loading test XML file only 97Mb in length, 450K lines (Small for my data and must be no sweat for Win32 Lazarus)
Lines of TMemo is a TStrings: https://lazarus-ccr.sourceforge.io/docs/lcl/stdctrls/tcustommemo.lines.html
TStrings has "Count", which is "integer": https://lazarus-ccr.sourceforge.io/docs/rtl/classes/tstrings.count.html
And Range of "Integer" depends on Mode: https://lazarus-ccr.sourceforge.io/docs/rtl/system/integer.html
Quote
The system unit defines Integer as a signed 16-bit integer. But when DELPHI or OBJFPC mode are active, then the objpas unit redefines Integer as a 32-bit integer.

16-Bit signed: --> 32767 max

What mode is in that module?
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

Thaddy

  • Hero Member
  • *****
  • Posts: 16160
  • Censorship about opinions does not belong here.
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #4 on: June 12, 2024, 02:20:41 pm »
Zvoni, that is not relevant here, the specs are from msdn.
The RICHEDIT2 (or msft)  control has different specs and can handle huge files,
but I don't know if they are surfaced in Lazarus.
If I smell bad code it usually is bad code and that includes my own code.

Zvoni

  • Hero Member
  • *****
  • Posts: 2738
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #5 on: June 12, 2024, 03:22:43 pm »
Zvoni, that is not relevant here, the specs are from msdn.
The RICHEDIT2 (or msft)  control has different specs and can handle huge files,
but I don't know if they are surfaced in Lazarus.
Yeah, just saw it (timestamp)
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

NOPIK

  • New Member
  • *
  • Posts: 13
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #6 on: June 12, 2024, 03:45:25 pm »
According to Microsoft:
"EM_SETLIMITTEXT message to the edit control. If the buffer exceeds either limit, the system sends the application an EN_ERRSPACE notification code. An application can retrieve the current text limit by sending an EM_GETLIMITTEXT."
Note, that no  errors was returned in my case.
max limit for control is  0x7FFFFFFE and all parameters is uint32.
Of course, all limits for the module set to objpas mode. I used TStrings as test and everything work as intended.
I will try 64bit Lazarus and manually calling WinAPI later.

Thaddy

  • Hero Member
  • *****
  • Posts: 16160
  • Censorship about opinions does not belong here.
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #7 on: June 12, 2024, 05:18:49 pm »
That has the effect as I described:
It is limited to 2 GB data and/or 32767 lines and/or available memory, whatever comes first.
It is a recurring misunderstanding that there are no limits to standard winapi controls on the low-level.
Also note that by default Tmemo becomes very, very slow when approaching the limits.
Usually you won't want that.
It has also nothing to do with TStrings/TStringlist: TStrings is mapped to the content.
Same as e.g. TListbox, Tcombobox and the likes that rely on the windows api controls.
« Last Edit: June 12, 2024, 05:33:22 pm by Thaddy »
If I smell bad code it usually is bad code and that includes my own code.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8015
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #8 on: June 12, 2024, 05:57:45 pm »
Also note that by default Tmemo becomes very, very slow when approaching the limits.

Hence what I described a week or so ago, where I was taking pains to prevent the amount of logged data displayed in a TMemo/TRichMemo from exceeding a few thousand lines.

I don't know what other participants think, but that really is a very, very large amount of data to put into a graphical control; in fact it reminds me of the problems that the database access facility of many a "productivity suite" had when asked to look at any non-trivial table.

I'd be tempted to look for something which was basically a frontend "window" onto a file, or- even better- a database. However the LCL/VCL's database controls have, TTBOMK, never played particularly nicely with database cursors etc.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

af0815

  • Hero Member
  • *****
  • Posts: 1379
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #9 on: June 12, 2024, 08:20:48 pm »
My first idea was - why is someone putting such an amount of data in a visual control ? I see no effort to do this. No person can work with this. 450.000 Lines in a visual control. It looks for me like da misdesigned user interface. Or did i miss something.

 
regards
Andreas

MarkMLl

  • Hero Member
  • *****
  • Posts: 8015
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #10 on: June 12, 2024, 08:54:11 pm »
My first idea was - why is someone putting such an amount of data in a visual control ? I see no effort to do this. No person can work with this. 450.000 Lines in a visual control. It looks for me like da misdesigned user interface. Or did i miss something.

I prefer not to criticise somebody's intent in this kind of situation. For example, this might be a debugging interface looking at raw XML that something else is having problems with.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 16160
  • Censorship about opinions does not belong here.
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #11 on: June 12, 2024, 08:59:51 pm »
Or did i miss something.
? I add the question mark for you. Yes, you missed the underlying native control specs.
If I smell bad code it usually is bad code and that includes my own code.

jamie

  • Hero Member
  • *****
  • Posts: 6735
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #12 on: June 12, 2024, 11:21:03 pm »
a VirtualMemo needs to be used here.

So only three pages of info are in memory, it can scroll the file .
The only true wisdom is knowing you know nothing

Thaddy

  • Hero Member
  • *****
  • Posts: 16160
  • Censorship about opinions does not belong here.
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #13 on: June 13, 2024, 10:02:51 am »
bottomless turns your control into a virtual memo with a windowed access over a filemapping. so that is basically the same.
If I smell bad code it usually is bad code and that includes my own code.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8015
Re: Heap overflow error in GetControlText after TMemo.Lines.LoadFromFile
« Reply #14 on: June 13, 2024, 11:02:34 am »
a VirtualMemo needs to be used here.

So only three pages of info are in memory, it can scroll the file .

bottomless turns your control into a virtual memo with a windowed access over a filemapping. so that is basically the same.

Pray continue...

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018