[Solved] Heap overflow error with TFileStream

[bobonwhidbey]

This approach works great for smaller files, but it's crashing with an heap overflow error at almost the 40,000th 100 byte record. Here's some streamlined code.

Code: Pascal  [Select][+][-]

1. var
2.   fstream: TFileStream;
3.   s : string
4. begin
5.   fstream := TFileStream.Create('MyFileName', fmCreate);
6.   s := GetTitleLine+CRLF; // CRLF=#13#10
7.   fstream.Write(s[1], length(s));
8.   while not Query.EOF do begin
9.      s := GetTextofRecord(Query)+CRLF;
10.      fstream.Write(s[1], length(s));
11.      Query.Next;
12.   end;
13.  
14.   fstream.Free;
15.   end;
16.  

Is there an approach I can use to create a large text file and avoid the heap error?  I'm not wedded to tFileStream. TStringlist also crashes.

[gucao]

I think the query is causing your problem,You can try to let the query extract 1000 pieces of data each time for testing.

Good luck.

[balazsszekely]

Query is a TSQLQuery? If yes, then set Unidirectional property to true just before opening the query and you are good to go.

[Warfley]

On 32 bit systems you only have 4 gigs of address space available. So the first thing you should do is compile your application as 64 bit.

After that the only limit you might reach is the physically available memory in your system, if that is a problem look at memory mapped files (mmap), which will allow you to map file data into a memory range without it being located in the RAM

[rvk]

This approach works great for smaller files, but it's crashing with an heap overflow error at almost the 40,000th 100 byte record. Here's some streamlined code.
<snip>
Is there an approach I can use to create a large text file and avoid the heap error?  I'm not wedded to tFileStream. TStringlist also crashes.

I'm sure there must be something else wrong with your program.
TFileStream can handle very very large files.

Just tried this and it created a file of 498.047KB file !!
Definitely bigger than your 40.000 x 100 bytes files (which is about 3.906KB)

Code: Pascal  [Select][+][-]

1. program Project1;
2. uses Classes;
3.  
4. const
5.   GetTitleLine = 'Title Line';
6.   CRLF = #13#10;
7.  
8. function GetTextofRecord(): String;
9. begin
10.   Result := 'This is just a testline';
11.   while Length(Result) < 100 do Result := Result + '.';
12. end;
13.  
14. var
15.   fstream: TFileStream;
16.   s : string;
17.   Iterations: integer;
18. begin
19.   fstream := TFileStream.Create('MyFileName', fmCreate);
20.   s := GetTitleLine+CRLF; // CRLF=#13#10
21.   fstream.Write(s[1], length(s));
22.   Iterations := 0;
23.   while Iterations < 5000000 do begin
24.      s := GetTextofRecord()+CRLF;
25.      fstream.Write(s[1], length(s));
26.      Inc(Iterations);
27.   end;
28.  
29.   fstream.Free;
30.  
31. end.

Show us your GetTextofRecord() function and we might be able to pinpoint the problem.

[marcov]

rvk: the line of text is only stored in memory one at a time.

Getmem & Gucao's remark that the query is somehow gigantic (and a query is much larger due to the variant use than the resulting bytes), and of course Warfley's remark to avoid 32-bti in the first place if you are a heavy memory user.

[Zvoni]

Agree with everyone: probably his GetTextOfRecord.
Considering TSQLQuery inherits from TCustomBufDataset, and TCustomBufDataset has a SaveToFile-Method, which internally creates a TFileStream........

Show us GetTextOfRecord

[rvk]

rvk: the line of text is only stored in memory one at a time.

TFileStream writes out every line to disk. So there is nothing stored in memory.

So the heap problem isn't related to the TFileStream at all (unlike what TS expected).
The problem is probably in GetTextOfRecord.

Getmem & Gucao's remark that the query is somehow gigantic (and a query is much larger due to the variant use than the resulting bytes), and of course Warfley's remark to avoid 32-bti in the first place if you are a heavy memory user.

The TQuery wouldn't make a difference. It could handle milion and milion of records. And TFileStream can handle that too.
It's the GetTextOfRecord whic is the problem.

You can see from my example that TFileStream handles this just fine.
It creates a file of 0,5GB. Way more than the 4MB file TS creates when the heap crash happens.

[marcov]

rvk: If the query doesn't prefetch everything.

But yeah, it also could be that something in the procedure is odd.

[Zvoni]

rvk: If the query doesn't prefetch everything.

But yeah, it also could be that something in the procedure is odd.

Especially, since the OP passes the whole Object to the Procedure.....
Obviously, not knowing his Project, i'm more surprised, why the OP doesn't make the Function a Property of his Query-Object, passing, say the current RecordIndex.....

[bobonwhidbey]

Yes - IDList is a TSQLQuery. My problem had nothing to do with saving to a TFileStream. Setting IDList.Unidirectional := true solved the problem.

Thank you GetMem and others for weighing in on this.

[rvk]

Color me amazed 

TSQLQuery is a descendant of BufDataset, a dataset that buffers the data it receives into memory.

https://wiki.freepascal.org/Working_With_TSQLQuery#Error_messages

Delphi definitely doesn't have the same problem.

Wonder why this wasn't done the other way around.

[marcov]

Delphi definitely doesn't have the same problem.

Using ZEOS on both solves that.

[rvk]

I've never had such problems on Delphi.

And from what I can remember, the transaction handling with zeos isn't all that.
So I kept with IBX on Delphi.

[Thaddy]

Using Zeos is a mistake. It should have been removed ions ago.
BTW a heap overflow error? I haven't seen that in 50 years of programming
Show CODE..

Because an EOutOfMemory should appear first.

Some rot has to be removed.