64-bit console app with lots of memory on Windows

[SymbolicFrank]

This is probably a stupid question, but I have a console application that loads a very large csv file, and as soon as the memory usage crosses the 2^32 border, everything slows down to a crawl. The CPU usage is still 25% (1 core), but the memory usage stops growing, and nothing else seems to happen.

I made multiple servers on Linux that were console applications and use 4+ GB of memory, so I know it should work.

So, how do I know if the application is actually 64-bit? I cannot find it in the settings. Or is the problem that Windows restricts the memory usage to 4GB? Do I get an out-of-memory somewhere? Or are classes restricted to a size of 2^32? I know Delphi classes could only grow to 2^31 size.

I use a TFPGObjectList to store 18 million records. Is that the problem?

[Pascal]

Just have look in the Windows taskmanger when it runs. It shows you if it is 32bit if you are on a 64bit system.
Which Windows version are you using?

[SymbolicFrank]

Windows 7 64-bit with 8 GB RAM. It doesn't say "32-bit" anywhere in taskman or properties.

And as far as I can find in the code, it should work, although there is no explicit definition of "Pointer", which I assume is 64-bit.

[Thaddy]

So, how do I know if the application is actually 64-bit? I cannot find it in the settings. Or is the problem that Windows restricts the memory usage to 4GB? Do I get an out-of-memory somewhere? Or are classes restricted to a size of 2^32? I know Delphi classes could only grow to 2^31 size.

No it does not in modern Delphi, nor does it in FPC. If you must use 32 bit use SetPeFlags. https://www.freepascal.org/docs-html/3.0.0/prog/progsu117.html#x125-1260001.3.34 {$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE}

I use a TFPGObjectList to store 18 million records. Is that the problem?

problems with 18 million records in memory depends on the SIZE of the record. ...

[Thaddy]

And as far as I can find in the code, it should work, although there is no explicit definition of "Pointer", which I assume is 64-bit.

Use PtrUINT instead. Anyway, the compiler knows about the size of the pointer for a platform, just do not assume it is an int32. Basics

[Pascal]

Windows 7 64-bit with 8 GB RAM. It doesn't say "32-bit" anywhere in taskman or properties.

See "*32" mark:

[SymbolicFrank]

Ok, so it's 64-bit.

Well, I really would want to know what happens, but that takes too much time. I'll reduce the size of the classes and see if it stays below 2^32 in size.

[SymbolicFrank]

Never mind. Human error. The 2^32 boundary was just a coincidence.

I load a list with all the global IP ranges into memory, and the last third are ipv6 addresses instead of ipv4 ones. And they are rejected, because unexpected. Which takes much longer than accepting them, for some reason.

The debugger crashes at some point (cannot read register, cannot suspend thread), WriteLn is very slow and most editors complain the csv file is too big, so it took a while to find out.

Sorry to have bothered you all with it.


Btw, just to make sure, I'm going to increase the size of all data types I use to 64-bit by default.