I have a 4 billion limit being reached in my variables somewhere, but where?

[Gizmo]

Hi

I'm exceeding a variable capacity somewhere in my code and I can't work out where. It must be to do with the 4 billion max capacity of an integer, but none of my variables in question are standard 32-bit integers.

Bascially, my program recurses a directory full of files and does stuff with each one. One of those things that it does is add it's size to the total amount of bytes already read from other files. So obviously if it's a directory full of many files that are many Gb's in size, the "Total Bytes Read" variable needs to be big.

The function I am using is from http://stackoverflow.com/questions/1285979/delphi-function-to-display-number-of-bytes-as-windows-does and is reproduced below :

Code: Pascal  [Select][+][-]

1.  
2.  
3. // This function converts the integer value of file size into human readable form
4. // Taken from:
5. // http://stackoverflow.com/questions/1285979/delphi-function-to-display-number-of-bytes-as-windows-does
6.  
7. function TForm1.FormatByteSize(const bytes: Longword): string;
8. var
9.   B: byte;
10.   KB: word;
11.   MB: QWord;
12.   GB: QWord;
13.   TB: UInt64;
14. begin
15.  
16.   B  := 1; //byte
17.   KB := 1024 * B; //kilobyte
18.   MB := 1024 * KB; //megabyte
19.   GB := 1024 * MB; //gigabyte
20.   TB := 1024 * GB; //terabyte
21.  
22.   if bytes > TB then
23.     result := FormatFloat('#.## TiB', bytes / TB)
24.   else
25.     if bytes > GB then
26.       result := FormatFloat('#.## GiB', bytes / GB)
27.     else
28.       if bytes > MB then
29.         result := FormatFloat('#.## MiB', bytes / MB)
30.       else
31.         if bytes > KB then
32.           result := FormatFloat('#.## KiB', bytes / KB)
33.         else
34.           result := FormatFloat('#.## bytes', bytes) ;
35. end;
36.  
37.  

Then there is a global variable called 'TotalBytesRead' that is accessed by two procedures :

Code: [Select]

=pascal]
public
   { public declarations }
   FileCounter, NoOfFilesInDir2: integer;
   TotalBytesRead : UInt64;
   StopScan : Boolean;
                               

The part of my code that calls FormatByteSize is here in a button click procedure. It is called for each file that is found so everything is initialised apart from TotalBytesRead, which is initialised just at the start of the program being launched :

Code: Pascal  [Select][+][-]

1. var
2.   SizeOfFile : int64;     (needed for each individual file)
3. ...
4. begin
5. FI := TFileIterator.Create;
6. SG := TStringGrid.Create(self);
7. SizeOfFile := 0;                            // (initialised to zero for each new file)
8. ...
9.  
10. ...                            
11.   SizeOfFile := FileSize(File);  
12.   ...
13.   TotalBytesRead := TotalBytesRead + SizeOfFile;
14.   edtTotalBytesExamined.Caption := FormatByteSize(TotalBytesRead);  
15.  

My problem is that the value is edtTotalBytesExamined.Caption, when it exceeds 4Gb, it resets to zero, and I cannot work out why.

Anyhelp appreciated as always


                                                     

[typo]

Try to replace LongWord here by Int64:

Code: [Select]

FormatByteSize(const bytes: LongWord): string;

[Marc]

or by qword

[Bart]

Consider counting the total amount in a floating point variable.
(Unless precision to the exact byte is needed even if the total amount is >>> Terabytes)

Bart

[Gizmo]

Of course! Thank you both of you - I will try that later and report back if it works. I am sure it will, having read  here http://www.freepascal.org/docs-html/ref/refsu5.html and the limitations of LongWord being 0 .. 4294967295  (which when representing bytes is 4 billion bytes, 4Gb).

I quess that seeing as the filesizes will always be positive, and in some cases the total data will be many terabytes, stored as long integer values, it makes most sense to use QWord, as it's 0..18446744073709551615 , whereas Int64 uses a lot of memory to store negative values (-9223372036854775808 .. 9223372036854775807) and will not allow quite such a high number as QWord. ??

Bart - why a floating point variable please? Why would you recommend that over an integer for this purpose? Forgive me - I still learning.

[Blaazen]

whereas Int64 uses a lot of memory to store negative values (-9223372036854775808 .. 9223372036854775807) and will not allow quite such a high number as QWord

Not a lot of memory. Just one bit. 

BTW when I look to your declaration:

Code: [Select]

function TForm1.FormatByteSize(const bytes: Longword): string;
var
  B: byte;
  KB: word;
  MB: QWord;
  GB: QWord;
  TB: UInt64;
................
QWord and UInt64 are the same. UInt64 is Unsigned Int64.

[Bart]

Bart - why a floating point variable please? Why would you recommend that over an integer for this purpose? Forgive me - I still learning.

Probably not necessary, as you pointed out High(QWord) = 18446744073709551615  (that is appr. 18 ExaByte).
Currently HardDisks are some Teraytes.
1024 Terabyte = 1 Petabyte
1024 Petabyte = 1 Exabyte

So wiht QWord you have appr. enough for 250 thousand 4 TeraByte harddisks full of data.

A Double can go to 1.7E308, so whenever we are past the Exo barrier, you need either a bigger integer type, a floating point, or make your own big-integer type.

Bart

[Gizmo]

Thanks for the explanation Bart.

All - yep - that was it. I've changed LongWord to QWord and now it works fine! Thanks guys - you're jolly good eggs.

[Marc]

So with QWord you have appr. enough for 250 thousand 4 TeraByte harddisks full of data.

A Double can go to 1.7E308, so whenever we are past the Exo barrier, you need either a bigger integer type, a floating point, or make your own big-integer type.

Bart

By that time we will have a int128

[marcov]

Consider counting the total amount in a floating point variable.
(Unless precision to the exact byte is needed even if the total amount is >>> Terabytes)
 

Somebody always needs the precise one. It does not make sense to duplicate such routines into an precise and unprecise one.