[Solved] How to save and load large text files?

[ludob]

The example you found is reading binary data from a file directly into an array. You have a file with comma separated string values. Remember also that using filestream to read huge files is consuming x times your file size in memory. Eny wrote that when reading a TStringList directly he notice a x10 memory peak.
The folowing code will read lines in the format 1,2,3,4,5,6,7,8,9 and put them in an array in one pass:

Code: [Select]

var
  myFile: TextFile;
  s,s2:string;
  i,j,ipos:integer;
  history: array [0..9, 0..8] of Integer;
begin
  if opendialog1.execute then
    begin
    AssignFile(myFile , OpenDialog1.filename);  // Try to open .txt file
    Reset(myFile);  //Reopen file for reading
    i:=0;
    While not EOF(myFile) do
      begin
      readln(myFile,s);
      j:=0;
      while s<>'' do
        begin
        ipos:=pos(',',s);
        if ipos>0 then
          s2:=copy(s,1,ipos-1)
        else
          s2:=s;
        delete(s,1,length(s2)+1);
        if j<=8 then
          history[i,j]:=StrToInt(s2)
        else
          writeln(format('Too many characters in line %d',[i]));
        j:=j+1;
        end;
      i:=i+1;
      if i>9 then
        begin
        writeln('Too many lines in file');
        break;
        end;
      end;
    CloseFile(myFile);
    end;
end;
Adapt the array as you need.

[Ramijami]

Thanks ludob. Is there a limit to the size of file that can be loaded? I have various sized files of 9 numbers per line that I use to test the code, and I get an external SIGSEGV error if the array is greater than 450000. E.g. if I set

Code: [Select]

history: array [0..450000, 0..8] of Integer;

I don't get the error, but   

Code: [Select]

history: array [0..460000, 0..8] of Integer;

will give the SIGSEGV error. Could it be that the PC I use doesn't have the necessary resources/memory available or is the file too large?

[User137]

You need to make it dynamic array

Code: [Select]

history: array of array of integer;

  setlength(history, 1000000, 9); // (0..8 is 9 values per row)

  history[142112, 4]:=4; // test...

[ludob]

Thanks ludob. Is there a limit to the size of file that can be loaded? I have various sized files of 9 numbers per line that I use to test the code, and I get an external SIGSEGV error if the array is greater than 450000. E.g. if I set

Code: [Select]

history: array [0..450000, 0..8] of Integer;

I don't get the error, but   

Code: [Select]

history: array [0..460000, 0..8] of Integer;

will give the SIGSEGV error. Could it be that the PC I use doesn't have the necessary resources/memory available or is the file too large?

Where do you get the SIGSEGV?
Is history a global var or a local one? Local ones are on the stack which could be limited. 450000 is only 16Mb which is not much but quite a lot for a stack. Use a global variable instead. Else use a dynamic array as User137 suggests because that is also stored on the heap. But don't make the mistake to resize the array while reading the lines because that would cause an enormous amount of data moving around for every resize. Initialize it in the beginning.

[Ramijami]

Thanks user137 and ludob.......used your suggestions and working fine now. Thanks again to all for your input........much appreciated 

[Peter_Vadasz]

will give the SIGSEGV error. Could it be that the PC I use doesn't have the necessary resources/memory available or is the file too large?

The problem is, you load a big file into the memory. If the size of the red and stored data (size of the array) is larger than the available memory you can give an error message. I know the disk operation is slower than the memory operation, but the question is, do you really want to load a huge text file into the memory?
Some reason why I suggest you the binary file in this case:
- smaller file size (there aren't commas, line ending and other charachters in the file)
- you don't need to convert data (string to integer)
- you don't need to load a huge file into the memory
- data access is easier (in my opinion; in this case)
- etc.

Peter