Lazarus

Free Pascal => General => Topic started by: dryzone on March 11, 2025, 09:35:26 pm

Title: File IO issues
Post by: dryzone on March 11, 2025, 09:35:26 pm
I tried my best to figure out how to read data with the Pascal "copy" command.
I much prefer to use bash for fileio through a shell command, but before doing that I want to see if I cannot get copy to work in a reliable form, which I failed to do so far.

Attached find a datafile.
The data will always be separated by a space between the data never mind how large the numbers grow.

How do I read a specified entry in the attached file by supplying a row number and column number using the "copy" command?

Whatever I tried, the file read is unreliable.


Title: Re: File IO issues
Post by: cdbc on March 11, 2025, 09:49:27 pm
Hi
I'm not entirely sure I'd use 'Copy', rather I think I'd make a function / procedure with a small simple 'Finite StateMachine' (parser) to pick the /fields/ in every line...
Just my 2 cent's worth.
Regards Benny
Title: Re: File IO issues
Post by: Roland57 on March 11, 2025, 10:47:54 pm
How I would do (because I love regular expressions):

Code: Pascal  [Select][+][-]
  1. {$MODE objfpc}{$H+}
  2.  
  3. uses
  4.   SysUtils, RegExpr, StrUtils;
  5.  
  6. function GetData(const ARow, ACol: integer): string;
  7. const
  8.   CColCount = 5;
  9. var
  10.   LFile: textfile;
  11.   LRow: integer;
  12.   LLine: string;
  13.   LExpr: TRegExpr;
  14. begin
  15.   Assign(LFile, 'example.txt');
  16.   Reset(LFile);
  17.   LRow := -1;
  18.   repeat
  19.     ReadLn(LFile, LLine);
  20.     Inc(LRow);
  21.   until LRow = ARow;
  22.   Close(LFile);
  23.  
  24.   LExpr := TRegExpr.Create(DupeString('\s+(-?\d+\.\d+)', CColCount));
  25.   if LExpr.Exec(LLine) then
  26.     result := LExpr.Match[Succ(ACol)]
  27.   else
  28.     result := EmptyStr;
  29.   LExpr.Free;
  30. end;
  31.  
  32. begin
  33.   WriteLn(GetData(2, 0) = '0.380');
  34.   WriteLn(GetData(2, 1) = '0.390');
  35.   WriteLn(GetData(3, 0) = '-3.000');
  36. end.

Code: Text  [Select][+][-]
  1. $ ./example
  2. TRUE
  3. TRUE
  4. TRUE
Title: Re: File IO issues
Post by: TRon on March 11, 2025, 11:01:09 pm
No idea what bash has to do with fileio and copy...

Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$h+}
  2.  
  3. uses
  4.   classes, sysutils;
  5.  
  6. function Get(Rows: TStrings; RowIdx, ColIdx: integer): string;
  7. var
  8.   Row     : string;
  9.   Columns : array of string;
  10. begin
  11.   result := 'error';
  12.   if RowIdx < Rows.Count then
  13.   begin
  14.     Row := Rows[RowIdx];
  15.     Columns := Row.Split([' '], TStringSplitOptions.ExcludeEmpty);
  16.     if ColIdx < Length(Columns)
  17.       then result := Columns[ColIdx];
  18.   end;
  19. end;
  20.  
  21. var
  22.   Rows : TStringList;
  23.  
  24. begin
  25.   Rows := TStringList.Create;
  26.   Rows.LoadFromFile('example.txt');
  27.   Writeln(Get(Rows, 2, 4));
  28.   Rows.Free;
  29. end.
  30.  
Title: Re: File IO issues
Post by: dryzone on March 11, 2025, 11:13:19 pm
Quote from: cdbc on March 11, 2025, 09:49:27 pm
Hi
I'm not entirely sure I'd use 'Copy', rather I think I'd make a function / procedure with a small simple 'Finite StateMachine' (parser) to pick the /fields/ in every line...
Just my 2 cent's worth.
Regards Benny

I obviously used it in a loop procedure, else how would I be able to read values for given row or column.
Copy is the engine of the procedure, therefore I only quoted that.
Title: Re: File IO issues
Post by: dryzone on March 11, 2025, 11:20:50 pm
Quote from: TRon on March 11, 2025, 11:01:09 pm
Quote
No idea what bash has to do with fileio and copy...

Have I ever said that it has anything to do with "copy"? You place words in my mouth.
Maybe read my post before making such claims ?

I said that I might have to use bash through shell commands as it is way-way more powerful than pascal fileio, but definitely slower which is the only drawback..
As speed is not a problem with this program's file io, bash would be a better choice as you dont have to program any Functions just to read something from a file with endless debug as is the case with Pascal.

Pascal/Delphi has been my main programming language my entire life, but the fileio in Pascal is generally horrendous. Same with Fortran.
If there is one thing the dreaded C and it's derivatives did right, is the great fileio they have.

I however just want to see if I cannot keep it in Pascal. 
Title: Re: File IO issues
Post by: dryzone on March 11, 2025, 11:22:57 pm
@ Roland57

Thank you for your example. I will try it out and see if I can make it work for my purposes.
Appreciated.

Title: Re: File IO issues
Post by: dryzone on March 11, 2025, 11:47:16 pm
Quote from: Roland57 on March 11, 2025, 10:47:54 pm
How I would do (because I love regular expressions):

I installed the full latest fpc, but somehow the RegExpr unit is missing when trying your code.

Looking online, there seems to be several versions by different authors floating around, so I need to know which regular expression unit you used  as I dont want to risk chasing false bugs.

Is there an fpc units package I maybe miss where RegExpr unit is included ?

Looking in my system directory where 3.2.2 is installed I get,
Quote
find |grep -i reg |grep -i ppu
./3.2.2/units/i386-linux/fcl-fpcunit/testregistry.ppu
./3.2.2/units/i386-linux/regexpr/uregexpr.ppu
./3.2.2/units/i386-linux/regexpr/oldregexpr.ppu
./3.2.2/units/i386-linux/regexpr/regexpr.ppu
./3.2.2/units/i386-linux/regexpr/regex.ppu
./3.2.2/units/i386-linux/fcl-registry/xmlreg.ppu
./3.2.2/units/i386-linux/fcl-registry/registry.ppu
./3.2.2/units/i386-linux/fcl-db/fpddregstd.ppu
./3.2.2/units/i386-linux/googleapi/googleserviceregistry.ppu
./3.2.2/units/x86_64-linux/fcl-fpcunit/testregistry.ppu
./3.2.2/units/x86_64-linux/regexpr/uregexpr.ppu
./3.2.2/units/x86_64-linux/regexpr/oldregexpr.ppu
./3.2.2/units/x86_64-linux/regexpr/regexpr.ppu
./3.2.2/units/x86_64-linux/regexpr/regex.ppu
./3.2.2/units/x86_64-linux/fcl-registry/xmlreg.ppu
./3.2.2/units/x86_64-linux/fcl-registry/registry.ppu
./3.2.2/units/x86_64-linux/fcl-db/fpddregstd.ppu
./3.2.2/units/x86_64-linux/googleapi/googleserviceregistry.ppu
./2.0.4/units/i386-linux/regexpr/regexpr.ppu
./2.0.4/units/i386-linux/fcl/xmlreg.ppu
./2.0.4/units/i386-linux/fcl/testregistry.ppu
./2.0.4/units/i386-linux/fcl/registry.ppu

So there are regular expression units as installed  but somehow what your function asks for is not covered by these units.
Title: Re: File IO issues
Post by: dryzone on March 12, 2025, 02:45:29 am
@Roland57
Do you maybe use Lazarus specific units, as what you use is not in the fpc tree.
I used 
wget https://sourceforge.net/projects/freepascal/files/Source/3.2.2/fpc-3.2.2.source.tar.gz
wget https://sourceforge.net/projects/freepascal/files/Linux/3.2.2/fpc-3.2.2.x86_64-linux.tar

to install

Here follows the missing units errors. I tried both ppc386 and ppcx64 compilers both 3.2.2 versions.
Quote
user@lt2:
$ ppc386_3.2.2 test_fileio.pas
Free Pascal Compiler version 3.2.2 [2021/05/16] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Linux for i386
Compiling test_fileio.pas
Compiling math.pas
Compiling regexpr.pas
PPU Loading /usr/lib/fpc/3.2.2/units/i386-linux/rtl/types.ppu
PPU Source: types.pp not available
PPU Source: typshrdh.inc not available
PPU Source: typshrd.inc not available
Recompiling Types, checksum changed for math.ppu {impl}
types.pp(138,10) Fatal: Can't find unit Types used by Classes
Fatal: Compilation aborted

user@t2:
$ ppcx64 test_fileio.pas
Free Pascal Compiler version 3.2.2 [2021/05/16] for x86_64
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Linux for x86-64
Compiling test_fileio.pas
Compiling regexpr.pas
Compiling math.pas
PPU Loading /usr/lib/fpc/3.2.2/units/x86_64-linux/rtl/types.ppu
PPU Source: types.pp not available
PPU Source: typshrdh.inc not available
PPU Source: typshrd.inc not available
Recompiling Types, checksum changed for math.ppu {impl}
types.pp(138,10) Fatal: Can't find unit Types used by Classes
Fatal: Compilation aborted

I would really like to use your example as it seems to be exactly what I am looking for.
Could you point me to what the heck is  up with the missing units, as I installed according to the instructions. I reinstalled everything again and and the problem remains.
Title: Re: File IO issues
Post by: dryzone on March 12, 2025, 04:32:39 am
I managed to resolve most of the issues by manually copying the missing units and includes from the installation directory straight to where the program resides. Now I miss only one include "typshrdh.inc". See below.
Cannot find the file  "typshrdh.inc"  online.
Anyone has it ?
Once I have that, the program will compile and run.

Quote
ppc386_3.2.2 test_fileio.pas
Free Pascal Compiler version 3.2.2 [2021/05/16] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Linux for i386
Compiling test_fileio.pas
Compiling regexpr.pas
Compiling types.pp
types.pp(101,4) Fatal: Cannot open include file "typshrdh.inc"
Fatal: Compilation aborted
Title: Re: File IO issues
Post by: dryzone on March 12, 2025, 06:41:42 am
I opened a separate thread regarding "typshrd.inc"
So we dont have to worry about it here.
Compiler refuses to read it, even if it is placed in the same directory as types unit or in the same directory as the program is.
Compiler cannot find it.
Title: Re: File IO issues
Post by: Roland57 on March 12, 2025, 07:42:43 am
Quote from: dryzone on March 11, 2025, 11:22:57 pm
Thank you for your example. I will try it out and see if I can make it work for my purposes.

You are welcome. But I think the example by TRon (https://forum.lazarus.freepascal.org/index.php/topic,70496.msg549581.html#msg549581) is better.  :)

To answer your question, my example uses the standard RegExpr unit shipped with FPC: you don't have to install anything.
Title: Re: File IO issues
Post by: TRon on March 12, 2025, 07:59:16 am
Quote from: Roland57 on March 12, 2025, 07:42:43 am
You are welcome. But I think the example by TRon (https://forum.lazarus.freepascal.org/index.php/topic,70496.msg549581.html#msg549581) is better.  :)
Although my example is perhaps easier to read it has the drawback that it reads the whole file in memory before processing. In that regards your example is far better for larger files and/or lower row numbers.

I would probably make a mixture of both yours and mine but when doing so probably also opt for something like streamreader for reading the lines.

Fastest solution would be filestream that pinpoints the starting and ending location by counting the rows and columns and stop reading when found, then copy out the match using the located start and end-position.
Title: Re: File IO issues
Post by: dryzone on March 12, 2025, 06:42:07 pm
Quote from: Roland57 on March 12, 2025, 07:42:43 am
Quote from: dryzone on March 11, 2025, 11:22:57 pm
Thank you for your example. I will try it out and see if I can make it work for my purposes.

You are welcome. But I think the example by TRon (https://forum.lazarus.freepascal.org/index.php/topic,70496.msg549581.html#msg549581) is better.  :)

To answer your question, my example uses the standard RegExpr unit shipped with FPC: you don't have to install anything.

Yes sure, but fpc compiler refuses to load regexp completely, look at the output I posted. It refuses to find the include file "typshrdh.inc" even if I move it to the same directory as where your program resides. A serious problem in the ppc distribution. The only way to resolve this is to take "typshrdh.inc" and copy it into the type unit and create a new type unit, basically fixing the ppc distribution error by removing the include call and pasting the include into the type unit.
There is a huge problem with the ppc distribution on Debian/MX. I now reinstalled 5 times and the result remains the same.
Title: Re: File IO issues
Post by: dryzone on March 15, 2025, 03:39:07 am
Quote from: cdbc on March 11, 2025, 09:49:27 pm
Hi
I'm not entirely sure I'd use 'Copy', rather I think I'd make a function / procedure
Regards Benny

You sure are right about that.
I managed to use it successfully for years on large datasets, but now it starts biting me hard.
Dont know what triggered it now to act up, and I sure need to move to something less lowbrow.

I am just not good with pascal file-io. It is the only thing in fpk that always seem to stump me. 
Title: Re: File IO issues
Post by: dryzone on March 15, 2025, 03:52:10 am
@Roland57, Tron

Thank you very much for the examples and helps me greatly.

However, to integrate it in my existing data  read routine is going to take a major rewrite of a perfectly working routine, ... except for my reliance on "Copy" to select data from a string.

I figured out an easy way to get around my problem but need your help.

My program reads from file then isolates a row of data by certain criteria in a large dataset.
The read part of the program is rugged and works great.
The string read from the dataset is e.g.
"                -6.000                 2.000                  30.000                 -12.000                 44.000                      "

Since I keep using dubious methods such as Copy to select e.g. one of the datapoints in the string, what command your suggestion I should use that would ruggedly and reliably read the data above for given index j=1..5 , where the spacing can be variable, but the data is always separated by at least one space.

I know this is a really trivial question someone like me should not ask, but reliability trumps complex questions in my case and my programming field. I clearly invented dubious ways (using Copy) and want to see what the correct way would be in your opinion to read from this supplied string in a reliable fashion.

Thanks

Title: Re: File IO issues
Post by: silvercoder70 on March 15, 2025, 04:55:37 am
Try this ...

Code: Pascal  [Select][+][-]
  1. var
  2.   s: string;
  3.   elem: TStringArray;
  4.   i: Integer;
  5.   d: double;
  6. begin
  7.   s := '   5.000       3.000       3.234     234.123  ';
  8.   elem := s.Split(' ', TStringSplitOptions.ExcludeEmpty);
  9.   for i := Low(elem) to High(elem) do
  10.   begin
  11.     d := elem[i].ToDouble;
  12.     writeln(i, '  ', d);
  13.   end;  
  14.  
Title: Re: File IO issues
Post by: dryzone on March 15, 2025, 06:26:21 am
That works great thanks
Title: Re: File IO issues
Post by: dryzone on March 15, 2025, 07:13:42 am
@SilverCoder70

Is there an equivalent for a string, meaning "dummy" will be a string in the following expression

Code: Pascal  [Select][+][-]
  1. dummy:= elem[j].ToExtended;

Here I have it as an extended, but I want dummy to be a string.
I tried
Code: Pascal  [Select][+][-]
  1. elem[j].ToString;

as ToString is used in several other languages, but it doesnt work in FPK
I get
Error: Illegal qualifier
Title: Re: File IO issues
Post by: paweld on March 15, 2025, 07:42:14 am
just
Code: Pascal  [Select][+][-]
  1. string_var := elem[i];
Title: Re: File IO issues
Post by: dryzone on March 15, 2025, 03:58:23 pm
Thanks to all who responded.

Problem solved.
Works way more reliable now.

TinyPortal © 2005-2018