File IO issues

[dryzone]

I tried my best to figure out how to read data with the Pascal "copy" command.
I much prefer to use bash for fileio through a shell command, but before doing that I want to see if I cannot get copy to work in a reliable form, which I failed to do so far.

Attached find a datafile.
The data will always be separated by a space between the data never mind how large the numbers grow.

How do I read a specified entry in the attached file by supplying a row number and column number using the "copy" command?

Whatever I tried, the file read is unreliable.

[cdbc]

Hi
I'm not entirely sure I'd use 'Copy', rather I think I'd make a function / procedure with a small simple 'Finite StateMachine' (parser) to pick the /fields/ in every line...
Just my 2 cent's worth.
Regards Benny

[Roland57]

How I would do (because I love regular expressions):

Code: Pascal  [Select][+][-]

1. {$MODE objfpc}{$H+}
2.  
3. uses
4.   SysUtils, RegExpr, StrUtils;
5.  
6. function GetData(const ARow, ACol: integer): string;
7. const
8.   CColCount = 5;
9. var
10.   LFile: textfile;
11.   LRow: integer;
12.   LLine: string;
13.   LExpr: TRegExpr;
14. begin
15.   Assign(LFile, 'example.txt');
16.   Reset(LFile);
17.   LRow := -1;
18.   repeat
19.     ReadLn(LFile, LLine);
20.     Inc(LRow);
21.   until LRow = ARow;
22.   Close(LFile);
23.  
24.   LExpr := TRegExpr.Create(DupeString('\s+(-?\d+\.\d+)', CColCount));
25.   if LExpr.Exec(LLine) then
26.     result := LExpr.Match[Succ(ACol)]
27.   else
28.     result := EmptyStr;
29.   LExpr.Free;
30. end;
31.  
32. begin
33.   WriteLn(GetData(2, 0) = '0.380');
34.   WriteLn(GetData(2, 1) = '0.390');
35.   WriteLn(GetData(3, 0) = '-3.000');
36. end.

Code: Text  [Select][+][-]

1. $ ./example
2. TRUE
3. TRUE
4. TRUE

[TRon]

No idea what bash has to do with fileio and copy...

Code: Pascal  [Select][+][-]

1. {$mode objfpc}{$h+}
2.  
3. uses
4.   classes, sysutils;
5.  
6. function Get(Rows: TStrings; RowIdx, ColIdx: integer): string;
7. var
8.   Row     : string;
9.   Columns : array of string;
10. begin
11.   result := 'error';
12.   if RowIdx < Rows.Count then
13.   begin
14.     Row := Rows[RowIdx];
15.     Columns := Row.Split([' '], TStringSplitOptions.ExcludeEmpty);
16.     if ColIdx < Length(Columns)
17.       then result := Columns[ColIdx];
18.   end;
19. end;
20.  
21. var
22.   Rows : TStringList;
23.  
24. begin
25.   Rows := TStringList.Create;
26.   Rows.LoadFromFile('example.txt');
27.   Writeln(Get(Rows, 2, 4));
28.   Rows.Free;
29. end.
30.  

[dryzone]

Hi
I'm not entirely sure I'd use 'Copy', rather I think I'd make a function / procedure with a small simple 'Finite StateMachine' (parser) to pick the /fields/ in every line...
Just my 2 cent's worth.
Regards Benny

I obviously used it in a loop procedure, else how would I be able to read values for given row or column.
Copy is the engine of the procedure, therefore I only quoted that.

[dryzone]

No idea what bash has to do with fileio and copy...

Have I ever said that it has anything to do with "copy"? You place words in my mouth.
Maybe read my post before making such claims ?

I said that I might have to use bash through shell commands as it is way-way more powerful than pascal fileio, but definitely slower which is the only drawback..
As speed is not a problem with this program's file io, bash would be a better choice as you dont have to program any Functions just to read something from a file with endless debug as is the case with Pascal.

Pascal/Delphi has been my main programming language my entire life, but the fileio in Pascal is generally horrendous. Same with Fortran.
If there is one thing the dreaded C and it's derivatives did right, is the great fileio they have.

I however just want to see if I cannot keep it in Pascal. 

[dryzone]

@ Roland57

Thank you for your example. I will try it out and see if I can make it work for my purposes.
Appreciated.

[dryzone]

How I would do (because I love regular expressions):

I installed the full latest fpc, but somehow the RegExpr unit is missing when trying your code.

Looking online, there seems to be several versions by different authors floating around, so I need to know which regular expression unit you used  as I dont want to risk chasing false bugs.

Is there an fpc units package I maybe miss where RegExpr unit is included ?

Looking in my system directory where 3.2.2 is installed I get,

find |grep -i reg |grep -i ppu
./3.2.2/units/i386-linux/fcl-fpcunit/testregistry.ppu
./3.2.2/units/i386-linux/regexpr/uregexpr.ppu
./3.2.2/units/i386-linux/regexpr/oldregexpr.ppu
./3.2.2/units/i386-linux/regexpr/regexpr.ppu
./3.2.2/units/i386-linux/regexpr/regex.ppu
./3.2.2/units/i386-linux/fcl-registry/xmlreg.ppu
./3.2.2/units/i386-linux/fcl-registry/registry.ppu
./3.2.2/units/i386-linux/fcl-db/fpddregstd.ppu
./3.2.2/units/i386-linux/googleapi/googleserviceregistry.ppu
./3.2.2/units/x86_64-linux/fcl-fpcunit/testregistry.ppu
./3.2.2/units/x86_64-linux/regexpr/uregexpr.ppu
./3.2.2/units/x86_64-linux/regexpr/oldregexpr.ppu
./3.2.2/units/x86_64-linux/regexpr/regexpr.ppu
./3.2.2/units/x86_64-linux/regexpr/regex.ppu
./3.2.2/units/x86_64-linux/fcl-registry/xmlreg.ppu
./3.2.2/units/x86_64-linux/fcl-registry/registry.ppu
./3.2.2/units/x86_64-linux/fcl-db/fpddregstd.ppu
./3.2.2/units/x86_64-linux/googleapi/googleserviceregistry.ppu
./2.0.4/units/i386-linux/regexpr/regexpr.ppu
./2.0.4/units/i386-linux/fcl/xmlreg.ppu
./2.0.4/units/i386-linux/fcl/testregistry.ppu
./2.0.4/units/i386-linux/fcl/registry.ppu

So there are regular expression units as installed  but somehow what your function asks for is not covered by these units.

[dryzone]

@Roland57
Do you maybe use Lazarus specific units, as what you use is not in the fpc tree.
I used 
wget https://sourceforge.net/projects/freepascal/files/Source/3.2.2/fpc-3.2.2.source.tar.gz
wget https://sourceforge.net/projects/freepascal/files/Linux/3.2.2/fpc-3.2.2.x86_64-linux.tar

to install

Here follows the missing units errors. I tried both ppc386 and ppcx64 compilers both 3.2.2 versions.

user@lt2:
$ ppc386_3.2.2 test_fileio.pas
Free Pascal Compiler version 3.2.2 [2021/05/16] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Linux for i386
Compiling test_fileio.pas
Compiling math.pas
Compiling regexpr.pas
PPU Loading /usr/lib/fpc/3.2.2/units/i386-linux/rtl/types.ppu
PPU Source: types.pp not available
PPU Source: typshrdh.inc not available
PPU Source: typshrd.inc not available
Recompiling Types, checksum changed for math.ppu {impl}
types.pp(138,10) Fatal: Can't find unit Types used by Classes
Fatal: Compilation aborted

user@t2:
$ ppcx64 test_fileio.pas
Free Pascal Compiler version 3.2.2 [2021/05/16] for x86_64
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Linux for x86-64
Compiling test_fileio.pas
Compiling regexpr.pas
Compiling math.pas
PPU Loading /usr/lib/fpc/3.2.2/units/x86_64-linux/rtl/types.ppu
PPU Source: types.pp not available
PPU Source: typshrdh.inc not available
PPU Source: typshrd.inc not available
Recompiling Types, checksum changed for math.ppu {impl}
types.pp(138,10) Fatal: Can't find unit Types used by Classes
Fatal: Compilation aborted

I would really like to use your example as it seems to be exactly what I am looking for.
Could you point me to what the heck is  up with the missing units, as I installed according to the instructions. I reinstalled everything again and and the problem remains.

[dryzone]

I managed to resolve most of the issues by manually copying the missing units and includes from the installation directory straight to where the program resides. Now I miss only one include "typshrdh.inc". See below.
Cannot find the file  "typshrdh.inc"  online.
Anyone has it ?
Once I have that, the program will compile and run.

ppc386_3.2.2 test_fileio.pas
Free Pascal Compiler version 3.2.2 [2021/05/16] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Linux for i386
Compiling test_fileio.pas
Compiling regexpr.pas
Compiling types.pp
types.pp(101,4) Fatal: Cannot open include file "typshrdh.inc"
Fatal: Compilation aborted

[dryzone]

I opened a separate thread regarding "typshrd.inc"
So we dont have to worry about it here.
Compiler refuses to read it, even if it is placed in the same directory as types unit or in the same directory as the program is.
Compiler cannot find it.

[Roland57]

Thank you for your example. I will try it out and see if I can make it work for my purposes.

You are welcome. But I think the example by TRon is better. 

To answer your question, my example uses the standard RegExpr unit shipped with FPC: you don't have to install anything.

[TRon]

You are welcome. But I think the example by TRon is better. 

Although my example is perhaps easier to read it has the drawback that it reads the whole file in memory before processing. In that regards your example is far better for larger files and/or lower row numbers.

I would probably make a mixture of both yours and mine but when doing so probably also opt for something like streamreader for reading the lines.

Fastest solution would be filestream that pinpoints the starting and ending location by counting the rows and columns and stop reading when found, then copy out the match using the located start and end-position.

[dryzone]

Thank you for your example. I will try it out and see if I can make it work for my purposes.

You are welcome. But I think the example by TRon is better. 

To answer your question, my example uses the standard RegExpr unit shipped with FPC: you don't have to install anything.

Yes sure, but fpc compiler refuses to load regexp completely, look at the output I posted. It refuses to find the include file "typshrdh.inc" even if I move it to the same directory as where your program resides. A serious problem in the ppc distribution. The only way to resolve this is to take "typshrdh.inc" and copy it into the type unit and create a new type unit, basically fixing the ppc distribution error by removing the include call and pasting the include into the type unit.
There is a huge problem with the ppc distribution on Debian/MX. I now reinstalled 5 times and the result remains the same.

[dryzone]

Hi
I'm not entirely sure I'd use 'Copy', rather I think I'd make a function / procedure
Regards Benny

You sure are right about that.
I managed to use it successfully for years on large datasets, but now it starts biting me hard.
Dont know what triggered it now to act up, and I sure need to move to something less lowbrow.

I am just not good with pascal file-io. It is the only thing in fpk that always seem to stump me.