Data Files Writing then Reading

[wp]

... because the guy does not listen ...

How do you know that? There's only one answer by him, immediately after your first post:
- Your /1: He does explain why he uses defines within the record definition
- Your /2: He does not react on your idea with fpread. But his problem is in writing, so why should he?
- Your /3: He does not react on the idea with the header because it is already there. GoodTyp and GoodLen are just that - you did not read carefully enough (like me...)

[Bazzao]

I am listening. I am swamped with code. Thanks to all.

Currently the packed record and array for the string solved the issues, and I now have 2 of the 3 record types working perfectly, so the third will be no problem.

I have just seen code submitted by taazz, so I will look into that. It will provide me with better and cleaner solutions.

And please do not be aggressive. Remember others are submitting hints, and you can only try one hint at a time.

And remember I have not touched Pascal for years.

Bazzao

[Blestan]

@Bazzao: glad that you found a working solution... keep using fpc and lazarus:) we will try to help ... more or less aggressively ( joke)
@wp: ok .. you are right ... i overreacted ) sorry again
@ : taazz : 1/ i will push some code later today
                  2/ just remark: in your code you do not take in consideration cpu endianness  .. it not a problem if the file is used always on the same machine but can become messy when file is moved to other device

[taazz]

@Bazzao: glad that you found a working solution... keep using fpc and lazarus:) we will try to help ... more or less aggressively ( joke)
@wp: ok .. you are right ... i overreacted ) sorry again
@ : taazz : 1/ i will push some code later today
                  2/ just remark: in your code you do not take in consideration cpu endianness  .. it not a problem if the file is used always on the same machine but can become messy when file is moved to other device

 

True I did not consider endiannes out of habit I guess. In any case I'll wait to see your code.

[Blestan]

@taazz : i do not understand why you assume max 12 records?

[Bazzao]

@Bazzao: glad that you found a working solution... keep using fpc and lazarus:) we will try to help ... more or less aggressively ( joke)
@wp: ok .. you are right ... i overreacted ) sorry again
@ : taazz : 1/ i will push some code later today
                  2/ just remark: in your code you do not take in consideration cpu endianness  .. it not a problem if the file is used always on the same machine but can become messy when file is moved to other device

Thanks Blestan & taazz & everybody.

I have a lot to read and decipher.

Bazzao

[taazz]

@taazz : i do not understand why you assume max 12 records?

I assume 12 is the max size of a single record not the max number of records. the number of records is calculated as "filesize" div "max record size" the smaller the "max record size" is the bigger the calculated total record number will be. I use 12 as 3 * sizeof(longint) leaving out the typ and txt fields, it is safe to use 13 since then goodTyp is always saved. It is one of those things that close enough is better than 100% accurate.

[Blestan]

hummmm .... ok

but this be pretty inaccurate if u have a lot of strings
at least add to the first calculation the number of _typ bytes ... or for every 12 records u will have an extra zombie one to truncate later...
... and sizeof() is better than just 12 (future proof)

anyway if i can find time today i will play with a solution

[taazz]

hummmm .... ok

but this be pretty inaccurate if u have a lot of strings
at least add to the first calculation the number of _typ bytes ... or for every 12 records u will have an extra zombie one to truncate later...
... and sizeof() is better than just 12 (future proof)

anyway if i can find time today i will play with a solution

that's pretty inaccurate even if you do not have any strings and it is by design. It helps to avoid buffer overruns while developing and testing the routines, after everything has been tested, then you fine tune it between memory usage and speed of reading.

[wp]

It is not necessary to know the record count exactly. Allocate a rough estimate (n) of the record count and use a counter (i) for the real record count. Whenever the counter value is evenly divisible by the preallocated block count, i.e. i mod n = 0, then allocate another n records. At the end SetLength to the counter value:

Code: Pascal  [Select][+][-]

1. const
2.   n = 500;
3. var
4.   i: Integer;
5.   data: array of TGoodRec;
6. ...
7.   i := 0;
8.   SetLength(data, 0);
9.   while not [end-of-file reached] do begin
10.     if i mod n = 0 then SetLength(data, Length(data) + n);
11.     data[i] := [Read record]
12.     inc(i);
13.   end;
14.   SetLength(data, i);

[Blestan]

another approach is to add a "header" record to indicate the number of records following ... 

[taazz]

I chose my current design for simplicity's shake, I showcased the basic usage of reading from and writing to files in the simplest way possible, I can always count on you guys to point out the shortcomings to the end user, required or not.

[Blestan]

voila

i also added an signature for testing if file is valid type

and it can be used on any stream including network

Code: [Select]

type
  TDataType= ( TypeStart=0, //indicates a start N elements
               Type1,
               Type2,
               Type3);

  TDataRecord = record
                typ: TDataType;
                lng,
                lat: Integer;
                txt: string
              end;

  TStreamRecord= packed record
                FHeader: Cardinal; // 4 bytes header;
                  FSize: Cardinal; // 4 bytes extra string for type 3 or N elements for type 0;
                   FLng,
                   FLat: Integer
  end;

  TDataArray=array of TDataRecord;


  function SaveTo( S: TStream; const AnArray: TDataArray; Append: Boolean): Integer;
  function ReadFrom( S: TStream; var AnArray: TDataArray): Integer;

var
  Form1: TForm1;

implementation

{$R *.lfm}

const Signature:Cardinal=$AABBCC00; // or what ever you want but keep the low byte 0

function SaveTo( S: TStream; const AnArray: TDataArray; Append: Boolean): Integer;
var Buf: TStreamRecord;
   i,ln: Integer;
begin
  Result:=0;
  if Append then S.Position := S.Size
    else begin
      S.Position := 0;
      S.Size := 0; //truncate
    end;
  // write first the header
  Buf.FHeader:=NtoLe(Signature); // Signature or ord(typeStart)
  Buf.FSize:=NtoLe(System.Length(AnArray));
  Buf.FLng:=0;  // zero them
  Buf.FLat:=0; //just in case
  S.WriteBuffer(Buf,SizeOf(Buf));

  // now the actual data

  for i:=Low(AnArray) to High(AnArray) do
   with AnArray[i] do
   begin
    Buf.FHeader:=NToLe(Signature or Byte(Ord(typ)));
    Ln:=System.Length(txt);
    Buf.FSize:=NtoLE(ln);
    Buf.FLat:=NtoLE(lat);
    Buf.FLng:=NtoLE(lng);
    S.WriteBuffer(Buf,SizeOf(Buf));
    // write extra text if any
    if ln>0 then S.WriteBuffer(txt[1],ln);
    Inc(Result)
   end
end;

function ReadFrom( S: TStream; var AnArray: TDataArray): Integer;
var BlockSz: Integer;
function ReadBlock: Integer;
var Buf: TStreamRecord;
   i,first,ln: Integer;
begin
  Result:=0;
  // read first the block header
  ln:=S.Read(Buf,SizeOf(Buf));
  if ln<>SizeOf(Buf) then exit;
  Buf.FHeader:=LEtoN(Buf.FHeader);
  if Buf.FHeader<>Signature then exit; // not a valid block header!
  ln:=LEtoN(Buf.FSize);
  first:=System.Length(AnArray);
  SetLength(AnArray,system.Length(AnArray)+Ln);
  for i:=first to High(AnArray) do
   begin
     ln:=S.Read(Buf,SizeOf(buf));
     if ln<>SizeOf(Buf) then exit(i);
     Buf.FHeader:=LeToN(Buf.FHeader);
     if (Buf.FHeader and $FFFFFF00)<>Signature then exit(i);
     AnArray[i].typ:=TDataType(Buf.FHeader);
     AnArray[i].lng:=LEtoN(Buf.FLng);
     AnArray[i].lat:=LEtoN(Buf.FLat);
     ln:=LEtoN(Buf.FSize);
     Pointer(AnArray[i].Txt):=nil; // safely init to '';
     if ln>0 then
      begin
        SetLength(AnArray[i].Txt,Ln);
        S.ReadBuffer(AnArray[1],ln);
      end
   end;
end;

begin
 Result:=0;
 repeat
   BlockSz:=ReadBlock;
   if BlockSz=0 then break
                else Inc(Result,BlockSz)
 until false;
end;