Recent

Author Topic: Binary files - explanation/examples needed.  (Read 3167 times)

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Binary files - explanation/examples needed.
« Reply #15 on: October 08, 2022, 07:04:43 pm »
Using AnsiString and dynamic arrays isn't going to cut the mustard.

You can hardly blame him for providing code that highlights the ambiguity of your suggestion. Particularly since OP (with all respect) has limited experience, you could have done with providing a concrete example when you joined this thread, and you're now in the position where you could usefully acknowledge his example.

OP (@TomTom): the problem with this code

Code: Pascal  [Select][+][-]
  1. var
  2.   a: record N: AnsiString; M: TDate; L: array of integer; end;
  3. begin
  4.   a.N := 'John Doe';
  5.   a.M := Now();
  6.   a.L := [10,20,30,40];
  7.  
  8.   MyFileStream.WriteBuffer(a, SizeOf(a));
  9.  

is that none of the three fields will contain useful text. An AnsiString and a dynamic array- neither declared with a length- is actually a pointer to somewhere else in memory (the heap), while a TDate is actually a binary floating-point number.

Even if the fields had been declared as e.g. String[32] they'd be problematic, since what would actually be saved would be a length byte- which might or might not be printable- followed by the text itself.

So writing a record to a file like that is absolutely fine when the fields are integers or carefully-padded arrays of characters, but in other cases one has to be careful.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Arioch

  • Sr. Member
  • ****
  • Posts: 421
Re: Binary files - explanation/examples needed.
« Reply #16 on: October 08, 2022, 07:42:19 pm »
...while a TDate is actually a binary floating-point number.

...which is not to be a problem for his stated task to prevent unexperienced users to meaningfully alter files in Notepad.

The other two types, though, were indeed chosen on purpose. And i did not even started with trees or linked lists of records :-)

Quote from: MarkMLl
is actually a pointer to somewhere else in memory (the heap)

...and? What would a novice make out of it?

See, Mark, this still says nothing to novices.
And if you really gonna explain them so they understand - you would eventually have to retell quite few chapters from programming textbooks.
Which, frankly, were written better explanaitons, than your or my ad hoc short forum snippet would ever be.

It is a rabbit hole. So i prefer just to say "don't jump into it".

Isn't a simple RECORD with all the inner fields needed to be enough? It's simple, just load the complete record with simple IO block read or something and directly have the app read from it in real time

...so says Jamie. And novices would trust him and do it.
And when novices would do how Jamie said, he would make U-turn and lash out on them for "simply just loading" because now, after they followed Jamie's advice and ruined their data by simply taking simple record with usual everyday types and simply dumping it with block I/O or something.
Because that was promised to "be enough".

you should know that managed types within a Record is not going to be savable to disk.

But you said it was not so! you said novice should "just load the complete record with simple IO block read" and that's it!
« Last Edit: October 08, 2022, 07:53:15 pm by Arioch »

Dzandaa

  • Full Member
  • ***
  • Posts: 248
  • From C# to Lazarus
Re: Binary files - explanation/examples needed.
« Reply #17 on: October 08, 2022, 08:55:42 pm »
Hi,

You can try to save your data as record.

Code: Pascal  [Select][+][-]
  1. unit LoadSaveRU;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, Spin;
  9.  
  10. type TestRecord = record
  11.   Name : string[20];
  12.   ID: Integer;
  13. end;
  14.  
  15. type
  16.  
  17.  { TLoadSaveRecordsForm }
  18.  
  19.  TLoadSaveRecordsForm = class(TForm)
  20.   BLoad: TButton;
  21.   BSave: TButton;
  22.   Display: TMemo;
  23.   LID: TLabel;
  24.   LName: TLabel;
  25.   OpenRecord: TOpenDialog;
  26.   SaveRecord: TSaveDialog;
  27.   SEID: TSpinEdit;
  28.   TBName: TEdit;
  29.   procedure BLoadClick(Sender: TObject);
  30.   procedure BSaveClick(Sender: TObject);
  31.  private
  32.  
  33.  public
  34.   MyRecord: TestRecord;
  35.   RStrm: file of TestRecord;
  36.  end;
  37.  
  38. var
  39.  LoadSaveRecordsForm: TLoadSaveRecordsForm;
  40.  
  41. implementation
  42.  
  43. {$R *.lfm}
  44.  
  45. { TLoadSaveRecordsForm }
  46.  
  47. procedure TLoadSaveRecordsForm.BSaveClick(Sender: TObject);
  48. begin
  49.  MyRecord.Name := TBName.Caption;
  50.  MyRecord.ID := SEID.Value;
  51.  Display.Append('Save:');
  52.  Display.Append('Name: ' + MyRecord.Name);
  53.  Display.Append('ID: ' + MyRecord.ID.ToString);
  54.  
  55.  if(SaveRecord.Execute) then
  56.  Begin
  57.   AssignFile(RStrm, SaveRecord.FileName);
  58.   rewrite(RStrm);
  59.   Write(RStrm, MyRecord);// Save Record
  60.  
  61.   CloseFile(RStrm);
  62.  end;
  63.  
  64. end;
  65.  
  66. procedure TLoadSaveRecordsForm.BLoadClick(Sender: TObject);
  67. begin
  68.  if(OpenRecord.Execute) then
  69.  Begin
  70.    AssignFile(RStrm, OpenRecord.FileName);
  71.    reset(RStrm);
  72.    MyRecord.Name := '';
  73.    MyRecord.ID := 0;
  74.    Read(RStrm, MyRecord);
  75.    Display.Append('Load:');
  76.    Display.Append('Name: ' + MyRecord.Name);
  77.    Display.Append('ID: ' + MyRecord.ID.ToString);
  78.  
  79.  
  80.    CloseFile(RStrm);
  81.  end;
  82. end;
  83.  
  84. end.
  85.  
  86.  

Just a quick idea :)

B->
« Last Edit: October 08, 2022, 08:57:25 pm by Dzandaa »
Dzandaa

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: Binary files - explanation/examples needed.
« Reply #18 on: October 08, 2022, 09:02:26 pm »
Hey, I like this poster  :)
The only true wisdom is knowing you know nothing

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Binary files - explanation/examples needed.
« Reply #19 on: October 09, 2022, 01:46:43 am »
You might want to take a look at Google's Protocol Buffers:
https://developers.google.com/protocol-buffers/docs/overview
https://github.com/lalexs75/protobuf-fpc

However, you could instead easily use any Object to JSON or XML serialization lib and simply crypt critical fields.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

BobDog

  • Sr. Member
  • ****
  • Posts: 394
Re: Binary files - explanation/examples needed.
« Reply #20 on: October 10, 2022, 12:40:53 am »

Save a text file in a shuffled state so it is unreadable, retrieve from file and shuffle back to original state.
I need my own random number generator so I can seed it.
I cannot seed the fp random number generator.
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. {$mode objfpc}
  4. uses
  5. sysutils;
  6. {$R+}
  7. // random number generator//
  8.   var
  9.   _a,_b,_c,_d:qword;
  10.  
  11. function rand64():qword;
  12. var
  13.  _e:qword;
  14.  begin
  15.    _e := _a - ((_b shl 7) or (_b shr (64 - 7)));
  16.    _a := _b xor ((_c shl 13) or (_c shr (64 - 13)));
  17.    _b := _c + ((_d shl 37) or (_d shr (64 - 37)));
  18.    _c := _d + _e;
  19.    _d := _e + _a ;
  20.    exit(_d);
  21. end;
  22.  
  23. function range(f:int64;l:int64):int64 ;
  24. begin
  25.     exit( f+ rand64 mod qword((l-f+1)));
  26. end;
  27.  
  28.  procedure seed(s:int64);
  29.  var
  30.  i:integer;
  31.  begin
  32.   _a:=s;_b:=s;_c:=s;_d:=s;
  33.     for i :=1 to 20 do
  34.     begin
  35.      rand64();
  36.     end;
  37. end;
  38.  // end of random number generator//  
  39.      
  40.   procedure save(s:ansistring ;filename:ansistring);
  41.    var
  42.    f:textfile;
  43.    begin
  44.    AssignFile(f,filename);
  45.    try
  46.    Rewrite(f);
  47.    writeln(f,s);
  48.    closefile(f);
  49.    except
  50.     on E: EInOutError do
  51.       writeln('File handling error occurred. Details: ', E.ClassName, '/', E.Message);
  52.   end;
  53.  end;
  54.        
  55.   function load(filename:ansistring):ansistring;
  56.    var
  57.     ret:ansistring='';
  58.     ans:ansistring='';
  59.     f:textfile;
  60.     begin
  61.     AssignFile(f,filename);
  62.     try
  63.     reset(f);
  64.     while not eof(f) do
  65.     begin
  66.       readln(f,ret);
  67.       ans:=ans+ret;
  68.     end;
  69.     CloseFile(f);
  70.      except
  71.     on E: EInOutError do
  72.      writeln('File handling error occurred. Details: ', E.Message);
  73.   end;
  74.     exit(ans);
  75.    end;
  76.  
  77.  
  78. procedure swap(var a, b:char);
  79. var temp: char;
  80. begin
  81.   temp := a; a := b; b := temp;
  82. end;
  83.  
  84. function shuffle( s:AnsiString):ansistring;
  85.    var
  86.    L1,n,i:int32;
  87.     begin
  88.     seed(2000);
  89.     L1:=length(s);
  90.     For n := 1 To length(s)-1 do  
  91.     begin
  92.     i:=range((n+1),L1);
  93.        Swap (s[n], s[i]);
  94.         end;
  95.      exit(s);
  96. End;
  97.  
  98. function shuffleback( s:AnsiString):ansistring;
  99.    var
  100.    L,L1,n:int32;
  101.    a:array of int32=nil;
  102.    begin
  103.    seed(2000);
  104.    L:=length(s);
  105.    L1:=length(s);
  106.    setlength(a,length(s));
  107.     For n := 1 To length(s)-1 do
  108.     a[L-n]:=range((n+1),L1);  
  109.     For n :=1 To length(s)-1 do
  110.      Swap (s[L-n],s[ (a[n]) ]);
  111.  exit(s);  
  112. End;
  113.  
  114. var
  115. s:ansistring ='I''m part of the union, ';
  116. res:ansistring;
  117. i:integer;
  118. begin
  119. for i:=1 to 5 do
  120. s:=s+s;
  121.  
  122. writeln('Original ansistring');
  123. writeln('"',s,'"');
  124. writeln;
  125. s:=shuffle(s);
  126. save(s,'cryptic.txt');
  127. res:=load('cryptic.txt');
  128. writeln('File content (cryptic.txt)');
  129. writeln('"',res,'"');
  130. s:=shuffleback(res);
  131. writeln;
  132. writeln('Back from the file and Shuffled back');
  133. writeln('"',s,'"');
  134. writeln;
  135. writeln('Press return to delete this file and exit  .. .. ');
  136.   readln;
  137.    DeleteFile('cryptic.txt');
  138. end.
  139.  

Kays

  • Hero Member
  • *****
  • Posts: 569
  • Whasup!?
    • KaiBurghardt.de
Re: Binary files – explanation/examples needed
« Reply #21 on: October 10, 2022, 01:16:39 am »
[…] [EXAMPLES needed]
Generics has only a couple of steps code-wise. […]
Yeah, but blindly writing code because “someone” said this is great is not “programming”. Programming entails understanding what you’re doing. TomTom needs to be able to come up with the same code as you posted.
[…] I cannot seed the fp random number generator.
Yes, you certainly can; usually by using randomize, but if your application needs to reproduce the same “random” values assign a value system.randSeed.
Yours Sincerely
Kai Burghardt

BobDog

  • Sr. Member
  • ****
  • Posts: 394
Re: Binary files - explanation/examples needed.
« Reply #22 on: October 10, 2022, 01:44:19 am »

Kays.
Randomize is no good for the above code.
I need to seed to get the same values again.
How do you use system.ranseed?
Thanks.
Regarding generics, it is the lesser of the two, generics or overloads.
TomTom's list:
floats, arrays, stringlists, strings.
All these need to be of fixed length of course, to save and retrieve from file.
This is why I introduced a shuffle method for text files as a possible alternative.
Pity I had to use my own random number generator, it looks chunky, but I need to be able to re-seed.



PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Binary files - explanation/examples needed.
« Reply #23 on: October 10, 2022, 07:21:55 am »
How do you use system.ranseed?

The values returned by Random are seeded by the value assigned to RandSeed. So if you assign the same value to RandSeed in different runs you'll get the same sequence of random numbers:

Code: Pascal  [Select][+][-]
  1. var
  2.   seed: LongInt;
  3. begin
  4.   Randomize;
  5.   seed := RandSeed;
  6.  
  7.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  8.  
  9.   Randseed := seed;
  10.  
  11.   // this will print the same numbers as above
  12.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  13. end.

You only need to be careful in multithreaded applications, because there is only a single random number state that is shared between threads and so if different threads call Random at the same time you will likely get a different sequence nevertheless.

abouchez

  • Full Member
  • ***
  • Posts: 110
    • Synopse
Re: Binary files - explanation/examples needed.
« Reply #24 on: October 10, 2022, 09:10:57 am »
mORMot allows binary or JSON serialization of a record or a dynamic array - since more than 10 years. ;)
(and also classes and other kind of instances, but a record could be fine for you)

The binary serialization is proprietary, but cross-platform (and compiler: you can share it with Delphi), and very fast, using the RTTI.
For JSON serialization, you only need to define once your record layout as text, since FPC still lacks of extended record RTTI (perhaps in next version?).

See for mORMot 1 (article from 10 years ago):
https://blog.synopse.info/?post/2011/03/12/TDynArray-and-Record-compare/load/save-using-fast-RTTI

Consider using mORMot 2, which is faster and a better target for the long term.
https://github.com/synopse/mORMot2

BobDog

  • Sr. Member
  • ****
  • Posts: 394
Re: Binary files - explanation/examples needed.
« Reply #25 on: October 10, 2022, 12:45:36 pm »
How do you use system.ranseed?

The values returned by Random are seeded by the value assigned to RandSeed. So if you assign the same value to RandSeed in different runs you'll get the same sequence of random numbers:

Code: Pascal  [Select][+][-]
  1. var
  2.   seed: LongInt;
  3. begin
  4.   Randomize;
  5.   seed := RandSeed;
  6.  
  7.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  8.  
  9.   Randseed := seed;
  10.  
  11.   // this will print the same numbers as above
  12.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  13. end.

You only need to be careful in multithreaded applications, because there is only a single random number state that is shared between threads and so if different threads call Random at the same time you will likely get a different sequence nevertheless.
thank you PascalDragon.
I can now use the built in random.
I notice that the built in randomrange (math unit) is a bit skew whiff.
Code: Pascal  [Select][+][-]
  1.  
  2. uses
  3. math;
  4.  
  5.  
  6. var
  7. a:array[1..3] of int32;
  8. i:longword;
  9.  
  10.  
  11. function range(mymin:int64;mymax:int64):int64;
  12.          begin
  13.       exit(trunc(int((Random*(Mymax-mymin+1)))+MyMin));
  14.  end;
  15.  var tmp:int64;
  16.  
  17. begin
  18. writeln('range ',low(a),'  to  ',high(a));
  19. for i:=low(a) to high(a) do a[i]:=0;
  20. for i:=1 to 100000 do
  21.  
  22. begin
  23. tmp:=randomrange(low(a),high(a));
  24. a[tmp]:=a[tmp]+1;
  25. end;
  26. for i:=1 to 3 do writeln(a[i]);
  27. writeln('-----');
  28. writeln(a[1]+a[2]+a[3]);
  29. writeln;
  30. for i:=low(a) to high(a) do a[i]:=0;
  31. for i:=1 to 100000 do
  32. begin
  33. tmp:=range(low(a),high(a));
  34. a[tmp]:=a[tmp]+1;
  35. end;
  36. for i:=1 to 3 do writeln(a[i]);
  37. writeln('-----');
  38. writeln(a[1]+a[2]+a[3]);
  39. readln;
  40. end.
  41.  
  42.        
  43.  
So I have to use a custom range.
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. {$mode objfpc}
  4. uses
  5. sysutils,Classes;
  6. {$R+}
  7.  
  8.  procedure SaveToFile(theString,filename:AnsiString);
  9. var
  10.   strm: TFileStream;
  11. begin
  12.   try
  13.     strm := TFileStream.Create(filename,fmCreate);
  14.     strm.Write(theString[1], length(theString));
  15.     strm.Free;
  16.   except
  17.     on E:Exception do
  18.       writeln('String could not be written. Details: ', E.ClassName, ': ', E.Message);
  19.   end;
  20. end;
  21.  
  22. function loadFromFile(filename:ansistring):ansistring;
  23. var
  24.   strm: TFileStream;
  25.   n: longint;
  26.   txt: ansistring='';
  27. begin
  28.   strm := TFileStream.Create(filename,fmOpenRead or fmShareDenyWrite);
  29.   try
  30.     n := strm.Size;
  31.     SetLength(txt, n);
  32.     strm.Read(txt[1], n);
  33.   finally
  34.     strm.Free;
  35.   end;
  36.   exit(txt);
  37. end;
  38.  
  39. function range(mymin:int64;mymax:int64):int64;
  40.  begin
  41.  exit(trunc(int((Random*(Mymax-mymin+1)))+MyMin));
  42.  end;
  43.        
  44.  
  45. procedure swap(var a, b:char);
  46. var temp: char;
  47. begin
  48.   temp := a; a := b; b := temp;
  49. end;
  50.  
  51. function shuffle( s:AnsiString):ansistring;
  52.    var
  53.    L1,n,i:int32;
  54.     begin
  55.      Randseed:=100;
  56.     L1:=length(s);
  57.     For n := 1 To length(s)-1 do  
  58.     begin
  59.     i:=range((n+1),L1);
  60.        Swap (s[n], s[i]);
  61.         end;
  62.      exit(s);
  63. End;
  64.  
  65. function shuffleback( s:AnsiString):ansistring;
  66.    var
  67.    L,L1,n:int32;
  68.    a:array of int32=nil;
  69.    begin
  70.     Randseed:=100;
  71.    L:=length(s);
  72.    L1:=length(s);
  73.    setlength(a,length(s));
  74.     For n := 1 To length(s)-1 do
  75.     a[L-n]:=range((n+1),L1);  
  76.     For n :=1 To length(s)-1 do
  77.      Swap (s[L-n],s[ (a[n]) ]);
  78.  exit(s);  
  79. End;
  80.  
  81. var
  82. s:ansistring ='You don''t get me, I''m part of the union. '+chr(10);
  83. res:ansistring;
  84. i:integer;
  85. begin
  86. for i:=1 to 3 do
  87. s:=s+s;
  88.  
  89. writeln('Original ansistring');
  90. writeln(s);
  91. writeln;
  92. s:=shuffle(s);
  93.  SaveToFile(S,'cryptic.txt');
  94.  res:=LoadFromFile('cryptic.txt');
  95. writeln('File content (cryptic.txt)');
  96. writeln(res);
  97. s:=shuffleback(res);
  98. writeln;
  99. writeln('Back from the file and Shuffled back');
  100. writeln(s);
  101. writeln;
  102. writeln('Press return to delete this file and exit  .. .. ');
  103.   readln;
  104.  DeleteFile('cryptic.txt');
  105. end.
  106.  
  107.  
Edit.
Use TfileStream for files of ansistring.

« Last Edit: October 11, 2022, 01:40:38 am by BobDog »

KodeZwerg

  • Hero Member
  • *****
  • Posts: 2007
  • Fifty shades of code.
    • Delphi & FreePascal
Re: Binary files - explanation/examples needed.
« Reply #26 on: October 10, 2022, 01:51:05 pm »
@BobDog, i would suggest to expand your experiment a little, add a crc/hash (encoded or not) directly after your string, so always read string in as two parts and disallow modified (crc/hash) mismatch to be "decoded". Just a suggestion to make it more stable. You can also play with some "salt" as input to make it work like a key.
« Last Edit: Tomorrow at 31:76:97 xm by KodeZwerg »

 

TinyPortal © 2005-2018