Lazarus

Free Pascal => Beginners => Topic started by: TomTom on October 07, 2022, 08:30:39 pm

Title: Binary files - explanation/examples needed.
Post by: TomTom on October 07, 2022, 08:30:39 pm
Hi. I have a program in my mind. It will need to save to and load data from files, but I don't want users to be able to edit those files outside my program (atleast easily). I always used text files and I don't know anything about using binary files which I imagine are harder to edit for a user. I need some guidance and examples on how to:

Save integers, floats, arrays, stringlists, strings to binary file
Load integers, floats, arrays, stringlists, strings from binary file

For example I have few tables like this one (array of arrays I guess)
Code: Pascal  [Select][+][-]
  1. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  2. | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 |
  3. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  4. | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 |
  5. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  6. | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 |
  7. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  8.  

How do I save this to a file and load it from a file?
Title: Re: Binary files - explanation/examples needed.
Post by: Arioch on October 07, 2022, 08:44:15 pm
you might want to us SQLite database and read mORMot documentation - a lot of new concepts to learn about the whole applciatio ndesign, from ground up. But in the long run would be worth it, the more contradicting concepts you know - the more flexibility you have about choosing better approaches to problem.

https://stephan-bester.medium.com/getting-started-with-mormot-and-lazarus-free-pascal-c7bc7e98a866

You might use TDbf component and set of files - but dbf files are read/edited by many tools and ISAM approach is often considered obsolete.

You might design a TComponent derivative to hold all your data and then save it into a binary .LFM file, but you would have to learn how LCL/VCL streaming works.

You might pick any "objects serialization" library (for example, again, mORMot. Many XML or JSON libraries would provide limited serialization features too), but text XML or text JSON document can be edited not much harder than your text. Also, 3rd-party serialization and .LFM compontent serialization is very similar things.
Title: Re: Binary files - explanation/examples needed.
Post by: domasz on October 07, 2022, 10:22:45 pm
Quote from: TomTom on October 07, 2022, 08:30:39 pm
Hi. I have a program in my mind. It will need to save to and load data from files, but I don't want users to be able to edit those files outside my program (atleast easily). I always used text files and I don't know anything about using binary files

How about saving to text files but encrypting the files? You can use a very simple algorithm like ROT13.
Title: Re: Binary files - explanation/examples needed.
Post by: Bart on October 07, 2022, 10:28:51 pm
You can save any arbitrary data using a TFileStream, it's reading it back that may be more difficult.
Either you need to know (at forehand) the order and type of things you have written to that file, or you need to have some sort of "marker" indicating type (and possibly length) of the data that is stored after that marker.
If the data you store has a fixed length (e.g. array of some type), read up on typed files (https://wiki.freepascal.org/typed_files).

Bart
Title: Re: Binary files - explanation/examples needed.
Post by: MarkMLl on October 07, 2022, 10:29:47 pm
Quote from: domasz on October 07, 2022, 10:22:45 pm
How about saving to text files but encrypting the files? You can use a very simple algorithm like ROT13.

That's wrong and dangerous. There is absolutely no way that ROT13 etc. can be described as encryption, it offers absolutely no protection.

The easiest way- which doesn't even need any programming- is to take a checksum of each file and store it separately.

MarkMLl
Title: Re: Binary files - explanation/examples needed.
Post by: marcov on October 07, 2022, 11:39:38 pm
I use a config file (an early fork of FPC/lazarus' TXMLConfig), but the user registry (the access control of the app) of course shouldn't be editable.

So I simply run that code through a tandem of AES (from dcpcrypt2) and  a base64 stream encoder and store that in an XML node and save it. With a few per application specific salts to avoid people copying one user reg to another. Viola, configuration editable in XML, except the pieces that shouldn't.

Not air tight (people that get a compromised account could back up and restore the configuration after the compromised account was wiped), but tackling that would complicate restoring backups in an environment that is already not too good in doing them.

You could also do that with textual files. All the difficult code already is done.

Wire the resulting stream through streamex/streamio's assignstream and you can even mostly reuse your preexisting read and write code.
Title: Re: Binary files - explanation/examples needed.
Post by: domasz on October 08, 2022, 01:59:19 am
Quote from: MarkMLl on October 07, 2022, 10:29:47 pm

That's wrong and dangerous. There is absolutely no way that ROT13 etc. can be described as encryption, it offers absolutely no protection.

He only needs to make things *harder* for people to edit the files. He doesn't need real encryption nor protection. And using ROT13 is safer than using binary.
Title: Re: Binary files - explanation/examples needed.
Post by: BobDog on October 08, 2022, 02:02:20 am
You could try this, (block save/load)
Some of the pascal experts will give you a yea or nay on this method.
Code: Pascal  [Select][+][-]
  1.  
  2. uses
  3. sysutils; // only for deletefile
  4.  
  5. type aoi = array [1..12]  of integer;
  6. type aod = array[1..8] of double;
  7. type aoa=array[1..3] of array[1..10] of string;
  8. type s26=string[26];
  9.  
  10. function filelength(filename:ansistring):int32;
  11. Var F : File Of byte;
  12. var L:longword;
  13. begin
  14. Assign (F,filename);
  15.   Reset (F);
  16.   L:=FileSize(F);
  17.   Close (F);
  18.   exit(L);
  19. end;
  20.  
  21. generic procedure loadfile<T>(var content:T;filename:ansistring);
  22.    Var Fin : File;
  23.    Var x:longint;
  24.    begin
  25.    x:=filelength(filename);
  26.    Assign (Fin,filename);
  27.    Reset (Fin,x);
  28.    BlockRead (Fin,content,1);
  29.    close(fin);
  30. end;
  31.  
  32.  generic procedure savefile<T>(content:T ;filename:ansistring);
  33.     var
  34.     fout:file;
  35.     begin
  36.     Assign(fout,filename);
  37.     Rewrite(fout,sizeof(content));
  38.     blockwrite(fout,content,1);
  39.     close(fout);
  40.   end;
  41.  
  42.   var
  43.   i,j,k:int32;
  44.   a:aoi=(-6,45,67,-34,2,0,30,-8,67,3,5,7);
  45.   return :aoi;
  46.   g:s26='abcdefghijklmnopqrstuvwxyz';
  47.   h:s26='';
  48.  
  49.   d:aod=(-0.7,45.5,90.3,-56.98,-0.006,45.1,100.6,-1.1);
  50.   dret:aod;
  51.  
  52.   e:aoa=( ('13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' ),
  53.            ( '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '2/44/70' , '22/44/70' ),
  54.            ( '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' ));
  55.  
  56.   eret:aoa;
  57.   begin
  58.   writeln('Array [1..12]of integer');
  59.   specialize savefile<aoi>(a,'test.txt');
  60.  i:=filelength('test.txt');
  61.  specialize loadfile<aoi>(return,'test.txt');
  62.   writeln('file length ',i);
  63.  for j:=low(return) to high(return) do writeln(return[j]);
  64.  writeln;
  65.  
  66.  writeln('string[26]');
  67.  specialize savefile<s26>(g,'teststring.txt');
  68.  i:=filelength('teststring.txt');
  69.  specialize loadfile<s26>(h,'teststring.txt');
  70.   writeln('file length ',i);
  71.  writeln(h);
  72.  writeln;
  73.  
  74.  writeln('Array [1..8] of double');
  75.   specialize savefile<aod>(d,'testdouble.txt');
  76.    i:=filelength('testdouble.txt');
  77.     specialize loadfile<aod>(dret,'testdouble.txt');
  78.      writeln('file length ',i);
  79.   for j:=low(dret) to high(dret) do writeln(dret[j] :5:5);
  80.   writeln;
  81.  
  82. writeln('array[1..3] of array[1..10] of string');
  83.     specialize savefile<aoa>(e,'testaoa.txt');
  84.     i:=filelength('testaoa.txt');
  85.     specialize loadfile<aoa>(eret,'testaoa.txt');
  86.     writeln('file length ',i);
  87.   for j:=1 to 3 do
  88.   begin
  89.   for k:=1 to 10 do
  90.   begin
  91.   write(eret[j,k],' ');
  92.   end;
  93.   writeln;
  94.   end;
  95.    
  96.   writeln('Press return to delete these files and exit  .. .. ');
  97.   readln;
  98.    DeleteFile('test.txt');
  99.    DeleteFile('teststring.txt');
  100.    DeleteFile('testdouble.txt');
  101.    DeleteFile('testaoa.txt');
  102.  
  103.   end.
  104.        
  105.    
  106.  
  107.  
Tested 64 bit 3.2.2
Title: Re: Binary files – explanation/examples needed
Post by: Kays on October 08, 2022, 02:45:25 am
Title: Re: Binary files - explanation/examples needed.
Post by: MarkMLl on October 08, 2022, 08:41:37 am
Quote from: domasz on October 08, 2022, 01:59:19 am
He only needs to make things *harder* for people to edit the files. He doesn't need real encryption nor protection. And using ROT13 is safer than using binary.

That's still nowhere near good enough. Anybody who mentions encryption and ROT-13 in the same sentence should be disbarred.

MarkMLl
Title: Re: Binary files - explanation/examples needed.
Post by: bobby100 on October 08, 2022, 11:14:25 am
I am using XOR with very long key, and a XOR-ed MD5 checksum at the end of the file.
The key shouldn't be stored as a plain string in your program (google about DeCSS).

I have seen some serious commercial products that are using 4kb XOR-key with offset-rotation after every 4kb of reading or writing.

In some projects, I've used a modified ZIP-files as a storage. Here was the intention to have a couple of the files in one file, something like the latest MS Office files (.docx and similar).

Some PC-viruses used modified Self-Extracting RAR files in the past. They just modified the Magic Number ( https://en.wikipedia.org/wiki/File_format#Magic_number , https://en.wikipedia.org/wiki/List_of_file_signatures ) from RAR to RAP, which was enough for anti-viruses to not be able to detect and open the archive.
Title: Re: Binary files – explanation/examples needed
Post by: BobDog on October 08, 2022, 11:15:08 am
Quote from: Kays on October 08, 2022, 02:45:25 am
  • FPC/Lazarus Wiki: typed files (https://wiki.freepascal.org/Special:PermaLink/133727) and file handling (https://wiki.freepascal.org/Special:PermaLink/154044)
  • Cantù: files in the Pascal language (https://www.marcocantu.com/epascal/English/ch12files.htm)
  • Pascal programming: binary files and records (https://web.archive.org/web/20210613140920/https://www.pascal-programming.info/lesson11.php#jump5) (memento)
  • WikiBook: Files (https://en.wikibooks.org/wiki/Special:PermaLink/4191487)
  • TStringList (https://www.freepascal.org/docs-html/rtl/classes/tstringlist.html) is a class. As such you will need to serialize/deserialize (https://wiki.lazarus.freepascal.org/Streaming_components) it: something like file of TStringList won’t work, because TStringList is actually a pointer (https://www.freepascal.org/docs-html/current/prog/progsu166.html). I think you’ll need to master the basics first before worrying about such issues.
  • @BobDog: Generics? Really? Isn’t that overkill? TomTom doesn’t sound confident in file handling beyond text. Do you think he’s familiar with generics then? KISS.
Don't know Kays.
Maybe.
The request:
Save integers, floats, arrays, stringlists, strings to binary file
Load integers, floats, arrays, stringlists, strings from binary file
[EXAMPLES needed]
Generics has only a couple of steps code-wise.
It is an easy enough set up, although the help files really make it look tough going.
Config files, ROT13, encryption, SQLite database . . . look KISS not, to me.




Title: Re: Binary files - explanation/examples needed.
Post by: jamie on October 08, 2022, 05:08:48 pm
Hmm,

Isn't a simple RECORD with all the inner fields needed to be enough?

It's simple, just load the complete record with simple IO block read or something and directly have the app read from it in real time instead of all this text to bin conversion stuff.

 if this is something that is integrated to the app then just make the first field a version byte and size of record and ensure the loading code test for that first.
Title: Re: Binary files - explanation/examples needed.
Post by: Arioch on October 08, 2022, 05:19:47 pm
Quote from: jamie on October 08, 2022, 05:08:48 pm
Isn't a simple RECORD with all the inner fields needed to be enough?

Depends. Without understanding serialization and implementation formats of data types...

Code: Pascal  [Select][+][-]
  1. var
  2.   a: record N: AnsiString; M: TDate; L: array of integer; end;
  3. begin
  4.   a.N := 'John Doe';
  5.   a.M := Now();
  6.   a.L := [10,20,30,40];
  7.  
  8.   MyFileStream.WriteBuffer(a, SizeOf(a));
  9.  

Looks so very simple, right? But what would actually be written to disk?
Title: Re: Binary files - explanation/examples needed.
Post by: jamie on October 08, 2022, 06:49:23 pm
Quote from: Arioch on October 08, 2022, 05:19:47 pm
Quote from: jamie on October 08, 2022, 05:08:48 pm
Isn't a simple RECORD with all the inner fields needed to be enough?

Depends. Without understanding serialization and implementation formats of data types...

Code: Pascal  [Select][+][-]
  1. var
  2.   a: record N: AnsiString; M: TDate; L: array of integer; end;
  3. begin
  4.   a.N := 'John Doe';
  5.   a.M := Now();
  6.   a.L := [10,20,30,40];
  7.  
  8.   MyFileStream.WriteBuffer(a, SizeOf(a));
  9.  

Looks so very simple, right? But what would actually be written to disk?

For someone that has been hanging out in Delphi land for so many years after claiming only a brief exposer etc., you should know that managed types within a Record is not going to be savable to disk.

 Using AnsiString and dynamic arrays isn't going to cut the mustard.

 So, you are either instigating or you really don't know the difference?
 
 Sorry for others that had to read this but!

That is my take on it.

Title: Re: Binary files - explanation/examples needed.
Post by: MarkMLl on October 08, 2022, 07:04:43 pm
Quote from: jamie on October 08, 2022, 06:49:23 pm
Using AnsiString and dynamic arrays isn't going to cut the mustard.

You can hardly blame him for providing code that highlights the ambiguity of your suggestion. Particularly since OP (with all respect) has limited experience, you could have done with providing a concrete example when you joined this thread, and you're now in the position where you could usefully acknowledge his example.

OP (@TomTom): the problem with this code

Code: Pascal  [Select][+][-]
  1. var
  2.   a: record N: AnsiString; M: TDate; L: array of integer; end;
  3. begin
  4.   a.N := 'John Doe';
  5.   a.M := Now();
  6.   a.L := [10,20,30,40];
  7.  
  8.   MyFileStream.WriteBuffer(a, SizeOf(a));
  9.  

is that none of the three fields will contain useful text. An AnsiString and a dynamic array- neither declared with a length- is actually a pointer to somewhere else in memory (the heap), while a TDate is actually a binary floating-point number.

Even if the fields had been declared as e.g. String[32] they'd be problematic, since what would actually be saved would be a length byte- which might or might not be printable- followed by the text itself.

So writing a record to a file like that is absolutely fine when the fields are integers or carefully-padded arrays of characters, but in other cases one has to be careful.

MarkMLl
Title: Re: Binary files - explanation/examples needed.
Post by: Arioch on October 08, 2022, 07:42:19 pm
Quote from: MarkMLl on October 08, 2022, 07:04:43 pm
...while a TDate is actually a binary floating-point number.

...which is not to be a problem for his stated task to prevent unexperienced users to meaningfully alter files in Notepad.

The other two types, though, were indeed chosen on purpose. And i did not even started with trees or linked lists of records :-)

Quote from: MarkMLl
is actually a pointer to somewhere else in memory (the heap)

...and? What would a novice make out of it?

See, Mark, this still says nothing to novices.
And if you really gonna explain them so they understand - you would eventually have to retell quite few chapters from programming textbooks.
Which, frankly, were written better explanaitons, than your or my ad hoc short forum snippet would ever be.

It is a rabbit hole. So i prefer just to say "don't jump into it".

Quote from: jamie on October 08, 2022, 05:08:48 pm
Isn't a simple RECORD with all the inner fields needed to be enough? It's simple, just load the complete record with simple IO block read or something and directly have the app read from it in real time

...so says Jamie. And novices would trust him and do it.
And when novices would do how Jamie said, he would make U-turn and lash out on them for "simply just loading" because now, after they followed Jamie's advice and ruined their data by simply taking simple record with usual everyday types and simply dumping it with block I/O or something.
Because that was promised to "be enough".

Quote from: jamie on October 08, 2022, 06:49:23 pm
you should know that managed types within a Record is not going to be savable to disk.

But you said it was not so! you said novice should "just load the complete record with simple IO block read" and that's it!
Title: Re: Binary files - explanation/examples needed.
Post by: Dzandaa on October 08, 2022, 08:55:42 pm
Hi,

You can try to save your data as record.

Code: Pascal  [Select][+][-]
  1. unit LoadSaveRU;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.  Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, Spin;
  9.  
  10. type TestRecord = record
  11.   Name : string[20];
  12.   ID: Integer;
  13. end;
  14.  
  15. type
  16.  
  17.  { TLoadSaveRecordsForm }
  18.  
  19.  TLoadSaveRecordsForm = class(TForm)
  20.   BLoad: TButton;
  21.   BSave: TButton;
  22.   Display: TMemo;
  23.   LID: TLabel;
  24.   LName: TLabel;
  25.   OpenRecord: TOpenDialog;
  26.   SaveRecord: TSaveDialog;
  27.   SEID: TSpinEdit;
  28.   TBName: TEdit;
  29.   procedure BLoadClick(Sender: TObject);
  30.   procedure BSaveClick(Sender: TObject);
  31.  private
  32.  
  33.  public
  34.   MyRecord: TestRecord;
  35.   RStrm: file of TestRecord;
  36.  end;
  37.  
  38. var
  39.  LoadSaveRecordsForm: TLoadSaveRecordsForm;
  40.  
  41. implementation
  42.  
  43. {$R *.lfm}
  44.  
  45. { TLoadSaveRecordsForm }
  46.  
  47. procedure TLoadSaveRecordsForm.BSaveClick(Sender: TObject);
  48. begin
  49.  MyRecord.Name := TBName.Caption;
  50.  MyRecord.ID := SEID.Value;
  51.  Display.Append('Save:');
  52.  Display.Append('Name: ' + MyRecord.Name);
  53.  Display.Append('ID: ' + MyRecord.ID.ToString);
  54.  
  55.  if(SaveRecord.Execute) then
  56.  Begin
  57.   AssignFile(RStrm, SaveRecord.FileName);
  58.   rewrite(RStrm);
  59.   Write(RStrm, MyRecord);// Save Record
  60.  
  61.   CloseFile(RStrm);
  62.  end;
  63.  
  64. end;
  65.  
  66. procedure TLoadSaveRecordsForm.BLoadClick(Sender: TObject);
  67. begin
  68.  if(OpenRecord.Execute) then
  69.  Begin
  70.    AssignFile(RStrm, OpenRecord.FileName);
  71.    reset(RStrm);
  72.    MyRecord.Name := '';
  73.    MyRecord.ID := 0;
  74.    Read(RStrm, MyRecord);
  75.    Display.Append('Load:');
  76.    Display.Append('Name: ' + MyRecord.Name);
  77.    Display.Append('ID: ' + MyRecord.ID.ToString);
  78.  
  79.  
  80.    CloseFile(RStrm);
  81.  end;
  82. end;
  83.  
  84. end.
  85.  
  86.  

Just a quick idea :)

B->
Title: Re: Binary files - explanation/examples needed.
Post by: jamie on October 08, 2022, 09:02:26 pm
Hey, I like this poster  :)
Title: Re: Binary files - explanation/examples needed.
Post by: avra on October 09, 2022, 01:46:43 am
You might want to take a look at Google's Protocol Buffers:
https://developers.google.com/protocol-buffers/docs/overview
https://github.com/lalexs75/protobuf-fpc

However, you could instead easily use any Object to JSON or XML serialization lib and simply crypt critical fields.
Title: Re: Binary files - explanation/examples needed.
Post by: BobDog on October 10, 2022, 12:40:53 am

Save a text file in a shuffled state so it is unreadable, retrieve from file and shuffle back to original state.
I need my own random number generator so I can seed it.
I cannot seed the fp random number generator.
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. {$mode objfpc}
  4. uses
  5. sysutils;
  6. {$R+}
  7. // random number generator//
  8.   var
  9.   _a,_b,_c,_d:qword;
  10.  
  11. function rand64():qword;
  12. var
  13.  _e:qword;
  14.  begin
  15.    _e := _a - ((_b shl 7) or (_b shr (64 - 7)));
  16.    _a := _b xor ((_c shl 13) or (_c shr (64 - 13)));
  17.    _b := _c + ((_d shl 37) or (_d shr (64 - 37)));
  18.    _c := _d + _e;
  19.    _d := _e + _a ;
  20.    exit(_d);
  21. end;
  22.  
  23. function range(f:int64;l:int64):int64 ;
  24. begin
  25.     exit( f+ rand64 mod qword((l-f+1)));
  26. end;
  27.  
  28.  procedure seed(s:int64);
  29.  var
  30.  i:integer;
  31.  begin
  32.   _a:=s;_b:=s;_c:=s;_d:=s;
  33.     for i :=1 to 20 do
  34.     begin
  35.      rand64();
  36.     end;
  37. end;
  38.  // end of random number generator//  
  39.      
  40.   procedure save(s:ansistring ;filename:ansistring);
  41.    var
  42.    f:textfile;
  43.    begin
  44.    AssignFile(f,filename);
  45.    try
  46.    Rewrite(f);
  47.    writeln(f,s);
  48.    closefile(f);
  49.    except
  50.     on E: EInOutError do
  51.       writeln('File handling error occurred. Details: ', E.ClassName, '/', E.Message);
  52.   end;
  53.  end;
  54.        
  55.   function load(filename:ansistring):ansistring;
  56.    var
  57.     ret:ansistring='';
  58.     ans:ansistring='';
  59.     f:textfile;
  60.     begin
  61.     AssignFile(f,filename);
  62.     try
  63.     reset(f);
  64.     while not eof(f) do
  65.     begin
  66.       readln(f,ret);
  67.       ans:=ans+ret;
  68.     end;
  69.     CloseFile(f);
  70.      except
  71.     on E: EInOutError do
  72.      writeln('File handling error occurred. Details: ', E.Message);
  73.   end;
  74.     exit(ans);
  75.    end;
  76.  
  77.  
  78. procedure swap(var a, b:char);
  79. var temp: char;
  80. begin
  81.   temp := a; a := b; b := temp;
  82. end;
  83.  
  84. function shuffle( s:AnsiString):ansistring;
  85.    var
  86.    L1,n,i:int32;
  87.     begin
  88.     seed(2000);
  89.     L1:=length(s);
  90.     For n := 1 To length(s)-1 do  
  91.     begin
  92.     i:=range((n+1),L1);
  93.        Swap (s[n], s[i]);
  94.         end;
  95.      exit(s);
  96. End;
  97.  
  98. function shuffleback( s:AnsiString):ansistring;
  99.    var
  100.    L,L1,n:int32;
  101.    a:array of int32=nil;
  102.    begin
  103.    seed(2000);
  104.    L:=length(s);
  105.    L1:=length(s);
  106.    setlength(a,length(s));
  107.     For n := 1 To length(s)-1 do
  108.     a[L-n]:=range((n+1),L1);  
  109.     For n :=1 To length(s)-1 do
  110.      Swap (s[L-n],s[ (a[n]) ]);
  111.  exit(s);  
  112. End;
  113.  
  114. var
  115. s:ansistring ='I''m part of the union, ';
  116. res:ansistring;
  117. i:integer;
  118. begin
  119. for i:=1 to 5 do
  120. s:=s+s;
  121.  
  122. writeln('Original ansistring');
  123. writeln('"',s,'"');
  124. writeln;
  125. s:=shuffle(s);
  126. save(s,'cryptic.txt');
  127. res:=load('cryptic.txt');
  128. writeln('File content (cryptic.txt)');
  129. writeln('"',res,'"');
  130. s:=shuffleback(res);
  131. writeln;
  132. writeln('Back from the file and Shuffled back');
  133. writeln('"',s,'"');
  134. writeln;
  135. writeln('Press return to delete this file and exit  .. .. ');
  136.   readln;
  137.    DeleteFile('cryptic.txt');
  138. end.
  139.  
Title: Re: Binary files – explanation/examples needed
Post by: Kays on October 10, 2022, 01:16:39 am
Quote from: BobDog on October 08, 2022, 11:15:08 am
[…] [EXAMPLES needed]
Generics has only a couple of steps code-wise. […]
Yeah, but blindly writing code because “someone” said this is great is not “programming”. Programming entails understanding what you’re doing. TomTom needs to be able to come up with the same code as you posted.
Quote from: BobDog on October 10, 2022, 12:40:53 am
[…] I cannot seed the fp random number generator.
Yes, you certainly can; usually by using randomize (https://www.freepascal.org/docs-html/current/rtl/system/randomize.html), but if your application needs to reproduce the same “random” values assign a value system.randSeed (https://www.freepascal.org/docs-html/current/rtl/system/randseed.html).
Title: Re: Binary files - explanation/examples needed.
Post by: BobDog on October 10, 2022, 01:44:19 am

Kays.
Randomize is no good for the above code.
I need to seed to get the same values again.
How do you use system.ranseed?
Thanks.
Regarding generics, it is the lesser of the two, generics or overloads.
TomTom's list:
floats, arrays, stringlists, strings.
All these need to be of fixed length of course, to save and retrieve from file.
This is why I introduced a shuffle method for text files as a possible alternative.
Pity I had to use my own random number generator, it looks chunky, but I need to be able to re-seed.


Title: Re: Binary files - explanation/examples needed.
Post by: PascalDragon on October 10, 2022, 07:21:55 am
Quote from: BobDog on October 10, 2022, 01:44:19 am
How do you use system.ranseed?

The values returned by Random are seeded by the value assigned to RandSeed. So if you assign the same value to RandSeed in different runs you'll get the same sequence of random numbers:

Code: Pascal  [Select][+][-]
  1. var
  2.   seed: LongInt;
  3. begin
  4.   Randomize;
  5.   seed := RandSeed;
  6.  
  7.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  8.  
  9.   Randseed := seed;
  10.  
  11.   // this will print the same numbers as above
  12.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  13. end.

You only need to be careful in multithreaded applications, because there is only a single random number state that is shared between threads and so if different threads call Random at the same time you will likely get a different sequence nevertheless.
Title: Re: Binary files - explanation/examples needed.
Post by: abouchez on October 10, 2022, 09:10:57 am
mORMot allows binary or JSON serialization of a record or a dynamic array - since more than 10 years. ;)
(and also classes and other kind of instances, but a record could be fine for you)

The binary serialization is proprietary, but cross-platform (and compiler: you can share it with Delphi), and very fast, using the RTTI.
For JSON serialization, you only need to define once your record layout as text, since FPC still lacks of extended record RTTI (perhaps in next version?).

See for mORMot 1 (article from 10 years ago):
https://blog.synopse.info/?post/2011/03/12/TDynArray-and-Record-compare/load/save-using-fast-RTTI

Consider using mORMot 2, which is faster and a better target for the long term.
https://github.com/synopse/mORMot2
Title: Re: Binary files - explanation/examples needed.
Post by: BobDog on October 10, 2022, 12:45:36 pm
Quote from: PascalDragon on October 10, 2022, 07:21:55 am
Quote from: BobDog on October 10, 2022, 01:44:19 am
How do you use system.ranseed?

The values returned by Random are seeded by the value assigned to RandSeed. So if you assign the same value to RandSeed in different runs you'll get the same sequence of random numbers:

Code: Pascal  [Select][+][-]
  1. var
  2.   seed: LongInt;
  3. begin
  4.   Randomize;
  5.   seed := RandSeed;
  6.  
  7.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  8.  
  9.   Randseed := seed;
  10.  
  11.   // this will print the same numbers as above
  12.   Writeln(Random(MaxLongInt), ' ', Random(MaxLongInt), ' ', Random(MaxLongInt));
  13. end.

You only need to be careful in multithreaded applications, because there is only a single random number state that is shared between threads and so if different threads call Random at the same time you will likely get a different sequence nevertheless.
thank you PascalDragon.
I can now use the built in random.
I notice that the built in randomrange (math unit) is a bit skew whiff.
Code: Pascal  [Select][+][-]
  1.  
  2. uses
  3. math;
  4.  
  5.  
  6. var
  7. a:array[1..3] of int32;
  8. i:longword;
  9.  
  10.  
  11. function range(mymin:int64;mymax:int64):int64;
  12.          begin
  13.       exit(trunc(int((Random*(Mymax-mymin+1)))+MyMin));
  14.  end;
  15.  var tmp:int64;
  16.  
  17. begin
  18. writeln('range ',low(a),'  to  ',high(a));
  19. for i:=low(a) to high(a) do a[i]:=0;
  20. for i:=1 to 100000 do
  21.  
  22. begin
  23. tmp:=randomrange(low(a),high(a));
  24. a[tmp]:=a[tmp]+1;
  25. end;
  26. for i:=1 to 3 do writeln(a[i]);
  27. writeln('-----');
  28. writeln(a[1]+a[2]+a[3]);
  29. writeln;
  30. for i:=low(a) to high(a) do a[i]:=0;
  31. for i:=1 to 100000 do
  32. begin
  33. tmp:=range(low(a),high(a));
  34. a[tmp]:=a[tmp]+1;
  35. end;
  36. for i:=1 to 3 do writeln(a[i]);
  37. writeln('-----');
  38. writeln(a[1]+a[2]+a[3]);
  39. readln;
  40. end.
  41.  
  42.        
  43.  
So I have to use a custom range.
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. {$mode objfpc}
  4. uses
  5. sysutils,Classes;
  6. {$R+}
  7.  
  8.  procedure SaveToFile(theString,filename:AnsiString);
  9. var
  10.   strm: TFileStream;
  11. begin
  12.   try
  13.     strm := TFileStream.Create(filename,fmCreate);
  14.     strm.Write(theString[1], length(theString));
  15.     strm.Free;
  16.   except
  17.     on E:Exception do
  18.       writeln('String could not be written. Details: ', E.ClassName, ': ', E.Message);
  19.   end;
  20. end;
  21.  
  22. function loadFromFile(filename:ansistring):ansistring;
  23. var
  24.   strm: TFileStream;
  25.   n: longint;
  26.   txt: ansistring='';
  27. begin
  28.   strm := TFileStream.Create(filename,fmOpenRead or fmShareDenyWrite);
  29.   try
  30.     n := strm.Size;
  31.     SetLength(txt, n);
  32.     strm.Read(txt[1], n);
  33.   finally
  34.     strm.Free;
  35.   end;
  36.   exit(txt);
  37. end;
  38.  
  39. function range(mymin:int64;mymax:int64):int64;
  40.  begin
  41.  exit(trunc(int((Random*(Mymax-mymin+1)))+MyMin));
  42.  end;
  43.        
  44.  
  45. procedure swap(var a, b:char);
  46. var temp: char;
  47. begin
  48.   temp := a; a := b; b := temp;
  49. end;
  50.  
  51. function shuffle( s:AnsiString):ansistring;
  52.    var
  53.    L1,n,i:int32;
  54.     begin
  55.      Randseed:=100;
  56.     L1:=length(s);
  57.     For n := 1 To length(s)-1 do  
  58.     begin
  59.     i:=range((n+1),L1);
  60.        Swap (s[n], s[i]);
  61.         end;
  62.      exit(s);
  63. End;
  64.  
  65. function shuffleback( s:AnsiString):ansistring;
  66.    var
  67.    L,L1,n:int32;
  68.    a:array of int32=nil;
  69.    begin
  70.     Randseed:=100;
  71.    L:=length(s);
  72.    L1:=length(s);
  73.    setlength(a,length(s));
  74.     For n := 1 To length(s)-1 do
  75.     a[L-n]:=range((n+1),L1);  
  76.     For n :=1 To length(s)-1 do
  77.      Swap (s[L-n],s[ (a[n]) ]);
  78.  exit(s);  
  79. End;
  80.  
  81. var
  82. s:ansistring ='You don''t get me, I''m part of the union. '+chr(10);
  83. res:ansistring;
  84. i:integer;
  85. begin
  86. for i:=1 to 3 do
  87. s:=s+s;
  88.  
  89. writeln('Original ansistring');
  90. writeln(s);
  91. writeln;
  92. s:=shuffle(s);
  93.  SaveToFile(S,'cryptic.txt');
  94.  res:=LoadFromFile('cryptic.txt');
  95. writeln('File content (cryptic.txt)');
  96. writeln(res);
  97. s:=shuffleback(res);
  98. writeln;
  99. writeln('Back from the file and Shuffled back');
  100. writeln(s);
  101. writeln;
  102. writeln('Press return to delete this file and exit  .. .. ');
  103.   readln;
  104.  DeleteFile('cryptic.txt');
  105. end.
  106.  
  107.  
Edit.
Use TfileStream for files of ansistring.

Title: Re: Binary files - explanation/examples needed.
Post by: KodeZwerg on October 10, 2022, 01:51:05 pm
@BobDog, i would suggest to expand your experiment a little, add a crc/hash (encoded or not) directly after your string, so always read string in as two parts and disallow modified (crc/hash) mismatch to be "decoded". Just a suggestion to make it more stable. You can also play with some "salt" as input to make it work like a key.
TinyPortal © 2005-2018