Recent

Author Topic: Binary files - explanation/examples needed.  (Read 1762 times)

TomTom

  • Full Member
  • ***
  • Posts: 163
Binary files - explanation/examples needed.
« on: October 07, 2022, 08:30:39 pm »
Hi. I have a program in my mind. It will need to save to and load data from files, but I don't want users to be able to edit those files outside my program (atleast easily). I always used text files and I don't know anything about using binary files which I imagine are harder to edit for a user. I need some guidance and examples on how to:

Save integers, floats, arrays, stringlists, strings to binary file
Load integers, floats, arrays, stringlists, strings from binary file

For example I have few tables like this one (array of arrays I guess)
Code: Pascal  [Select][+][-]
  1. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  2. | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 | 13/23/55 |
  3. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  4. | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 | 22/44/70 |
  5. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  6. | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 | 33/74/90 |
  7. +----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  8.  

How do I save this to a file and load it from a file?

Arioch

  • Sr. Member
  • ****
  • Posts: 414
Re: Binary files - explanation/examples needed.
« Reply #1 on: October 07, 2022, 08:44:15 pm »
you might want to us SQLite database and read mORMot documentation - a lot of new concepts to learn about the whole applciatio ndesign, from ground up. But in the long run would be worth it, the more contradicting concepts you know - the more flexibility you have about choosing better approaches to problem.

https://stephan-bester.medium.com/getting-started-with-mormot-and-lazarus-free-pascal-c7bc7e98a866

You might use TDbf component and set of files - but dbf files are read/edited by many tools and ISAM approach is often considered obsolete.

You might design a TComponent derivative to hold all your data and then save it into a binary .LFM file, but you would have to learn how LCL/VCL streaming works.

You might pick any "objects serialization" library (for example, again, mORMot. Many XML or JSON libraries would provide limited serialization features too), but text XML or text JSON document can be edited not much harder than your text. Also, 3rd-party serialization and .LFM compontent serialization is very similar things.

domasz

  • Full Member
  • ***
  • Posts: 103
Re: Binary files - explanation/examples needed.
« Reply #2 on: October 07, 2022, 10:22:45 pm »
Hi. I have a program in my mind. It will need to save to and load data from files, but I don't want users to be able to edit those files outside my program (atleast easily). I always used text files and I don't know anything about using binary files

How about saving to text files but encrypting the files? You can use a very simple algorithm like ROT13.

Bart

  • Hero Member
  • *****
  • Posts: 4818
    • Bart en Mariska's Webstek
Re: Binary files - explanation/examples needed.
« Reply #3 on: October 07, 2022, 10:28:51 pm »
You can save any arbitrary data using a TFileStream, it's reading it back that may be more difficult.
Either you need to know (at forehand) the order and type of things you have written to that file, or you need to have some sort of "marker" indicating type (and possibly length) of the data that is stored after that marker.
If the data you store has a fixed length (e.g. array of some type), read up on typed files.

Bart

MarkMLl

  • Hero Member
  • *****
  • Posts: 5561
Re: Binary files - explanation/examples needed.
« Reply #4 on: October 07, 2022, 10:29:47 pm »
How about saving to text files but encrypting the files? You can use a very simple algorithm like ROT13.

That's wrong and dangerous. There is absolutely no way that ROT13 etc. can be described as encryption, it offers absolutely no protection.

The easiest way- which doesn't even need any programming- is to take a checksum of each file and store it separately.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 10392
  • FPC developer.
Re: Binary files - explanation/examples needed.
« Reply #5 on: October 07, 2022, 11:39:38 pm »
I use a config file (an early fork of FPC/lazarus' TXMLConfig), but the user registry (the access control of the app) of course shouldn't be editable.

So I simply run that code through a tandem of AES (from dcpcrypt2) and  a base64 stream encoder and store that in an XML node and save it. With a few per application specific salts to avoid people copying one user reg to another. Viola, configuration editable in XML, except the pieces that shouldn't.

Not air tight (people that get a compromised account could back up and restore the configuration after the compromised account was wiped), but tackling that would complicate restoring backups in an environment that is already not too good in doing them.

You could also do that with textual files. All the difficult code already is done.

Wire the resulting stream through streamex/streamio's assignstream and you can even mostly reuse your preexisting read and write code.

domasz

  • Full Member
  • ***
  • Posts: 103
Re: Binary files - explanation/examples needed.
« Reply #6 on: October 08, 2022, 01:59:19 am »

That's wrong and dangerous. There is absolutely no way that ROT13 etc. can be described as encryption, it offers absolutely no protection.

He only needs to make things *harder* for people to edit the files. He doesn't need real encryption nor protection. And using ROT13 is safer than using binary.

BobDog

  • Sr. Member
  • ****
  • Posts: 363
Re: Binary files - explanation/examples needed.
« Reply #7 on: October 08, 2022, 02:02:20 am »
You could try this, (block save/load)
Some of the pascal experts will give you a yea or nay on this method.
Code: Pascal  [Select][+][-]
  1.  
  2. uses
  3. sysutils; // only for deletefile
  4.  
  5. type aoi = array [1..12]  of integer;
  6. type aod = array[1..8] of double;
  7. type aoa=array[1..3] of array[1..10] of string;
  8. type s26=string[26];
  9.  
  10. function filelength(filename:ansistring):int32;
  11. Var F : File Of byte;
  12. var L:longword;
  13. begin
  14. Assign (F,filename);
  15.   Reset (F);
  16.   L:=FileSize(F);
  17.   Close (F);
  18.   exit(L);
  19. end;
  20.  
  21. generic procedure loadfile<T>(var content:T;filename:ansistring);
  22.    Var Fin : File;
  23.    Var x:longint;
  24.    begin
  25.    x:=filelength(filename);
  26.    Assign (Fin,filename);
  27.    Reset (Fin,x);
  28.    BlockRead (Fin,content,1);
  29.    close(fin);
  30. end;
  31.  
  32.  generic procedure savefile<T>(content:T ;filename:ansistring);
  33.     var
  34.     fout:file;
  35.     begin
  36.     Assign(fout,filename);
  37.     Rewrite(fout,sizeof(content));
  38.     blockwrite(fout,content,1);
  39.     close(fout);
  40.   end;
  41.  
  42.   var
  43.   i,j,k:int32;
  44.   a:aoi=(-6,45,67,-34,2,0,30,-8,67,3,5,7);
  45.   return :aoi;
  46.   g:s26='abcdefghijklmnopqrstuvwxyz';
  47.   h:s26='';
  48.  
  49.   d:aod=(-0.7,45.5,90.3,-56.98,-0.006,45.1,100.6,-1.1);
  50.   dret:aod;
  51.  
  52.   e:aoa=( ('13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' , '13/23/55' ),
  53.            ( '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '22/44/70' , '2/44/70' , '22/44/70' ),
  54.            ( '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' , '33/74/90' ));
  55.  
  56.   eret:aoa;
  57.   begin
  58.   writeln('Array [1..12]of integer');
  59.   specialize savefile<aoi>(a,'test.txt');
  60.  i:=filelength('test.txt');
  61.  specialize loadfile<aoi>(return,'test.txt');
  62.   writeln('file length ',i);
  63.  for j:=low(return) to high(return) do writeln(return[j]);
  64.  writeln;
  65.  
  66.  writeln('string[26]');
  67.  specialize savefile<s26>(g,'teststring.txt');
  68.  i:=filelength('teststring.txt');
  69.  specialize loadfile<s26>(h,'teststring.txt');
  70.   writeln('file length ',i);
  71.  writeln(h);
  72.  writeln;
  73.  
  74.  writeln('Array [1..8] of double');
  75.   specialize savefile<aod>(d,'testdouble.txt');
  76.    i:=filelength('testdouble.txt');
  77.     specialize loadfile<aod>(dret,'testdouble.txt');
  78.      writeln('file length ',i);
  79.   for j:=low(dret) to high(dret) do writeln(dret[j] :5:5);
  80.   writeln;
  81.  
  82. writeln('array[1..3] of array[1..10] of string');
  83.     specialize savefile<aoa>(e,'testaoa.txt');
  84.     i:=filelength('testaoa.txt');
  85.     specialize loadfile<aoa>(eret,'testaoa.txt');
  86.     writeln('file length ',i);
  87.   for j:=1 to 3 do
  88.   begin
  89.   for k:=1 to 10 do
  90.   begin
  91.   write(eret[j,k],' ');
  92.   end;
  93.   writeln;
  94.   end;
  95.    
  96.   writeln('Press return to delete these files and exit  .. .. ');
  97.   readln;
  98.    DeleteFile('test.txt');
  99.    DeleteFile('teststring.txt');
  100.    DeleteFile('testdouble.txt');
  101.    DeleteFile('testaoa.txt');
  102.  
  103.   end.
  104.        
  105.    
  106.  
  107.  
Tested 64 bit 3.2.2
« Last Edit: October 09, 2022, 02:55:32 pm by BobDog »

Kays

  • Sr. Member
  • ****
  • Posts: 464
  • Whasup!?
    • KaiBurghardt.de
Re: Binary files – explanation/examples needed
« Reply #8 on: October 08, 2022, 02:45:25 am »
Yours Sincerely
Kai Burghardt

MarkMLl

  • Hero Member
  • *****
  • Posts: 5561
Re: Binary files - explanation/examples needed.
« Reply #9 on: October 08, 2022, 08:41:37 am »
He only needs to make things *harder* for people to edit the files. He doesn't need real encryption nor protection. And using ROT13 is safer than using binary.

That's still nowhere near good enough. Anybody who mentions encryption and ROT-13 in the same sentence should be disbarred.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

bobby100

  • Full Member
  • ***
  • Posts: 111
    • Malzilla
Re: Binary files - explanation/examples needed.
« Reply #10 on: October 08, 2022, 11:14:25 am »
I am using XOR with very long key, and a XOR-ed MD5 checksum at the end of the file.
The key shouldn't be stored as a plain string in your program (google about DeCSS).

I have seen some serious commercial products that are using 4kb XOR-key with offset-rotation after every 4kb of reading or writing.

In some projects, I've used a modified ZIP-files as a storage. Here was the intention to have a couple of the files in one file, something like the latest MS Office files (.docx and similar).

Some PC-viruses used modified Self-Extracting RAR files in the past. They just modified the Magic Number ( https://en.wikipedia.org/wiki/File_format#Magic_number , https://en.wikipedia.org/wiki/List_of_file_signatures ) from RAR to RAP, which was enough for anti-viruses to not be able to detect and open the archive.
https://gitlab.com/bobby100 - my Lazarus components and units
https://sourceforge.net/u/boban_spasic/profile/ - my open source apps

https://malzilla.org/ - remainder at my previous life as a web security expert

BobDog

  • Sr. Member
  • ****
  • Posts: 363
Re: Binary files – explanation/examples needed
« Reply #11 on: October 08, 2022, 11:15:08 am »
Don't know Kays.
Maybe.
The request:
Save integers, floats, arrays, stringlists, strings to binary file
Load integers, floats, arrays, stringlists, strings from binary file
[EXAMPLES needed]
Generics has only a couple of steps code-wise.
It is an easy enough set up, although the help files really make it look tough going.
Config files, ROT13, encryption, SQLite database . . . look KISS not, to me.





jamie

  • Hero Member
  • *****
  • Posts: 4905
Re: Binary files - explanation/examples needed.
« Reply #12 on: October 08, 2022, 05:08:48 pm »
Hmm,

Isn't a simple RECORD with all the inner fields needed to be enough?

It's simple, just load the complete record with simple IO block read or something and directly have the app read from it in real time instead of all this text to bin conversion stuff.

 if this is something that is integrated to the app then just make the first field a version byte and size of record and ensure the loading code test for that first.
The only true wisdom is knowing you know nothing

Arioch

  • Sr. Member
  • ****
  • Posts: 414
Re: Binary files - explanation/examples needed.
« Reply #13 on: October 08, 2022, 05:19:47 pm »
Isn't a simple RECORD with all the inner fields needed to be enough?

Depends. Without understanding serialization and implementation formats of data types...

Code: Pascal  [Select][+][-]
  1. var
  2.   a: record N: AnsiString; M: TDate; L: array of integer; end;
  3. begin
  4.   a.N := 'John Doe';
  5.   a.M := Now();
  6.   a.L := [10,20,30,40];
  7.  
  8.   MyFileStream.WriteBuffer(a, SizeOf(a));
  9.  

Looks so very simple, right? But what would actually be written to disk?

jamie

  • Hero Member
  • *****
  • Posts: 4905
Re: Binary files - explanation/examples needed.
« Reply #14 on: October 08, 2022, 06:49:23 pm »
Isn't a simple RECORD with all the inner fields needed to be enough?

Depends. Without understanding serialization and implementation formats of data types...

Code: Pascal  [Select][+][-]
  1. var
  2.   a: record N: AnsiString; M: TDate; L: array of integer; end;
  3. begin
  4.   a.N := 'John Doe';
  5.   a.M := Now();
  6.   a.L := [10,20,30,40];
  7.  
  8.   MyFileStream.WriteBuffer(a, SizeOf(a));
  9.  

Looks so very simple, right? But what would actually be written to disk?

For someone that has been hanging out in Delphi land for so many years after claiming only a brief exposer etc., you should know that managed types within a Record is not going to be savable to disk.

 Using AnsiString and dynamic arrays isn't going to cut the mustard.

 So, you are either instigating or you really don't know the difference?
 
 Sorry for others that had to read this but!

That is my take on it.

The only true wisdom is knowing you know nothing

 

TinyPortal © 2005-2018