Store fileversion info in a FileOfRecord

[varg300]

I have a program that stores a data record in a standard binary file:
Var
Data : File OF SomeRecord

I'm certain that i'm at some piont will make another version of this program that is likley to have more variables in the record, whitch raises the following problem. If my newer program atempts to read a file from the old version that has fewer variables stored in it, the program will obviously crash.

So is there anyway to store the fileversion of the datafile in it, and have the program check what version the datafile is before it atempts to read it, so i can make it only read the variables that are expected to be present in the file?

Code: [Select]

Procedure CheckFileVersion
var version : SomeType
      datafile : file of SomeRecord

begin
FileVersion(data,version)     //Somehow check the fileversion
If version = 1 then ReadThisSetOfVariables
if version = 2 then begin
                     ReadThisSetOfVariables
                     ReadAdditionalVariables
                        end;
if version = 3 then begin
                      ReadThisSetOfVariables
                      ReadAdditionalVariables
                      ReadEvenMoreVariables
                        end;
This is obviously not going to work, but you see what i'm thinking 

And of course i will need some way to store the version in the file somehow

[Dibo]

I always store 4 bytes (size of integer) with my "database" version at the beginning of the file and then save rest of data. This same with load, first I read 4 bytes and check version and then load right method (I store history of load function to get backward compability like LoadVer1, LoadVer2 etc.
Maybe you can better use some kind of INI or XML file? (lazarus has TXMLConfig class)? Then this problem disappear. If you worry that some one could edit this file, then encrypt xml first and then save to file

[ludob]

You can change the definition of SomeRecord to become "dual purpose" by using a variant part:

Code: [Select]

type
  SomeRecord= record
    case boolean of
    true: (
          // your existing record members come here
          );
    false: (
       RecordVersion:dword;
       );
  end;
This way you  can keep your file of SomeRecord. In the first record you read or write SomeRecord.RecordVersion and the next records are your data which you use as before.

[User137]

This is one of the reasons why i use TFileStream.

[varg300]

Hehe i know FileStream is a better option but i have not figured out how to uses it.

Dibo: After the program has read the first 4 bytes, and then opens the file to read the record, would not that cause the cursor to start from the top, making it read the first bytes instead of the record?
Can you elaborate please? 

 ludob:
Does this mean that if one try to read a variable that's not present in the record (Or file for that matter) the boolean returns false, making the "False variable" been read, and if you read a variable that exeist in the record, the value turns true, giving you the actual record?

Both of your approaches are exacltly what i'm looking for, but i'm afraid i'll need some more explanation before i decide. Whatever is easiest to get working

[Dibo]

Dibo: After the program has read the first 4 bytes, and then opens the file to read the record, would not that cause the cursor to start from the top, making it read the first bytes instead of the record?
Can you elaborate please? 

I have always use TFileStream instead of file of record because it is more flexible. TFileStream.Read automaticaly move cursor by read bytes (writting too) or you can do this manualy by TFileStream.Seek method. Using is very simple:

Code: Pascal  [Select][+][-]

1. var
2.   f: TFileStream;
3. const
4.   iVersion: Integer = 5;
5. begin
6.   f := TFileStream.Create(FileName, fmCreate);
7.   try
8.     f.Write(iVersion, SizeOf(Integer));
9.     f.Write(SomeRecord, SizeOf(SomeRecord));
10.   finally
11.     f.Free;
12.   end;
13. end;
14.  

Reading:

Code: Pascal  [Select][+][-]

1. var
2.   f: TFileStream;
3.   iVersion: Integer;
4. begin
5.   f := TFileStream.Create(FileName, fmOpenRead);
6.   try
7.     // Version
8.     f.Read(iVersion, SizeOf(Integer));
9.     // Cursor is after 4 bytes now
10.     case iVersion of
11.     1: // Your code;
12.     2: // Your code;
13.     end;
14.   finally
15.     f.Free;
16.   end;
17. end;
18.  

[varg300]

Thanks. It will need some modifications to the code, but that's definatly worth it if i get it working.

The other solution also seemed like a good idea, but i did not fully understand how to use it.
If i read the file with a TRUE argument, will it red the main variables, and just the version number if i read the file with a FALSE statment?
How about writing to the file, will it be overwritten with just one of the "Parts" of the record or will they both be preseved?

 I'm gonna copy the entire project and test it on the copy just in case i screw it up hehe.

[KpjComp]

If i read the file with a TRUE argument

The TRUE & FALSE is confusing you, it is a weird coding concept I know. 

It's how you do Unions in Pascal, basically any variables declared in the True section uses the same memory location as those in the False and vice-versa.

One disadvantage to this approach is that the first record that you want to store your version information is going to be the same size as the rest.   A TFileStream is much better.

[KpjComp]

Code: [Select]

f.Read(iVersion, SizeOf(Integer));

The only thing I would say is use f.ReadBuffer instead.  ReadBuffer will raise an exception if not all bytes can be read,  Read will just return the number of bytes read, that could be less.   The same goes for write, use WriteBuffer, you never know you might just run out of disk space.

[ludob]

ludob:
Does this mean that if one try to read a variable that's not present in the record (Or file for that matter) the boolean returns false, making the "False variable" been read, and if you read a variable that exeist in the record, the value turns true, giving you the actual record?

Both of your approaches are exacltly what i'm looking for, but i'm afraid i'll need some more explanation before i decide. Whatever is easiest to get working

Perhaps I wasn't clear enough. The unnamed variant part (case boolean) means that this is basically the same as the C union. You don't use the true or false. That is why I said

and the next records are your data which you use as before

It just means that you defined 2 overlaying definitions for the same chunck of memory. Example:

Code: [Select]

  SomeRecord= record
    case boolean of
    true: (
          a,b,c,d,e,f,g,h,i;byte;
          );
    false: (
       val:dword;
       );
This is a record of 8 bytes. If you store $01020304 in Somerecord.val. You will find $04 in Somerecord.a, $03 in Somerecord.b, etc (little endian cpu such as i386). This technique is used quite frequently in lowel level api's where memory blocks can have different meanings according to circumstances.

One disadvantage to this approach is that the first record that you want to store your version information is going to be the same size as the rest.

This is exactly the purpose. I introduced this as an easy option that allowed to continue using File of SomeRecode which the OP is already using. He can use this technique with minimal changes to his existing program. The only change to his program is that the first data start at record 2 instead of 1 and that record 1contains the version.
 

[KpjComp]

This is exactly the purpose. I introduced this as an easy option

Yes, it was good advice.  I wasn't contradicting you here, it's just my personal preference is Streams.

[varg300]

@ludob
so what you are saying is that if i -as an example have this record

Code: [Select]

SomeRecord= record
    case boolean of
    true: (
          shortstring1,shortstring2,something : shortstring;
          somenumber,anothernumber : integer;
          someboolean : boolean;
          );
    false: (
       version:dword;
       );when the file is read the record will have all the variables as usual, but 'version' will hold a pre-defined version number in addition to all the variables?

so if i then decide to make another version that can read the "old verion" for compatibiity reasons, the recordfile will still read even if the program is trying to read variables not present in the record file?

To be honest, i find this a bit confusing.
May i ask you to provide a noob with a working example, sir? 

[KpjComp]

when the file is read the record will have all the variables as usual, but 'version' will hold a pre-defined version number in addition to all the variables?

No,.  Basically you'll be using the first record to just store the version number.
All you need to remember, is when accessing the first record the values in version will have the correct version number.   When you read record number 2 it will contain garbage, because all the others fields will be taking this space up instead.

In your example, each record is about 256*3+4*2+1 (32bit) = 777 bytes, your first record will be storing the version number, it will still take 777 bytes but the first 4 bytes of this will store the version info.   So what will happen is your first record is actually record 2, record 1 is reserved for your version info.

[Pascaluvr]

hi varg300,

I think you may still be still a little confused how the variant record works.

Suppose you define this:

Code: [Select]

type
  SomeRecord= record
    case boolean of
    true: (
          // your existing record members come here
          );
    false: (
       RecordVersion:dword;
       VersionDesc:  String[31];
       VersionDate:  tDateTime;
       );
  end;

SomeRecord will have length of the longer of all the fields in the True clause or all the fields in the false clause;  You also are not restricted to just 2 record formats.

 You could code:

Code: [Select]

type
  SomeRecord= record
    case Integer of of
     0: (
       RecordVersion:dword;
       VersionDesc:  String[31];
       VersionDate:  tDateTime;
       );
   1: (
          // your existing record members come here
          );
   2: (
          // your modified record members come here
          );
    3: (
          // your 'modified' modified record members come here
          );
   etc.

 end;


You never need to set True or False ( or 1, 2. 3, etc) anywhere in your program.


The problem you will have though is that the record length will (usually) change with each modification and so you will have problems accessing different versions of the file.

A technique to avoid this is to simply add a 'Reserved for future use field' as the last field in the record:

Code: [Select]

type
  SomeRecord= record
    case boolean of
    true: (
          // your existing record members come here

         // and then
         Reserved:  array[0..7] of LongInt;  // Reserve 32 bytes for future use     
          );
    false: (
       RecordVersion:dword;
       VersionDesc:  String[31];
       VersionDate:  tDateTime;
       );
  end;

Now, as you add new fields to the record, simply reduce the size of the Reserved array.

A few notes on the above:

1) You can ensure that your 'new' fields are initialised in your old file by:
    FillChar(mySomeRecord, SizeOf(SomeRecord), #0);
   before you create the first record.  Doing this will ensure that new binary fields will be zero and new strings will be ''

2) Clearly this wastes disk space, but considering that I used this technique 40 years ago when disk space was at a premium, with todays terabyte disks, the wastage should be minimal.

3) Note to Reserve bytes for future use I used an array of LongInt.  I could have used an array of char, but this introduces a new problem  Fields in a record may have some 'padding' between them.  For example, a LongInt field will be aligned on a 4 byte boundary.  (google Free Pascal PackedRecords for more info).
A useful technique here is to try to keep the record 4 byte aligned.  e.g. if you have a field that is string[30], make it string[31] so that it is a multiple of 4 bytes in length.


***********

If you are prepared to learn a new file structure, then the FileStrean mentioned above is certainly one approach, but there is another that I think is even better:

I am just learning SQL (specifically SQLITE) and it is amazing.  To achieve what you want you do not even need to worry about a FileVersionNumber.  In the future, when you wish to add another field to your record you simply ADD COLUMN and the new fields in your existing file can either be set to NULL or given a DEFAULT value.  Your modified program will still run on your old data file.

Hope this helps

[varg300]

yes, that helped me a lot!
I will give it a try tomorrow (its midnight here now) and see how it goes.
I might nag some more if it don't get it working though... hehe

Thanks for all your help guys! 

By the way: The SQL thing you were talking about, requires an SQL server, right? I might try that some time in the future, as i do have the MySQL server installed on my computer. I'm planning to use this program at work, and there we have a dedicated server just for SQL databases.
But i'll stick to the old-fashioned FileOfRecor, at least for now...