Recent

Author Topic: Reading variant from a stream  (Read 3367 times)

Renat.Su

  • Full Member
  • ***
  • Posts: 230
    • Renat.Su
Reading variant from a stream
« on: February 24, 2018, 10:01:10 am »
A small puzzle.
In a stream (TFileStream or TMemoryStream, for example), a Variant type is stored, written there from code compiled in 32-bit FPC mode. How to read the same block of memory from the 64-bit FreePascal code?
Nuance ;) Variant in 32-bit FPC is stored in 16 bytes of memory. Variant 64-bit - 24 bytes
Are the desicion simpler and more beautiful? Is this possible without a low-level parsing of variant record memory block?

Sample code, if someone does not understand the task
Sample code compiled in 32-bit FPC where is storing variant
Code: Pascal  [Select][+][-]
  1. ASomeStream.WriteBuffer(AVariantVar, SizeOf(AVariantVar));
  2. { where AVariantVar is variant (ordinal type, float etc not pointer or String or TObject etc) }
Wrong sample with 64-bit FP compiler where is reading variant
Code: Pascal  [Select][+][-]
  1. ASomeStream.ReadBuffer(AVariantVar, SizeOf(AVariantVar));
  2. { This not right sample. Because in 32bit variant is stored in 16 bytes but in 64bit FPC compiled is stored in 24 bytes}
I understand that it's better not to save the variant in the stream. But nevertheless, how to read it?
« Last Edit: February 24, 2018, 10:04:10 am by Renat.Su »

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Reading variant from a stream
« Reply #1 on: February 24, 2018, 10:16:09 am »
That is the wrong way to save a variant. You need to identify its data type and save the type and data or raise an exception if the data in the variant is not supported. For example saving a variant with string data the way you shown, will not save the string data only the pointer to the string.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

ASerge

  • Hero Member
  • *****
  • Posts: 2223
Re: Reading variant from a stream
« Reply #2 on: February 24, 2018, 10:32:39 am »
As pointed out @taazz, it is more correct to read/write depending on the type. But if it's rough, taking into account that there are no pointers, then instead of SizeOf (TVarData) use 16 in read/write, and add "TVarData(V).PRecInfo := nil" for clearing after reading.

Thaddy

  • Hero Member
  • *****
  • Posts: 14211
  • Probably until I exterminate Putin.
Re: Reading variant from a stream
« Reply #3 on: February 24, 2018, 10:38:21 am »
Both You, Taazz and ASerge may be wrong: and refer to a variant type, while you probably mean a variant record (a.k.a UNION). To store and read those, you would declare the variant record as packed and indeed use SizeOf(Record). Don't confuse the two. I suspect that is the case here.
But you may want to store the actual type used as a separate field, so it becomes easier to read the actual type back. This is not strictly necessary, though.

Code: Pascal  [Select][+][-]
  1. type
  2.   TMyVariantRecord = packed record
  3.   case byte of
  4.   0:(AsInteger:integer);
  5.   1:(AsSingle:Single);
  6.   2:(AsDouble:Double);
  7.   end;
This can subsequently easily be stored and read back to and from a stream on any system. The stream blocks should be of size SizeOf(TMyVariantRecord) which equals the size of the largest variant field.
« Last Edit: February 24, 2018, 10:52:46 am by Thaddy »
Specialize a type, not a var.

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Reading variant from a stream
« Reply #4 on: February 24, 2018, 10:52:18 am »
Both You, Taazz and ASerge may be wrong: and refer to a variant type, while you probably mean a variant record (a.k.a UNION). To store and read those, you would declare the variant record as packed and indeed use SizeOf(Record). Don't confuse the two. I suspect that is the case here.

Why do you suspect that when Renat.Su explicitly used the phrase "Variant type" in his original post?

Thaddy

  • Hero Member
  • *****
  • Posts: 14211
  • Probably until I exterminate Putin.
Re: Reading variant from a stream
« Reply #5 on: February 24, 2018, 10:55:32 am »
Why do you suspect that when Renat.Su explicitly used the phrase "Variant type" in his original post?
From content. It is indeed my  interpretation but loads of people mix up variant types with variant records. Which then ends up in over-complicated code... Trust me... this is not the first time this happens.. :(
His streaming code seems to indicate (only works) with variant records, not variant types.
« Last Edit: February 24, 2018, 10:57:11 am by Thaddy »
Specialize a type, not a var.

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Reading variant from a stream
« Reply #6 on: February 24, 2018, 10:58:08 am »
Both You, Taazz and ASerge may be wrong: and refer to a variant type, while you probably mean a variant record (a.k.a UNION). To store and read those, you would declare the variant record as packed and indeed use SizeOf(Record). Don't confuse the two. I suspect that is the case here.
1) No! I'm talking about the type variant and not a variant record.
2) a variant is a record in memory that stores metadata with the data. Depending on the size of the data they might be saved inside the record it self (integers, booleans, some floats etc) or a pointer to the actual data (strings, objects, interfaces etc) is saved.
So saving the variant the way its shown it saves the metadata and in some cases the data it self. In others it only saves pointers to the data which can not be used after loading.

But you may want to store the actual type used as a separate field, so it becomes easier to read the actual type back. This is not strictly necessary, though.
well the type is saved with the metadata the data on the other hand not so much.
Are you talking about open parameters? I doubt that is what the TS is talking about.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Thaddy

  • Hero Member
  • *****
  • Posts: 14211
  • Probably until I exterminate Putin.
Re: Reading variant from a stream
« Reply #7 on: February 24, 2018, 11:00:37 am »
I know you are talking about the variant type, because he talks about variant type. I think he mixed things up and means a variant record, which are easy to stream as you well know...
Let him explain what he really means...
Specialize a type, not a var.

rvk

  • Hero Member
  • *****
  • Posts: 6112
Re: Reading variant from a stream
« Reply #8 on: February 24, 2018, 11:02:48 am »
His streaming code seems to indicate (only works) with variant records, not variant types.
Are you sure his code doesn't work. He knows what types work and which don't because he specifically mentioned those.
Quote
{ where AVariantVar is variant (ordinal type, float etc not pointer or String or TObject etc ) }

Renat.Su

  • Full Member
  • ***
  • Posts: 230
    • Renat.Su
Re: Reading variant from a stream
« Reply #9 on: February 24, 2018, 12:15:02 pm »
Thank you all for your answers and discussion!
1) About that is to store variant in a stream this is bad and not necessary - I know and that I mentioned at the end of the first post. About ways of how to save the variant value in stream - it is also understandable and the task's salt is not in it.
I will specify. Imagine: we can not change what writes to the stream the variant in the 32-bit program. But question how is the best way to read the variant from the stream in 64-bit code software where the variant is was stored in 32-bit software
2) I'm sorry about the fact that I did not quite correctly identify the type that is stored in the stream. This is an variant, that is,
Code: Pascal  [Select][+][-]
  1. var AVariantVar: Variant;
When I mentioned the record, I meant that the variant is a kind of structured memory block like a Pascal record if is need to parse.
3) I'm sorry for my english ;)
« Last Edit: February 24, 2018, 12:19:16 pm by Renat.Su »

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Reading variant from a stream
« Reply #10 on: February 24, 2018, 12:43:51 pm »
Thank you all for your answers and discussion!
1) About that is to store variant in a stream this is bad and not necessary - I know and that I mentioned at the end of the first post. About ways of how to save the variant value in stream - it is also understandable and the task's salt is not in it.
I will specify. Imagine: we can not change what writes to the stream the variant in the 32-bit program. But question how is the best way to read the variant from the stream in 64-bit code software where the variant is was stored in 32-bit software
2) I'm sorry about the fact that I did not quite correctly identify the type that is stored in the stream. This is an variant, that is,
Code: Pascal  [Select][+][-]
  1. var AVariantVar: Variant;
When I mentioned the record, I meant that the variant is a kind of structured memory block like a Pascal record if is need to parse.
3) I'm sorry for my english ;)
the variant record is defined as TVarData you can see the details for your self. https://www.freepascal.org/docs-html/rtl/system/tvardata.html you have to change the code to save the size before the record it self. after that you can use the given size to decide if you are reading a 64bit or a 32bit variant and use the vbytes array to move things around if needed. From a quick look looks like a direct read should work the two pointers that change the size of the record for 64bit are part of the varRecord which is the last portion of the record itself.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

jamie

  • Hero Member
  • *****
  • Posts: 6091
Re: Reading variant from a stream
« Reply #11 on: February 24, 2018, 03:41:39 pm »
I recently ran into an issue using variants where as I have an old app from the D days that used the variant
type in a file storage format and works great, I don't use them for any pointer references.  But when I ported
the app over it only worked in the 32 bit port.

 After digging in I found that the variant size has changed to 24 bytes over the 16 bytes so I then had to
make a TVarData32 type so I could port the same app to 32 bits..

 Maybe we could have a TvarData64 and TvarData32 types that are specific and the
TVarData type that points to the current target as it does now..

 At least when it comes to file storage for those variant types within that are not effected by pointers it would
be a better transition.

 I know I should of written a specific variant record years ago for these I used it in but that is how things are.

EDIT:
  To port to 64 Bits from 32 bit types  is what I was referring to..

« Last Edit: February 24, 2018, 03:51:49 pm by jamie »
The only true wisdom is knowing you know nothing

Renat.Su

  • Full Member
  • ***
  • Posts: 230
    • Renat.Su
Re: Reading variant from a stream
« Reply #12 on: February 24, 2018, 03:59:08 pm »
jamie, You have accurately described the problem and the way to solve it

Renat.Su

  • Full Member
  • ***
  • Posts: 230
    • Renat.Su
Re: Reading variant from a stream
« Reply #13 on: February 24, 2018, 04:01:59 pm »
But everything was even easier
 :o 
Code: Pascal  [Select][+][-]
  1. ASomeStream.ReadBuffer(AVariantVar, {SizeOf(AVariantVar)}16);
and no problems :D
So stupid ... Apparently, only the first 16 bytes are significant?

rvk

  • Hero Member
  • *****
  • Posts: 6112
Re: Reading variant from a stream
« Reply #14 on: February 24, 2018, 04:03:46 pm »
So stupid ... Apparently, only the first 16 bytes are significant?
That's what ASerge already pointed out in his first post.

 

TinyPortal © 2005-2018