Lazarus

Free Pascal => General => Topic started by: furious programming on March 14, 2021, 02:14:40 pm

Title: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: furious programming on March 14, 2021, 02:14:40 pm
Can someone tell me why the output is the way it is?

Code: Pascal  [Select][+][-]
  1. {$MODE OBJFPC}{$LONGSTRINGS ON}
  2.  
  3. type
  4.   TStatus = bitpacked record
  5.     Up: Boolean;
  6.     Down: Boolean;
  7.     Left: Boolean;
  8.     Right: Boolean;
  9.     Select: Boolean;
  10.     Start: Boolean;
  11.     B: Boolean;
  12.     A: Boolean;
  13.   end;
  14.  
  15. var
  16.   Status: TStatus;
  17.   StatusInt: UInt8 absolute Status;
  18. begin
  19.   Status.Up := True;
  20.   Status.Down := True;
  21.   Status.Left := True;
  22.   Status.Right := True;
  23.   Status.Select := True;
  24.   Status.Start := True;
  25.   Status.B := True;
  26.   Status.A := True;
  27.  
  28.   WriteLn('Status: ', BinStr(StatusInt, 8), LineEnding);
  29.  
  30.   WriteLn('Up:     ', StatusInt and &00000001 <> 0);
  31.   WriteLn('Down:   ', StatusInt and &00000010 <> 0);
  32.   WriteLn('Left:   ', StatusInt and &00000100 <> 0);
  33.   WriteLn('Right:  ', StatusInt and &00001000 <> 0);
  34.   WriteLn('Select: ', StatusInt and &00010000 <> 0);
  35.   WriteLn('Start:  ', StatusInt and &00100000 <> 0);
  36.   WriteLn('B:      ', StatusInt and &01000000 <> 0);
  37.   WriteLn('A:      ', StatusInt and &10000000 <> 0);
  38.  
  39.   ReadLn();
  40. end.

Output:

Code: Pascal  [Select][+][-]
  1. Status: 11111111
  2.  
  3. Up:     TRUE
  4. Down:   TRUE
  5. Left:   TRUE
  6. Right:  FALSE
  7. Select: FALSE
  8. Start:  FALSE
  9. B:      FALSE
  10. A:      FALSE

I feel like an idiot.
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: Jonas Maebe on March 14, 2021, 02:22:09 pm
& is octal, % is binary: https://freepascal.org/docs-html/ref/refse6.html
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: MathMan on March 14, 2021, 02:23:29 pm
First I admit that I didn't look up anything prior to this replay, but ...

... if ampersand '&' declares octal numbers this would easily explain it.

What happens if you declare with percent '%'?

Cheers,
MathMan

ah - beaten by Jonas, by a fraction
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: Jonas Maebe on March 14, 2021, 02:26:48 pm
Also keep in mind that the bit order of bitpacked records/arrays is different for big and little endian systems so if you want to be portable, only use the record/array types and not the underlying values directly.
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: furious programming on March 14, 2021, 02:49:40 pm
So I'm an idiot. Thanks for the replies…

Also keep in mind that the bit order of bitpacked records/arrays is different for big and little endian systems so if you want to be portable, only use the record/array types and not the underlying values directly.

AFAIK endianness determines the order of bytes, not bits, so using such a record is safe. I need such a packed type, because I'll have tens of thousands of such records in the stream. By packing eight logical values into one byte, the size of this data in the stream will be eight times smaller, and in my case it matters a lot.
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: Blaazen on March 14, 2021, 04:42:13 pm
Quote
AFAIK endianness determines the order of bytes, not bits, so using such a record is safe.

Until you add the ninth boolean to the record.
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: Jonas Maebe on March 14, 2021, 04:51:33 pm
So I'm an idiot. Thanks for the replies…

Also keep in mind that the bit order of bitpacked records/arrays is different for big and little endian systems so if you want to be portable, only use the record/array types and not the underlying values directly.

AFAIK endianness determines the order of bytes, not bits, so using such a record is safe.
In general this is true, but bitpacked records/arrays also use a different bit order on big/little endian systems. The reason is that otherwise accessing bitpacked data changes the bit number depending on the size of the access (1/2/4/8 byte).

From the compiler source:
Quote
  (*
    Subsetrefs are used for (bit)packed arrays and (bit)packed records stored
    in memory. They are like a regular reference, but contain an extra bit
    offset (either constant -startbit- or variable -bitindexreg-, always OS_INT)
    and a bit length (always constant).

    Bit packed values are stored differently in memory depending on whether we
    are on a big or a little endian system (compatible with at least GPC). The
    size of the basic working unit is always the smallest power-of-2 byte size
    which can contain the bit value (so 1..8 bits -> 1 byte, 9..16 bits -> 2
    bytes, 17..32 bits -> 4 bytes etc).

    On a big endian, 5-bit: values are stored like this:
      11111222 22333334 44445555 56666677 77788888
    The leftmost bit of each 5-bit value corresponds to the most significant
    bit.

    On little endian, it goes like this:
      22211111 43333322 55554444 77666665 88888777
    In this case, per byte the left-most bit is more significant than those on
    the right, but the bits in the next byte are all more significant than
    those in the previous byte (e.g., the 222 in the first byte are the low
    three bits of that value, while the 22 in the second byte are the upper
    two bits.

    Big endian, 9 bit values:
      11111111 12222222 22333333 33344444 ...

    Little endian, 9 bit values:
      11111111 22222221 33333322 44444333 ...
    This is memory representation and the 16 bit values are byteswapped.
    Similarly as in the previous case, the 2222222 string contains the lower
    bits of value 2 and the 22 string contains the upper bits. Once loaded into
    registers (two 16 bit registers in the current implementation, although a
    single 32 bit register would be possible too, in particular if 32 bit
    alignment can be guaranteed), this becomes:
      22222221 11111111 44444333 33333322 ...
      (l)ow  u     l     l    u     l   u

    The startbit/bitindex in a subsetreference always refers to
    a) on big endian: the most significant bit of the value
       (bits counted from left to right, both memory an registers)
    b) on little endian: the least significant bit when the value
       is loaded in a register (bit counted from right to left)

    Although a) results in more complex code for big endian systems, it's
    needed for compatibility both with GPC and with e.g. bitpacked arrays in
    Apple's universal interfaces which depend on these layout differences).

    Note: when changing the loadsize calculated in get_subsetref_load_info,
    make sure the appropriate alignment is guaranteed, at least in case of
    {$defined cpurequiresproperalignment}.
  *)
Title: Re: Bitpacked record of Boolean flags — problem with reading flag states
Post by: furious programming on March 14, 2021, 05:18:06 pm
Until you add the ninth boolean to the record.

I will never do this, because NES controller had only 8 buttons. 8)


In general this is true, but bitpacked records/arrays also use a different bit order on big/little endian systems.

Yes, but this only applies to bitpacked types whose size is greater than one byte, or data types whose units cannot be aligned to 8 bits. Therefore, 5-bit and 9-bit values are given in this comment and therefore the indexes of the bits in the units do not matter (therefore, the unit bit indexes are not specified).

In my case, the data packet is always 8 bits long, so endianness does not concern me.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: Jonas Maebe on March 14, 2021, 05:22:04 pm
On little endian, the first bit of the bitpacked record is at 1 shl 0.

On big endian, the first bit of the bitpacked record is at 1 shl 7.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: winni on March 14, 2021, 10:39:02 pm
On little endian, the first bit of the bitpacked record is at 1 shl 0.

On big endian, the first bit of the bitpacked record is at 1 shl 7.

Yes. Another step back.

On UCSD-machines it was always the same.
No matter if you worked on an Apple II or a PC.
I am talking about 1980.

Winni
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: Jonas Maebe on March 14, 2021, 10:54:08 pm
On little endian, the first bit of the bitpacked record is at 1 shl 0.

On big endian, the first bit of the bitpacked record is at 1 shl 7.

Yes. Another step back.

On UCSD-machines it was always the same.
No matter if you worked on an Apple II or a PC.
I am talking about 1980.

Winni
The downside of having it the same everywhere is that the bit numbering changes depending on the access size. You then always have to access everything byte by byte on either big or little endian (depending on which bit order you pick) while with the current approach (which is defined by all ABIs I know of) allows operations to be performed using a single 16/32/64 bit access when applicable.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: winni on March 15, 2021, 12:19:38 am
[
The downside of having it the same everywhere is that the bit numbering changes depending on the access size. You then always have to access everything byte by byte on either big or little endian (depending on which bit order you pick) while with the current approach (which is defined by all ABIs I know of) allows operations to be performed using a single 16/32/64 bit access when applicable.

Hi!

Already clear.

The reason was the p-machine and the ability to run the p-code-executable on any p-machine: No matter what byte-sex.  So you could run an executable compiled on an Apple II on a PC without any problems. That's the reason why Java has "stolen" the p-code.

Winni
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: furious programming on March 15, 2021, 12:32:10 am
On little endian, the first bit of the bitpacked record is at 1 shl 0.
On big endian, the first bit of the bitpacked record is at 1 shl 7.

Nevermind. Anyway, this code is supposed to work only on Windows, so I'm not going to mess too much.


This is interesting to me. I am creating a simple file format that allows me to store a recording of Tetris gameplay. The format provides for storing data in binary form, as a set of frames in uncompressed form (each frame is a key frame) and compressed (so that the file on the disk takes up as little space as possible, no key frames).

Each keyframe must contain, inter alia, controller state (eight buttons), which I want to represent as a bitpacked record with a size of 8 bits (1 byte), and the state of the game field in the form of a bitpacked array, in which each cell must contain one of the four types of cubes (empty, color A, B or C) — the playing field is 10×20 in size, which is 400 bits (50 bytes).

From what you write, it appears that if I send someone a binary file containing a gameplay recording (a recording made on my computer), to someone who has a different endianness, the program will not be able to correctly read the data from this file. I understand well?
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: PascalDragon on March 15, 2021, 01:28:12 pm
From what you write, it appears that if I send someone a binary file containing a gameplay recording (a recording made on my computer), to someone who has a different endianness, the program will not be able to correctly read the data from this file. I understand well?

Correct. You'll need to pick one bitness where you then do a bitswap upon reading the corresponding data.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: furious programming on March 15, 2021, 02:09:19 pm
Is there such a need if the software is to run exclusively on Windows?

Theoretically, only me will use this program, but I would prefer it not to crash in the future when I change my computer to a newer/different one.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: ASBzone on March 15, 2021, 10:21:19 pm
Is there such a need if the software is to run exclusively on Windows?

Theoretically, only me will use this program, but I would prefer it not to crash in the future when I change my computer to a newer/different one.

Well, since it is only you, you could always recompile it. :)

I don't believe that any Windows editions -- not even the ones for MIPS and DEC Alpha -- were ever Big Endian.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: furious programming on March 16, 2021, 09:19:00 am
Well, since it is only you, you could always recompile it. :)

This is true, but while the source code can be recompiled without a problem, there is a problem with the already generated gameplay files, because I would have to write an endianess converter for the new program to understand them.

And that's what I'd rather avoid. I am thinking for the future, because for the time being I will be using this program myself, so one day it may turn into a full-fledged product. And I would not like to have to rewrite the entire recorder code and convert the recording files. It is true that the record files will not be used for sharing (they will only be used for local software needs), but still.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: alpine on April 10, 2021, 08:15:35 pm
IMHO, You should pack them by yourself and not depend on 'packed' feature and the endianess.

Something like:
Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. type
  4.  
  5.   { TMyPacking }
  6.   TMyPacking = class
  7.   private
  8.     function GetBit(AIndex: Integer): Boolean;
  9.     procedure SetBit(AIndex: Integer; AValue: Boolean);
  10.   public
  11.     Data: array [0..512] of Byte;
  12.     property Bit[I: Integer]: Boolean read GetBit write SetBit;
  13.   end;
  14.  
  15.   TTetrisPacking = class(TMyPacking)
  16.   public
  17.     property Left: Boolean index 0 read GetBit write SetBit;
  18.     property Right: Boolean index 1 read GetBit write SetBit;
  19.     // More boolean properties as needed with appropriate index
  20.     // ...
  21.   end;
  22.  
  23. { TMyPacking }
  24.  
  25. function TMyPacking.GetBit(AIndex: Integer): Boolean;
  26. begin
  27.   Result := 0 <> (Data[AIndex div 8] or (1 shl (AIndex mod 8)));
  28. end;
  29.  
  30. procedure TMyPacking.SetBit(AIndex: Integer; AValue: Boolean);
  31. begin
  32.   if AValue then
  33.     Data[AIndex div 8] := Data[AIndex div 8] or (1 shl (AIndex mod 8)) else
  34.     Data[AIndex div 8] := Data[AIndex div 8] and not (1 shl (AIndex mod 8))
  35. end;
  36.  
  37. var
  38.   P: TTetrisPacking;
  39.  
  40. begin
  41.   P := TTetrisPacking.Create;
  42.   P.Left := True;
  43.   P.Right := False;
  44.   // Write or read the P.Data in a byte stream...
  45.   P.Free;
  46. end.
  47.  
  48.  


It can be further extended as you like.

Regards,
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: winni on April 10, 2021, 11:13:27 pm


I don't believe that any Windows editions -- not even the ones for MIPS and DEC Alpha -- were ever Big Endian.

Hi!

DEC Alpha was Big Endian. Nice machine.

Winni
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: Thaddy on April 11, 2021, 03:58:50 pm
You do not need a record in most cases.
sysutils now contains type helpers for most standard ordinal types,
Suppose you need to test 8 flags (byte) we can write this (0..7)
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}
  2. uses sysutils;
  3. var a:byte = 54;i:integer;
  4. begin
  5.   for i := 0 to 7 do
  6.     writeln(i,' ',a.testbit(i);
  7. end.
I implemented this - and more - in syshelph.inc and it is in 3.2.0 and trunk.
So choose am ordinal up to qword, optionally declare a range and no record is needed.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: Thaddy on April 11, 2021, 04:10:36 pm
To demonstrate with a range:
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}
  2. uses sysutils;
  3. type
  4.   Tmyrange = 0..5;
  5.  
  6. var a:byte = 54;i:TmyRange;
  7. begin
  8.   for i := Low(TMyRange) to high(TmyRange) do
  9.     writeln(i,' ',a.testbit(i));
  10. end.
It is also platform agnostic.

Obviously you can also set, clear and toggle the bits through the type helper.
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: Thaddy on April 11, 2021, 04:15:31 pm
It can be further extended as you like.
Regards,
See my last answers... not necessary. Comes as standard from 3.2.0.
Just include sysutils in the uses clause. (up to qword range, that's a lot of state bits) Limit a range that is less than an ordinal by definng it. See above example.
Regards,
Thaddy
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: alpine on April 11, 2021, 04:25:39 pm
It can be further extended as you like.
Regards,
See my last answers... not necessary. Comes as standard.

Yup, very nice!

As far as I understand, the problem is how to pack/save/restore a large bit array but still use a meaningful symbolic names. So my suggestion for using getters/setters to pack them in a byte array and not to worry about endianess.

No doubt, It can be achieved in different ways.

Regards,
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: PascalDragon on April 11, 2021, 09:13:27 pm


I don't believe that any Windows editions -- not even the ones for MIPS and DEC Alpha -- were ever Big Endian.

Hi!

DEC Alpha was Big Endian. Nice machine.

No, DEC Alpha could run as both big and little endian and Windows always used little endian on processors that support both (MIPS, DEC Alpha, PowerPC, ARM, Aarch64).
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: ASBzone on April 15, 2021, 05:38:59 pm

I don't believe that any Windows editions -- not even the ones for MIPS and DEC Alpha -- were ever Big Endian.

Hi!

DEC Alpha was Big Endian. Nice machine.

Winni

It is my understanding that DEC Alpha is bi-endian, and that it ran in little endian mode for Windows.

https://en.wikipedia.org/wiki/DEC_Alpha
Title: Re: [SOLVED] Bitpacked record of Boolean flags — problem with reading flag states
Post by: ASBzone on April 15, 2021, 05:41:57 pm
No, DEC Alpha could run as both big and little endian and Windows always used little endian on processors that support both (MIPS, DEC Alpha, PowerPC, ARM, Aarch64).


LOL I ended up replying to a message I already had open, and totally missed your answer.  Would have saved me some time.
TinyPortal © 2005-2018