Recent

Author Topic: BitHelpers - bit manipulation for standard pascal types  (Read 19883 times)

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #15 on: June 25, 2021, 01:14:48 am »
Looks very good.
Thanks. I'm glad you like it.  :D

Wondering if this should become a FPC standard...
Original idea was that once BitHelpers is feature complete and well tested, it becomes a FPC package. Maybe that time has come. We'll see how that goes...
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #16 on: June 25, 2021, 09:46:23 am »
Note there is overlap with the standard Bithelpers that come with FPC3.2.0 and up, as you know. Maybe we should look at integration.
Specialize a type, not a var.

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #17 on: June 25, 2021, 12:15:57 pm »
Note there is overlap with the standard Bithelpers that come with FPC3.2.0 and up, as you know. Maybe we should look at integration.
Nothing was mentioned about it at https://wiki.freepascal.org/FPC_New_Features_3.2.0 so I thought it was still in trunk only. Therefore I have compared 3.0 and 3.2 sources, and I see that standard type helpers have been extended for bit access with SetBit(), ClearBit(), ToggleBit() and TestBit(). I have a feeling that if we add all methods from my bithelpers to standard types then it would be overcrowded. I don't mind doing it, but although bit manipulation is very important - out of embedded world it is actually not used that often to justify it's place in standard type default helpers, so I would prefer to see it in a separate FPC package and just add bithelpers unit to uses code section when needed. What do you think?

In latest update I have also moved away from standard SHL based bit getting/setting to precalculated BitValueLookupTable[] for speed optimization, but did not make any benchmarks yet to see if there was actual benefit (SHL should be very fast in modern processors). Patches should be made only after benchmarking decision.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #18 on: June 25, 2021, 02:35:20 pm »
I have a feeling that if we add all methods from my bithelpers to standard types then it would be overcrowded. I don't mind doing it, but although bit manipulation is very important - out of embedded world it is actually not used that often to justify it's place in standard type default helpers, so I would prefer to see it in a separate FPC package and just add bithelpers unit to uses code section when needed. What do you think?
Well, several points:
- I wrote that code with some great feedback and tests by Bart, but I should probably also write a request for documentation (and a proposal documentation to go with it)
- Apart from embedded, it is is useful in several ways, especially Database application, e.g. SQL is based on bit tables and it is highly useful for AI as well. (schuler likes your code not for nothing)
- I wrote it initially because I needed it for electronics on Raspberry Pi and the likes.
- I still think we should avoid overlaps, but lets discuss just the candidates that really warrant inclusion in my four.
Quote
In latest update I have also moved away from standard SHL based bit getting/setting to precalculated BitValueLookupTable[] for speed optimization, but did not make any benchmarks yet to see if there was actual benefit (SHL should be very fast in modern processors). Patches should be made only after benchmarking decision.
I agree. Note I have some updates for implemention of ROL/ROR SAR etc, which is also useful.

About speed: that is indeed highly CPU dependent. Lets come up with some tests.

I really appreciate your contribution! It is a nice library.
« Last Edit: June 25, 2021, 02:37:04 pm by Thaddy »
Specialize a type, not a var.

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #19 on: June 25, 2021, 03:41:33 pm »
Apart from embedded, it is useful in several ways, especially Database application, e.g. SQL is based on bit tables and it is highly useful for AI as well.
So you prefer adding to syshelp*.inc instead of adding a separate package? If other devs also support that idea then I could do that (if there is a devs resistance to that idea then I would more prefer rejection before then after doing all the work).

- I still think we should avoid overlaps, but lets discuss just the candidates that really warrant inclusion in my four.
What overlaps? I do not have your ToggleBit() and ClearBit() at all, and instead of your TestBit() and SetBit() I use array like access property Bit[] which I like very much and want to keep. Maybe I can change names of that property's getter and setter, since that is the only place where I see an overlap (or to be more precise a method name conflict). If preferred, besides renaming I might also hide those methods...

I really appreciate your contribution! It is a nice library.
You're very kind. Thanks!
« Last Edit: June 25, 2021, 03:53:40 pm by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #20 on: July 12, 2021, 01:04:55 pm »
I have run some simple benchmarks (maybe too simple, but anyway) on x86 Win64, and shift based bit setting was faster 3% if variable was qword, and 10% if variable was word. So I will give up table based bit access and avoid spending 64 qwords for a lookup table. However inlining brings significant speed improvement in loops so I will leave it on.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #21 on: January 19, 2022, 11:43:01 am »
I am so glad to announce that BitHelpers are now part of trunk FPC RTL. Type helpers for direct access to bit, nibble, byte, word and dword parts in ordinal types are now part of sysutils unit, and customizable boolean, binary and hexadecimal data localized string representation is now part of syshelpers unit. Syntax has been slightly changed to be more in line with FPC (using plurals like SomeOrdType.Bits[3] instead of old SomeOrdType.Bit[3] and similar).

More details here:
https://gitlab.com/freepascal.org/fpc/source/-/issues/39268
https://gitlab.com/freepascal.org/fpc/documentation/-/issues/39267

From now on, all further development will be in FPC trunk. Bitbucket repo will continue to exist only for historical reasons and old code.

This is the output from new console demo:
Code: [Select]
True.ToString                                          = True
True.ToTrueFalseString(scfUpperCase)                   = TRUE
True.ToString('OnState', 'OffState')                   = OnState
True.ToString('Running', 'Stopped', scfUpperCase)      = RUNNING
True.ToString('Ради', 'Не ради', scfUpperCase)         = Ради (console uppers ASCII only, gui uppers unicode as well)
True.ToString(MyBitFormatSettings)                     = MyTrue
MyNativeInt                                            = -10000
MyNativeInt.ToBinString                                = 11111111111111111101100011110000
MyNativeInt.Bits[MyNativeInt.MaxBit].ToTrueFalseString = True
MyDword.ToBinString          = 00000000000111000000001001001010
MyDword.ToBinString(true)    = 00000000000111000000001001001010
MyDword.ToBinString(false)   = 111000000001001001010
MyDword.Bytes[0].ToBinString = 0100_1010   (BinNibbleSeparator = "_")
MyDword.HighestSetBitPos     = 20
MyDword.LowestSetBitPos      = 1
MyDword.SetBitsCount         = 7
MyDword.ToHexString          = 001C024A
MyDword.ToHexString(7)       = 01C024A
MyDword.ToHexString(false)   = 1C024A

DefaultBitFormatSettings:
FALSE : False, False, 0, Off
TRUE  : True, True, 1, On
MyDword.Bits[5].ToTrueFalseString = False (BitTrueString = True)
MyDword.Bits[5].ToString          = False
TryStrToBool('trUE', MyBool)      = TRUE
BoolToStr(true, true)             = True, TrueBoolStrs[0] = True

CustomBitFormatSettings:
FALSE : Неистина, Неистина, O, Искључено
TRUE  : Истина, Истина, X, Укључено
MyDword.Bits[5].ToTrueFalseString = Неистина (BitTrueString = Истина)
MyDword.Bits[5].ToString          = Неистина
TryStrToBool('Истина', MyBool)    = Истина, (MyBool = TRUE)
TryStrToBool('ИСТИна', MyBool)    = Неистина (console uppers ASCII only, gui uppers unicode as well)
BoolToStr(true, true)             = Истина, TrueBoolStrs[0] = Истина
True.ToString                     = Истина

DefaultBinFormatSettings:
MyQword.ToBinString              = 0000000000000000000010101011110011011110111111111111111111111111
MyQword.ToBinString(false)       =                     10101011110011011110111111111111111111111111
MyQword.ToBinString(50)          =               00000010101011110011011110111111111111111111111111

CustomBinFormatSettings:
MyQword.ToBinString       = OOOOOOOO.OOOOOOOO-OOOOXOXO.XOXXXXOO_XXOXXXXO.XXXXXXXX-XXXXXXXX.XXXXXXXX
MyQword.ToBinString(false)       =                XOXO.XOXXXXOO_XXOXXXXO.XXXXXXXX-XXXXXXXX.XXXXXXXX
MyQword.ToBinString(50)          =         OO-OOOOXOXO.XOXXXXOO_XXOXXXXO.XXXXXXXX-XXXXXXXX.XXXXXXXX

DefaultHexFormatSettings:
MyQword.ToHexString                             = 00000ABCDEFFFFFF
MyQword.ToHexString(3)                          =      ABCDEFFFFFF
MyQword.ToHexString(13)                         =    00ABCDEFFFFFF
MyQword.ToHexString(true)                       = 00000ABCDEFFFFFF
MyQword.ToHexString(false)                      =      ABCDEFFFFFF

CustomHexFormatSettings1:
MyQword.ToHexString                             = [$0.0^0.0_0.A^B.C--D.E^F.F_F.F^F.F]
MyQword.ToHexString(true)                       = [$0.0^0.0_0.A^B.C--D.E^F.F_F.F^F.F]
MyQword.ToHexString(false)                      =           [$A^B.C--D.E^F.F_F.F^F.F]

CustomHexFormatSettings2:
MyQword.ToHexString(MyHexFormatSettings)        = $0000:0ABC:DEFF:FFFF
MyQword.ToHexString(MyHexFormatSettings, false) =       $ABC:DEFF:FFFF
MyQword.ToHexString(MyHexFormatSettings, 13)    =    $0:0ABC:DEFF:FFFF

Have fun!  :D 8-) :D
« Last Edit: January 19, 2022, 09:17:58 pm by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

mischi

  • Full Member
  • ***
  • Posts: 170
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #22 on: January 19, 2022, 11:48:08 am »
Nice. Great work.👍

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #23 on: January 19, 2022, 12:49:51 pm »
I am so glad to announce that BitHelpers are now part of trunk FPC RTL.
I am also glad that you improved my original code (that was already accepted). Thanks a lot. We had a little communication and everything you added is very useful.
Specialize a type, not a var.

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #24 on: January 19, 2022, 01:13:55 pm »
Nice. Great work.👍
Thanks!  ::)

I am so glad to announce that BitHelpers are now part of trunk FPC RTL.
I am also glad that you improved my original code (that was already accepted). Thanks a lot. We had a little communication and everything you added is very useful.
I'm glad you like it. Thank you for your kind words!  O:-)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #25 on: April 27, 2022, 01:12:25 pm »
I am glad to announce that documentation and console example have been added to the latest FPC trunk.

https://gitlab.com/freepascal.org/fpc/source/-/tree/main/packages/rtl-objpas/examples
https://gitlab.com/freepascal.org/fpc/documentation/-/blob/main/xml/syshelpers.xml
https://gitlab.com/freepascal.org/fpc/documentation/-/blob/main/xml/sysutils.xml

GUI example does not belong to FPC so it will remain available at https://gitlab.com/freepascal.org/fpc/source/-/issues/39541 as file attachment 20220120-3-examples.zip.

This concludes migration of bithelpers to FPC trunk. Of course, you may continue to shoot related questions here.

Enjoy!  :D
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

WayneSherman

  • Full Member
  • ***
  • Posts: 243
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #26 on: May 01, 2022, 02:42:41 am »
Thank you for your work on these and getting them merged into fpc.

The bit functions such as ClearBit, SetBit, and TestBit take bit index values, but they don't work with bit mask values.  It would be nice to add bit functions that work with masks as well since many APIs and libraries use them.  Something like this:

Code: Pascal  [Select][+][-]
  1. FLAG1 = $01;
  2. FLAG2 = $02;
  3. FLAG3 = $04;
  4. FLAG4 = $08;
  5. ...
  6.  
  7. MyFlags.ClearFlags(FLAG1 or FLAG3);
  8. MyFlags.SetFlags(FLAG2);
  9. if MyFlags.HasAnyFlags(FLAG1 or FLAG4) then...
  10. if MyFlags.HasAllFlags(FLAG1 or FLAG4) then...

File attributes on Windows are defined like this:
Code: Pascal  [Select][+][-]
  1.      FILE_ATTRIBUTE_READONLY             = $0000001;
  2.      FILE_ATTRIBUTE_HIDDEN               = $0000002;
  3.      FILE_ATTRIBUTE_SYSTEM               = $0000004;
  4.      FILE_ATTRIBUTE_DIRECTORY            = $0000010;
  5.      FILE_ATTRIBUTE_ARCHIVE              = $0000020;
  6.      FILE_ATTRIBUTE_DEVICE               = $0000040;
  7.      FILE_ATTRIBUTE_NORMAL               = $0000080;
  8.  
  9. //so you could do this
  10. if MyFileAttr.HasFlags(FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_SYSTEM) then

I recently had to do this:
Code: Pascal  [Select][+][-]
  1.   if GetConsoleMode(TextRec(Input).Handle, ConsoleMode) then begin
  2.     ConsoleMode := ConsoleMode and not ENABLE_QUICK_EDIT_MODE;
  3.     SetConsoleMode(TextRec(Input).Handle, ConsoleMode);
  4.   end;

But would have loved to do this:
Code: Pascal  [Select][+][-]
  1.   if GetConsoleMode(TextRec(Input).Handle, ConsoleMode) then
  2.       SetConsoleMode(TextRec(Input).Handle, ConsoleMode.ClearFlags(ENABLE_QUICK_EDIT_MODE));

« Last Edit: May 01, 2022, 03:00:24 am by WayneSherman »

WayneSherman

  • Full Member
  • ***
  • Posts: 243
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #27 on: May 01, 2022, 09:52:42 am »
I haven't decided if I like "Mask" or "Flag":

Code: Pascal  [Select][+][-]
  1. MyFlag.ClearMask(FLAG1);
  2. MyFlag.SetMask(FLAG3 or FLAG4);
  3. if MyFlags.TestMask(FLAG1 or FLAG3) then...
  4. if MyFlags.TestMaskAll(FLAG1 or FLAG3) then...
  5. if MyFlags.TestMaskAny(FLAG1 or FLAG3) then...

Also, we have a Clear operation, but I did not see a helper to set all bits:
Code: Pascal  [Select][+][-]
  1. MyFlag.Clear;  //not a function, operates on the variable directly
  2. MyFlag.SetAllBits;  //not a function, operates on the variable directly

440bx

  • Hero Member
  • *****
  • Posts: 3944
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #28 on: May 01, 2022, 09:56:36 am »
I haven't decided if I like "Mask" or "Flag":
I "vote" for "Mask"  because it's more descriptive than (not as generic as) "Flag" and also because it's consistent with the helpers' names.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

WayneSherman

  • Full Member
  • ***
  • Posts: 243
Re: BitHelpers - bit manipulation for standard pascal types
« Reply #29 on: May 01, 2022, 05:13:13 pm »
or maybe add the word "Bits":

Code: Pascal  [Select][+][-]
  1.     MyFlag.ClearMaskBits(FLAG1);
  2.     MyFlag.SetMaskBits(FLAG3 or FLAG4);
  3.     if MyFlags.TestMaskBits(FLAG1 or FLAG3) then...  //all bits must match for true
  4.  
  5.    //is this even needed?
  6.     if MyFlags.TestMaskAny(FLAG1 or FLAG3) then...  //any bit must match for true

Maybe this was discussed, but the way helpers look, using a dot operator, leads one to believe the variable itself gets modified.  Should a helper like this be written as a function or procedure?

Code: Pascal  [Select][+][-]
  1. MyFlag.ClearMaskBits(FLAG1);  //not a function, operate on the variable directly
  2. //versus
  3. MyFlag := MyFlag.ClearMaskBits(FLAG1);  //a function, MyFlag is not changed by the helper

For example, a record or object generally are expected to operate on their own data:
Code: Pascal  [Select][+][-]
  1. MyStrRec.SetValue('hello world'); //would be expected to change the record variable
  2. MyObject.SetValue(True); //would be expected to modify data in the object.
« Last Edit: May 01, 2022, 09:59:51 pm by WayneSherman »

 

TinyPortal © 2005-2018