Lazarus

Announcements => Third party => Topic started by: correa.elias on October 11, 2018, 04:39:16 pm

Title: ToStr - Convert any value to String using RTTI
Post by: correa.elias on October 11, 2018, 04:39:16 pm
Automatically converts any value to a string representation, including records and arrays, in the same spirit of 'print' functions found in scripting languages. Its usefull for logging and debugging.

https://github.com/correaelias/TypeUtils
Title: Re: ToStr - Convert any value to String using RTTI
Post by: avra on October 11, 2018, 07:14:22 pm
Thank you, very useful !    :)
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Trenatos on October 11, 2018, 07:25:02 pm
I haven't tried it out, but if it works well I'd love to see it included with FPC core.

Debuggers aren't always useable (I'm looking at you, OSX) and the ability to just dump some data to a terminal/textfield is often all that's needed.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: correa.elias on October 11, 2018, 07:35:31 pm
Thanks! Im planning better examples to demonstrate the power of this function.

Code: Pascal  [Select][+][-]
  1. program Test;
  2.  
  3. uses TypeUtils, SysUtils;
  4.  
  5. type
  6.   TEmployee = record
  7.     FName: ShortString;
  8.     FId: LongInt;
  9.     FSalary: Currency;
  10.   end;
  11.  
  12. var
  13.   Employees: array of TEmployee;
  14.   I: LongInt;
  15. begin
  16.   SetLength(Employees, 10);
  17.  
  18.   for I := 0 to High(Employees) do
  19.   begin
  20.     with Employees[I] do
  21.     begin
  22.       FName := 'John Doe ' + I.ToString;
  23.       FId := I;
  24.       FSalary := 2000.25 * I;
  25.     end;
  26.   end;
  27.  
  28.   WriteLn(ToStr(Employees, TypeInfo(Employees)));
  29. end.
  30.  

outputs:
[('John Doe 0', 0, 0), ('John Doe 1', 1, 2000.25), ('John Doe 2', 2, 4000.5), ('John Doe 3', 3, 6000.75), ('John Doe 4', 4, 8001), ('John Doe 5', 5, 10001.25), ('John Doe 6', 6, 12001.5), ('John Doe 7', 7, 14001.75), ('John Doe 8', 8, 16002), ('John Doe 9', 9, 18002.25)]
Title: Re: ToStr - Convert any value to String using RTTI
Post by: zeljko on October 11, 2018, 09:53:16 pm
Nice job :)
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 08:30:20 am
@correa.elias

I have reviewed the code and it looks very nice.
I have a couple of remarks, though:

1. ManagedFldCount is deprecated (at least in 3.2.0 and trunk).
    You can replace it with TotalFieldCount. (The compiler actually gives two suggestions. I chose this one)
2. I prefer {$mode Delphi}{$coperators on} over mode objfpc (see below code why!)
3. You can get rid of the typeinfo parameter and introduce type safety like so:
Code: Pascal  [Select][+][-]
  1. {$mode delphi}{$coperators on}
  2.  
  3. function ToStr(var AX; ATypeInfo: PTypeInfo): AnsiString;overload;
  4. function Tostr<T>(var Value:T):AnsiString;overload;  // <----
  5.  
  6. implementation
  7.  
  8. function Tostr<T>(var Value:T):AnsiString;overload;
  9. begin
  10.   Result :=Tostr(value,typeinfo(t));
  11. end;

My initial tests show no regressions with these minor changes. 

(objfpc mode generics look silly and won't allow this, delphi mode does allow it)
(To me objfpc mode for generics are deprecated)



Title: Re: ToStr - Convert any value to String using RTTI
Post by: andresayang on October 12, 2018, 09:48:34 am
Thanks
Title: Re: ToStr - Convert any value to String using RTTI
Post by: avra on October 12, 2018, 10:19:06 am
I haven't tried it out, but if it works well I'd love to see it included with FPC core.
+1  :)

3. You can get rid of the typeinfo parameter and introduce type safety
Very nice, thanks! I would really appreciate this change.   8)
Title: Re: ToStr - Convert any value to String using RTTI
Post by: wp on October 12, 2018, 11:20:32 am
I haven't tried it out, but if it works well I'd love to see it included with FPC core.
I don't want to disencourage you, but in my eyes, it is not flexible enough for inclusion in fcl: What if I want floats in scientific notation, integers in hex, a different field separator?
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 11:59:23 am
I don't want to disencourage you, but in my eyes, it is not flexible enough for inclusion in fcl: What if I want floats in scientific notation, integers in hex, a different field separator?

His stated purpose (in his other thread) was something similar to what Python offers: a one stop shop to print any content in a readable way.
I encouraged him to write it  :P O:-) Which he did!
That means inherently taking over some limitations.
Within that context he did a pretty good job! Within that context it is also valuable, not to the FCL but to the rtl: in sysutils.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: marcov on October 12, 2018, 12:05:58 pm
I haven't tried it out, but if it works well I'd love to see it included with FPC core.
I don't want to disencourage you, but in my eyes, it is not flexible enough for inclusion in fcl: What if I want floats in scientific notation, integers in hex, a different field separator?

That requires custom attributes.  Then you can annotate that you want a color to be printed in hex etc, and the dump routine can query it.

Having both a fancy and less fancy is nice though, the fancier, the slower probably, and sometimes you just want to dump for debugging/logging purposes. This is fairly universal.

If you use this in generation of a wire protocol or file format, you want more control, hence the fancy version.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: avra on October 12, 2018, 12:12:38 pm
I haven't tried it out, but if it works well I'd love to see it included with FPC core.
I don't want to disencourage you, but in my eyes, it is not flexible enough for inclusion in fcl: What if I want floats in scientific notation, integers in hex, a different field separator?
One of the ways to enable customization for users is like we already have with DefaultFormatSettings record which is of TFormatSettings type. Anyone wanting customization could populate DefaultToStrSettings record with his own likings and everyone is happy. We could even allow optional parameter for settings record parameter. For scientific notation of floats, I think that ToStr should already use DefaultFormatSettings from FPC.

I would still prefer Str() function name instead of ToStr() - and yes, I am aware there is already a Str() procedure.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 12:20:06 pm
The function is actually similar to one of the many ToJson implementations. That has also restrictions.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: AlexTP on October 12, 2018, 12:44:38 pm
Added new page http://wiki.freepascal.org/ToStr
Title: Re: ToStr - Convert any value to String using RTTI
Post by: mr-highball on October 12, 2018, 05:29:28 pm
@correa.elias

I have reviewed the code and it looks very nice.
I have a couple of remarks, though:

1. ManagedFldCount is deprecated (at least in 3.2.0 and trunk).
    You can replace it with TotalFieldCount. (The compiler actually gives two suggestions. I chose this one)
2. I prefer {$mode Delphi}{$coperators on} over mode objfpc (see below code why!)
3. You can get rid of the typeinfo parameter and introduce type safety like so:
Code: Pascal  [Select][+][-]
  1. {$mode delphi}{$coperators on}
  2.  
  3. function ToStr(var AX; ATypeInfo: PTypeInfo): AnsiString;overload;
  4. function Tostr<T>(var Value:T):AnsiString;overload;  // <----
  5.  
  6. implementation
  7.  
  8. function Tostr<T>(var Value:T):AnsiString;overload;
  9. begin
  10.   Result :=Tostr(value,typeinfo(t));
  11. end;

My initial tests show no regressions with these minor changes. 

(objfpc mode generics look silly and won't allow this, delphi mode does allow it)
(To me objfpc mode for generics are deprecated)

thanks this is nice, also I would second the templated version
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 06:08:27 pm
Note that both versions should be in:
In case of a var - so not a type - the original works magic.
If you know the type my addition is helpful.
For example a var that is an array.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: garlar27 on October 12, 2018, 07:39:22 pm
I haven't tried it out, but if it works well I'd love to see it included with FPC core.
I don't want to disencourage you, but in my eyes, it is not flexible enough for inclusion in fcl: What if I want floats in scientific notation, integers in hex, a different field separator?
I agree with the formatting thing, and its very easy to change it :
Code: Pascal  [Select][+][-]
  1.    function ToStr(var AX; ATypeInfo: PTypeInfo; FormatSettings: TFormatSettings): AnsiString;
  2.    type
  3.      TArray = array of Byte;
  4.    var
  5.      I: LongInt;
  6.      TypeData: PTypeData;
  7.      //FormatSettings: TFormatSettings;
  8.    begin
  9.      //FillByte(FormatSettings, SizeOf(TFormatSettings), 0);
  10.      //FormatSettings.DecimalSeparator := '.';
  11.      TypeData := GetTypeData(ATypeInfo);
  12.      Result := '"'+ATypeInfo^.Name + '" : '; // I added something here...
  13.      case ATypeInfo^.Kind of
  14.        tkChar: Result += '''' + Char(AX) + '''';
  15.        tkWChar: Result += '''' + AnsiString(WChar(AX)) + '''';
  16.        tkBool: Result += BoolToStr(Boolean(AX), True);
  17.        tkInteger:
  18.        begin
  19.          case TypeData^.OrdType of
  20.            otSByte: Result += IntToStr(ShortInt(AX));
  21.            otUByte: Result += IntToStr(Byte(AX));
  22.            otSWord: Result += IntToStr(SmallInt(AX));
  23.            otUWord: Result += IntToStr(Word(AX));
  24.            otSLong: Result += IntToStr(LongInt(AX));
  25.            otULong: Result += IntToStr(LongWord(AX));
  26.          end;
  27.        end;
  28.        tkInt64: Result += IntToStr(Int64(AX));
  29.        tkQWord: Result += IntToStr(QWord(AX));
  30.        tkFloat:
  31.        begin
  32.          case TypeData^.FloatType of
  33.            ftSingle: Result += FormatFloat('0.###', Single(AX), FormatSettings);
  34.            ftDouble: Result += FormatFloat('0.###', Double(AX), FormatSettings);
  35.            ftExtended: Result += FormatFloat('0.###', Extended(AX), FormatSettings);
  36.            ftComp: Result += FormatFloat('0.###', Comp(AX), FormatSettings);
  37.            ftCurr: Result += FormatFloat('0.###', Currency(AX), FormatSettings);
  38.          end;
  39.        end;
  40.        tkSString: Result += '''' + ShortString(AX) + '''';
  41.        tkAString: Result += '''' + AnsiString(AX) + '''';
  42.        tkWString: Result += '''' + AnsiString(WideString(AX)) + '''';
  43.        tkUString: Result += '''' + AnsiString(UnicodeString(AX)) + '''';
  44.        tkRecord:
  45.        begin
  46.          Result += '(';
  47.          for I := 0 to TypeData^.ManagedFldCount - 1 do
  48.          begin
  49.            if I > 0 then Result += ', ';
  50.            with PManagedField(PByte(@TypeData^.ManagedFldCount) + SizeOf(TypeData^.ManagedFldCount) + (I * SizeOf(TManagedField)))^ do
  51.              Result += ToStr((PByte(@AX) + FldOffset)^, TypeRef);
  52.          end;
  53.          Result += ')';
  54.        end;
  55.        tkArray:
  56.        begin
  57.          Result += '[';
  58.          for I := 0 to TypeData^.ArrayData.ElCount - 1 do
  59.          begin
  60.            if I > 0 then Result += ', ';
  61.            Result += ToStr((PByte(@AX) + (I * (TypeData^.ArrayData.Size div TypeData^.ArrayData.ElCount)))^, TypeData^.ArrayData.ElType);
  62.          end;
  63.          Result += ']';
  64.        end;
  65.        tkDynArray:
  66.        begin
  67.          Result += '[';
  68.          for I := 0 to Length(TArray(AX)) - 1 do
  69.          begin
  70.            if I > 0 then Result += ', ';
  71.            Result += ToStr((PByte(@TArray(AX)[0]) + (I * TypeData^.ElSize))^, TypeData^.ElType2);
  72.          end;
  73.          Result += ']';
  74.        end;
  75.        tkClass: Result += TObject(AX).ToString;
  76.        tkEnumeration: Result += GetEnumName(ATypeInfo, Integer(AX));
  77.        tkSet: Result += SetToString(ATypeInfo, Integer(AX), True).Replace(',', ', ');
  78.        tkVariant: Result += VarToStr(Variant(AX)); //TODO: Write VariantToString for consistency (for string and float point numbers)
  79.        else Result += ATypeInfo^.Name + '@' + HexStr(@AX);
  80.      end;
  81.    end;
  82.    function ToStr(var AX; ATypeInfo: PTypeInfo): AnsiString;
  83.    begin
  84.      Result := ToStr(AX, ATypeInfo, DefaultFormatSettings);
  85.    end;
  86.  
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 08:33:50 pm
Close and almost a cigar, but why not:
Code: Pascal  [Select][+][-]
  1.   function ToStr(var AX; ATypeInfo: PTypeInfo; FormatSettings: TFormatSettings = defaultformatsettings): AnsiString;
You can subsequently do the same with my template addition suggestion...

Good suggestion! 8)

Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 09:06:42 pm
@correa.elias et. al.
This is a very enjoyable subject. I hope this leads to an inclusion in the core sources.
Maybe some shavings here and there, but really a good effort and really good contributions.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: correa.elias on October 12, 2018, 09:42:56 pm
Quote
I have reviewed the code and it looks very nice.
I have a couple of remarks, though:

1. ManagedFldCount is deprecated (at least in 3.2.0 and trunk).
    You can replace it with TotalFieldCount. (The compiler actually gives two suggestions. I chose this one)
2. I prefer {$mode Delphi}{$coperators on} over mode objfpc (see below code why!)
3. You can get rid of the typeinfo parameter and introduce type safety like so:
Code: Pascal  [Select]

    {$mode delphi}{$coperators on}
     
    function ToStr(var AX; ATypeInfo: PTypeInfo): AnsiString;overload;
    function Tostr<T>(var Value:T):AnsiString;overload;  // <----
     
    implementation
     
    function Tostr<T>(var Value:T):AnsiString;overload;
    begin
      Result :=Tostr(value,typeinfo(t));
    end;


My initial tests show no regressions with these minor changes. 
I will include the generic version, its fantastic! (as long as it compiles for people using the unit in objfpc mode too).
About the name deprecation, i am using 3.0.4 so i cant change it for now.

Quote
I don't want to disencourage you, but in my eyes, it is not flexible enough for inclusion in fcl: What if I want floats in scientific notation, integers in hex, a different field separator?
As suggested by marcov, avra, thaddy and garlar27 its easy to remedy by adding a FormatSettings parameter with a default value for the lazy :D. I will implement this: A FormatSettings record that includes float point, integer, boolean ect formatting and custom delimiters for array and record(currently '[]' and '()'), and the delimiter itself (currently a coma).

Quote
Within that context he did a pretty good job! Within that context it is also valuable, not to the FCL but to the rtl: in sysutils.
Theres other functions that uses typeinfo for string conversion like SetToString and GetEnumName, both in typinfo unit (i mean, its another place where ToStr could reside if its included in core).

Quote
I would still prefer Str() function name instead of ToStr() - and yes, I am aware there is already a Str() procedure.
Its a good name too, and could be used as an overload. I choose 'ToStr' because conversion functions uses it: IntToStr, FloatToStr and BoolToStr. It could be called VarToStr too...

Quote
Added new page http://wiki.freepascal.org/ToStr
Thanks alextp!

Quote
I'd love to see it included with FPC core
It would be great!
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 12, 2018, 10:26:25 pm
I will include the generic version, its fantastic! (as long as it compiles for people using the unit in objfpc mode too).
About the name deprecation, i am using 3.0.4 so i cant change it for now.
Yes, that works..... I understand your problem with FPC 3.0.4. just make a note ( or an ifdef)
[edit]
And var should be const...

Best I have seen in a long time apart from the core developers...- and Maciej.. O:-) - Well done...
There's a big difference between seeing good code and try to improve on it and seeing bad code you need to fix.
I think we all agree...
This is one of those occasions from my point of view.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: garlar27 on October 12, 2018, 11:10:53 pm
I would still prefer Str() function name instead of ToStr() - and yes, I am aware there is already a Str() procedure.
I would name it "xToStr" or "AnythingToStr" instead.

[EDIT]
By the way. Is it possible to get the record's field's name?
Title: Re: ToStr - Convert any value to String using RTTI
Post by: ASerge on October 12, 2018, 11:22:30 pm
Some optimizations (so the file with a different name). Longer source code but shorter generated (about 30% on Win64).
1. Changed var to const.
2. Added simple validation of invalid input parameters.
3. Unnecessary code call is excluded. For example, TFormatSettings is only needed for floating types, or calling GetTypeData only for some cases.
4. Removed hints in clear places.
5. Repeated code (enumeration of array or record elements) is extracted to a separate procedure.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: garlar27 on October 13, 2018, 12:04:53 am
Some optimizations (so the file with a different name). Longer source code but shorter generated (about 30% on Win64).
1. Changed var to const.
2. Added simple validation of invalid input parameters.
3. Unnecessary code call is excluded. For example, TFormatSettings is only needed for floating types, or calling GetTypeData only for some cases.
4. Removed hints in clear places.
5. Repeated code (enumeration of array or record elements) is extracted to a separate procedure.

WOW !!!
That's awesome!!!
Title: Re: ToStr - Convert any value to String using RTTI
Post by: JernejL on October 13, 2018, 09:42:46 am
If there was ability to get TypeInfo for each array of const or just any parameter of called function, this feature would be much nicier to use:
You could just do print([some_variable])
 
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 13, 2018, 12:17:17 pm
If there was ability to get TypeInfo for each array of const or just any parameter of called function, this feature would be much nicier to use:
You could just do print([some_variable])
Well, that's not possible. That's why I wrote the generics addition. Tostr<SomeType>(SomeVariable);  // strongly typed.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: ASerge on October 13, 2018, 05:47:48 pm
That's why I wrote the generics addition. Tostr<SomeType>(SomeVariable);  // strongly typed.
I think generic is not good here.
Lets see small project:
Code: Pascal  [Select][+][-]
  1. {$MODE OBJFPC}
  2. uses TypInfo;
  3.  
  4. function ToStr2(const AValue; ATypeInfo: PTypeInfo): string;
  5. begin
  6.   Result := '';
  7. end;
  8.  
  9. generic function ToStr<T>(const AValue: T): string; overload;
  10. begin
  11.   Result := ToStr2(AValue, TypeInfo(T));
  12. end;
  13.  
  14. var
  15.   i: Integer = 1;
  16. begin
  17.   ToStr2(i, TypeInfo(i));
  18.   specialize ToStr<Byte>(i);
  19. end.
1. It's compiled with trunk (FPC 3.1.1), but not in current 3.0.4.
2. When trying to rename ToStr2 to ToStr, a compiler error occurs (I know in $mode delphi it's OK).
3. The syntax for generic looks no easier than for a normal function (line 17 and 18).
4. It's not very strict, note that the types of variable and specialization are different in line 18.
5. In a specialization, you always need to write a type (in a normal function, only the variable name) and if the variable type needs to be changed, you need to correct all calls or errors will occur as a point above.
6. With generics the compiler will generate many functions.

With $mode delphi only point 2 will be resolved.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 13, 2018, 06:53:28 pm
No in mode delphi you don't ruin your keyboard. I am still curious why people use objfpc mode for generics if even the devs don't. It is not a religion but a relict.
Otherwise you did a good job refactoring the code, btw.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: PascalDragon on October 15, 2018, 01:39:25 pm
No in mode delphi you don't ruin your keyboard. I am still curious why people use objfpc mode for generics if even the devs don't. It is not a religion but a relict.
Otherwise you did a good job refactoring the code, btw.
Hellooho? I only use mode ObjFPC. Mode Delphi is only used for testing, cause aside from bugs (like the one with the overload here; did anyone report that already?) mode ObjFPC's generics are superior to the mode Delphi ones as one can use complex inline specializations that currently aren't possible in mode Delphi.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Xor-el on October 15, 2018, 02:05:44 pm
Well, while some specializations/constructs in mode ObjFPC are far superior to mode Delphi, trust me, mode Delphi is far more developer friendly for majority of Pascal developers.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: Thaddy on October 15, 2018, 02:31:47 pm
Hellooho? I only use mode ObjFPC. Mode Delphi is only used for testing, cause aside from bugs (like the one with the overload here; did anyone report that already?) mode ObjFPC's generics are superior to the mode Delphi ones as one can use complex inline specializations that currently aren't possible in mode Delphi.
If that's indeed the case it would be a reason to me to revisit them if inlining is an issue, which it is sometime. Why is that not in {$mode delphi}, just curious? (I noticed it, but assumed objfpc suffered the same)
Otherwise I always use {$mode Delphi} because it is more concise and I maintain Delphi code too.
Title: Re: ToStr - Convert any value to String using RTTI
Post by: correa.elias on October 15, 2018, 09:42:04 pm
Hello.
I Improved the code based in Thaddy and ASerge(Awesome insights in your attached file) suggestions:

-Replaced 'var' by 'const'
-Moved 'FormatSettings' initialization to 'tkFloat' case.
-Removed useless string concatenations.
-Removed the need for the 'TypeData' variable.
-Added ifdef to handle a name change in trunk (ManagedFldCount to TotalFieldCount).

Tried to add the generic version but it dont compiled in 3.0.4 (i am not a trunk user for now).

I am thinking about the inclusion of custom formatting but there's just too many options:
- Integers can be binary, octal or hexadecimal hexadecimal...
- Floats already have formatsettings...
- Booleans can be 0 or 1 ou true or false...
- Custom delimitters
My main inspiration to write this was 'console.log' in javascript or python 'print' and there's no way to format the output in this functions (as they are used for logging and debugging, not serialization). I think that cases where this type of fine tuning is needed you cant escape but write your own ToString in a TypeHelper or advanced record or whatever...

https://github.com/correaelias/TypeUtils
TinyPortal © 2005-2018