TFPExpressionParser ...and jargon about strToFloat etc.

[ArtLogi]

Hello

With expression parser is there a way to extract all non-defined (not number, not in build-in categories, not programmer specified "user defined function") strings from expression to array (..or some other list style data structure).
Example if user gives a string to edit1.text as 'sin(x)+cos(y)+z' is there a subprogram (or OOP equivalent) which then returns a list of sometype [x, y, z].

Just figured it might be better to ask, before I do some string function/procedure by myself and later do find out that re-inventing the wheel is not so funny after all.


PS. It seems that something has changed as wiki is saying: "Floating point constants in expressions must have a point as decimal separator, not a comma" for my PC+OS (win) the DecimalSeparator (which now seems to be deprecated  ) for me is comma (,) and when StrToFloat(edit1.text) where edit1.text=0.5, the program raises error as "not float" or equivalent. However with 0,5 it does seem to work and FParser.Evaluate.ResFloat return correct answer.

Also no errors with FParser.Identifiers.AddFloatVariable('x', 0.5);, but the result is given with comma.

[marcov]

Symbolic supports it, see its examples. packages/symbolic/examples/evaltest.pas.  The  SymVars:=Expr.SymbolicValueNames;  line  gets the list with symbols detected in the expression:

Code: Pascal  [Select][+][-]

1. {
2.     $ id:                                                       $
3.     Copyright (c) 2000 by Marco van de Voort (marco@freepascal.org)
4.     member of the Free Pascal development team
5.  
6.     See the file COPYING.FPC, included in this distribution,
7.     for details about the copyright. (LGPL)
8.  
9.     Most basic test for TEvaluator class.
10.  
11.     This program is distributed in the hope that it will be useful,
12.     but WITHOUT ANY WARRANTY; without even the implied warranty of
13.     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
14.  
15.  **********************************************************************
16. }
17.  
18. program EvalTest;
19.  
20. {$AppType Console}
21.  
22. Uses Classes,Symbolic, SysUtils;
23.  
24. VAR Expr    : TExpression;
25.     SymVars : TStringList;
26.     I       : Longint;
27.     VarName : TStringList;
28.     Eval    : TEvaluator;
29.     Vars    : Array[0..1] OF ArbFloat;
30.  
31. begin
32.  {Lets create in a nice equation. Totally nonsense. Don't try to
33.                  make sense of it}
34.  
35.  Expr:=TExpression.Create('pi*sin(x-x0)+x^(t-1)+exp(x*t)+5');
36.  Writeln('Expression after parsing :',Expr.InfixExpr);
37.  
38.  {Hmm. But the user could have typed that in. Let's check if he used
39.   symbolic values}
40.  
41.  SymVars:=Expr.SymbolicValueNames;
42.  
43.  If SymVars.Count>0 Then
44.    For I:=0 TO SymVars.Count-1 DO
45.      Writeln(I:5,' ',Symvars[I]);
46.  
47.  {Assume the user selected X and T from above stringlist as our variables}
48.  
49.   VarName:=TStringList.Create;
50.   VarName.Add('X');
51.   VarName.Add('T');
52.  
53.  {Create the Evaluator Object}
54.  
55.  Eval:=TEvaluator.Create(VarName,Expr);
56.  
57.  {My HP48g provided this value for PI:}
58.  
59.  IF Symvars.IndexOf('PI')<>-1 THEN      {If PI exists, then assume it is the
60.                                          circle radius vs diameter ratio}
61.     Eval.SetConstant('PI',3.14159265359);
62.  
63.  IF Symvars.IndexOf('X0')<>-1 THEN      {Set X0 to Douglas' number}
64.     Eval.SetConstant('X0',42);
65.  
66.  {All this above looks slow isn't? It probably even is. Unit symbolic has
67.   evaluations as plus, not as target. The evaluation is built for
68.   fast repeated evaluations, not just one.
69.   However the Evaluate method is hopefully reasonably fast.
70.   Under FPC TEvaluator.Evaluate is about 600-700 assembler instructions,
71.   without operation on pointer trees and without recursion.
72.   If your compiler (and hopefully FPC too) can inline the math unit functions,
73.   the speed gain could be dramatic.}
74.  
75.  Writeln('Stackdepth needed for evaluation: ',eval.EvalDepth);
76.  
77.  FOR I:=1 TO 50 DO
78.   begin
79.    Vars[0]:=1/I *1.1;
80.    Vars[1]:=1/I*2;
81.    Writeln(VarName.Strings[0] + '=' + FloatToStrF(Vars[0], ffFixed, 4, 4) + ' ' +
82.            VarName.Strings[1] + '=' + FloatToStrF(Vars[1], ffFixed, 4, 4) + ' = ' +
83.                                       FloatToStrF(Eval.Evaluate(Vars), ffFixed, 4, 4));
84.   end;
85.  
86.  Eval.Free;
87.  Expr.Free;
88.  SymVars.Free;
89. // VarName.Free;  {Is freed by TEvaluator.Destroy. Should TEvaluator copy it?}
90.  
91.   Readln;
92. end.
93.  
94.  
95. {
96.   $Log$
97.   Revision 1.1  2002/12/15 21:01:22  marco
98.   Initial revision
99.  
100. }
101.  

[wp]

Each variable must be added to the Identifiers list of the parser. By iterating through the Identifiers you can list all variables. Just try this program:

Code: Pascal  [Select][+][-]

1. program Project1;
2.  
3. uses
4.   fpexprPars;
5.  
6. var
7.   parser: TFPExpressionParser;
8.   i: Integer;
9.  
10. begin
11.   parser := TFpExpressionparser.Create(nil);
12.   try
13.     parser.Identifiers.AddFloatVariable('x', 0.5);
14.     parser.Identifiers.AddFloatVariable('y', 1.0);
15.     parser.Expression := 'x+y';
16.  
17.     for i := 0 to parser.Identifiers.Count-1 do
18.       WriteLn(parser.Identifiers[i].Name, '=', parser.Identifiers[i].AsFloat:0:3);
19.  
20.     WriteLn(parser.Expression, '=', parser.AsFloat:0:3);
21.   finally
22.     parser.Free;
23.   end;
24.  
25.   ReadLn;
26. end.

Some time ago I submitted a patch to allow also decimal commas in the expression, but it was rejected. Therefore, I think that the wiki documentation is still up-to-date. I just tested that the parser absolutely requires a decimal point when a constant is used in an expression (e.g. parser.Expression := '1.0 + 0.5';)  even if the format settings of your system want a comma. Of course this is also true for floating point variables which always have been written with a point in the Pascal language. And of course, when you convert a floating point result to a string or vice versa using StrToFloat() or FloatToStr() the decimal separator of the system's FormatSettings is needed.

[marcov]

I updated the post above with the code.

[ArtLogi]

Thank you. I will read these tomorrow and return with better reply.

But as quick reply marcovs answer is more in line what I'm interested at the moment. I was and and I'm still interested if expression parser could somehow extract all identifiers from free form user (not programmer) customer written expression (think like cropping undeclared litter), before identifiers are declared to expression parser. ...But it might be that the expression parser is not actually doing anything that advanced internally which would give such feature "for free" and therefore I do assume it do not exist, now thinking of it. 

Also about the wiki... I think it is my brainfart and poor test program (which I do correct also tomorrow) there is obviously double inversion. And the fact IIRC that S2F and F2S do use internally DecimalSeparator (or equivalent) NOT DecimalSeparator and/or dot. 

Code: Pascal  [Select][+][-]

1.  [In: , ]->[S2F: DecimalSeparator => dot]->[exprPars]->[F2S: dot => DecimalSeparator] ->[Out: ,]
2.  

Code: Pascal  [Select][+][-]

1. How I was thinking:
2.          +----------str to float-------------+              
3. [input]--|-+-[ is dot   ]----[conversion]--+ |
4.          | |                               | |
5.          | +-[ is comma ]----[conversion]--+-|--[OUT]
6.          +-----------------------------------+
7.  
8. What it actually does:
9.          +----------str to float-------------+              
10. [input]--|-+-[ is D.S   ]----[conversion]--+ |
11.          | |                               | |
12.          | +-[ is NOT D.S ]--[jmp error]   +-|--[OUT]
13.          +-----------------------------------+
14.  

... But I'm too tired, reading above still doesn't make sense ... tldr. brainfart.

[wp]

Code: Pascal  [Select][+][-]

1. How I was thinking:
2.          +----------str to float-------------+              
3. [input]--|-+-[ is dot   ]----[conversion]--+ |
4.          | |                               | |
5.          | +-[ is comma ]----[conversion]--+-|--[OUT]
6.          +-----------------------------------+
7.  

If this were true, how should a string be handled which is allowed to contain both decimal and thousand separator? So, which value could be extracted from the string '1,000' when the decimal separator is not fixed as in your sketch?

[ArtLogi]

Code: Pascal  [Select][+][-]

1. How I was thinking:
2.          +----------str to float-------------+              
3. [input]--|-+-[ is dot   ]----[conversion]--+ |
4.          | |                               | |
5.          | +-[ is comma ]----[conversion]--+-|--[OUT]
6.          +-----------------------------------+
7.  

If this were true, how should a string be handled which is allowed to contain both decimal and thousand separator? So, which value could be extracted from the string '1,000' when the decimal separator is not fixed as in your sketch?

A valid question, the wrong number most probably.  How it at the moment does handle a number 1,000.00 or 1 000.00 or 1'000 ??? 

[winni]

Hi!

Behave like you do in a foreign country:

Speak that language the people understand.

1,000.00 or 1 000.00 or 1'000  are unknown languages to fpc.

Winni

[wp]

How it at the moment does handle a number 1,000.00 or 1 000.00 or 1'000 ??? 

AFAIK not at all: StrToFloat assumes that a ThousandSeparator is not present and raises an exception otherwise. Your code must remove the ThousandSepator before passing the string to a string-to-float conversion function.

[ArtLogi]

Hi!

Behave like you do in a foreign country:

Speak that language the people understand.

1,000.00 or 1 000.00 or 1'000  are unknown languages to fpc.

Winni

Hello yes, that's why the whole OS radix thing is so annoying as everyone knows that computer only consumes floats as type 1.0 or 1.0e+0 not 1,0 (ps. which is not techinally true as ie. ARM IIRC only consumes integers)

.. And then it does make things messy if I do want to output the float in other than OS radix since s2f and f2s assumes OS radix. Why just not commands like toOSradix and fromOSradix and the f2s and s2f follow the internal convention of no thousand separation and fixed dot as radix. *Phew*     ...must calm down and start to investigate the expression parser further.

[marcov]

But as quick reply marcovs answer is more in line what I'm interested at the moment.

To accept both comma and dot as separater, it is P\probably a matter of hacking parsexpr.inc:170 to 190 a bit.

It would be a nice feature. Here in the NL use both English (dot) and Dutch (comma) notation, and not always matching their locale. In most my applications I accept both.

The apps are non financial though, so thousand separator use is rare.
 

[lucamar]

The problem, I think, is that conversion functions of that type are meant basically to parse IO to/from human users, who are notoriously inconsistent in what they want at each specific time.

A solution might be to add an "Options" (with adequate defaults) parameter (and/or some overloads) to those kind of functions to deal with (almost) all kinds of input/output: with/without thou. sep., accept any decimal sep., follow strictly local conventions, etc.

Wouldn't make life very easy for the implementors, though

[ArtLogi]

First of I must say that this topic and depth starts to be way way out of my technical expertise.

But as quick reply marcovs answer is more in line what I'm interested at the moment.

To accept both comma and dot as separater, it is P\probably a matter of hacking parsexpr.inc:170 to 190 a bit.

It would be a nice feature. Here in the NL use both English (dot) and Dutch (comma) notation, and not always matching their locale. In most my applications I accept both.

The apps are non financial though, so thousand separator use is rare.

While indeed it would be really nice for string to float type conversion, it would be logical suicide when one would need to convert from float to string. It is logically better (for human that is) to have them both behave the same way. Also it would create even more branching inside the implementation and slow things down.

Code: Pascal  [Select][+][-]

1.           +----------str to float-------------+              
2. [In: ./,]-|-+-[ is dot   ]------------------+ |
3.           | |                               | |
4.           | +-[ is comma ]----[conversion]--+-|--[OUT]
5.           +-----------------------------------+
6.  
7.  
8.           +----------float to str-------------+              
9. [IN:DOT]--|-+-[    dot   ]--------------------|--[OUT]
10.           | |                                 |         Which one of these branches ???
11.           | +-[    comma ]----[conversion]----|--[OUT]
12.           | |                                 |
13.           | +-[  Des.Sep.]--+-[dot]-----------|--[OUT]
14.           |                 |                 |
15.           |                 +-[,]-[conv]------|--[OUT]
16.           +-----------------------------------+
17.  
18.  

The problem, I think, is that conversion functions of that type are meant basically to parse IO to/from human users, who are notoriously inconsistent in what they want at each specific time.

I think the original author of this feature s2f/f2s did think it in the same way (falsely???). I do see it purely a type conversion under the hood. Only when the resulting string is then send or read to somewhere as a data it becomes an HMI input/output. However I do understand the reasoning most of its use is when someone is exchanging some data between visible representation and float variable.

Code: Pascal  [Select][+][-]

1. Pure type conversion:
2. [FLOAT AS STRING]-----[Conv.]----[FLOAT AS FLOAT]
3. [FLOAT AS FLOAT]-----[Conv.]----[FLOAT AS STRING]
4.  
5. What actually now happens:
6. [FLOAT AS STRING]--[Localizer]--[Conv.]--[FLOAT AS FLOAT]
7. [FLOAT AS FLOAT]--[Conv.]--[Localizer]--[FLOAT AS STRING]

???

Edit. Hmm... https://www.freepascal.org/docs-html/rtl/sysutils/strtofloat.html it have been working correctly in the past, but now fixed to be broken at low level. 
Edit2. Which unfortunately makes this statement false:

FloatToStr: Convert a float value to a string using a fixed format
https://www.freepascal.org/docs-html/rtl/sysutils/floattostr.html

Since now the format is dependent of the OS decimal separator. 
Edit3.

AnsiSameStr: Checks whether 2 strings are the same (case sensitive)
AnsiSameText: Checks whether 2 strings are the same (case insensitive)

With same naming convention logic these current strToFloat and FloatToStr functions should have been renamed to be:
- TxtToFlt
- FltToTxt




Simple solutions are beautiful and fits for simple person.

.... And my expression parser study have not progressed at all today. 

[lucamar]

Edit2. Which unfortunately makes this statement false:

FloatToStr: Convert a float value to a string using a fixed format
https://www.freepascal.org/docs-html/rtl/sysutils/floattostr.html

Since now the format is dependent of the OS decimal separator. 

What "fixed format" means there is that it returns a result in a function-choosen "fixed" notation (or rather, "general" notation) instead of having to specify it in the call. In that sense, FloatToStr() is a "specialization" of FloatToStrF(), so the notation is "fixed" even if it depends on the underlying TFormatSettings (which, BTW, can be changed in some overloaded forms).

Note that FloatToStr() overloads also allow you to specify format settings other than the system ones.

Of course, all that means little (yet?) with regard to TFPExpressionParser, which is the topic of this thread.

[ArtLogi]

Edit2. Which unfortunately makes this statement false:

FloatToStr: Convert a float value to a string using a fixed format
https://www.freepascal.org/docs-html/rtl/sysutils/floattostr.html

Since now the format is dependent of the OS decimal separator. 

What "fixed format" means there is that it returns a result in a function-choosen "fixed" notation (or rather, "general" notation) instead of having to specify it in the call. In that sense, FloatToStr() is a "specialization" of FloatToStrF(), so the notation is "fixed" even if it depends on the underlying TFormatSettings (which, BTW, can be changed in some overloaded forms).

Note that FloatToStr() overloads also allow you to specify format settings other than the system ones.

Of course, all that means little (yet?) with regard to TFPExpressionParser, which is the topic of this thread.

The floaty thing is mostly arguing just for sake of it. When the functionality is known to be dependent of localization one always can write a de-localizer or de-localized conversion function (although I do see it a bit stupid that I must undo something that RTL is doing as library of very basic lego-blocks) and use it, instead of using the basic blocks to construct something more complex if not already included to libraries/books/units however one like to call them.

I'm just a bit perplexed why the the two functions are developed as they have been (assuming from the doc) from SISO (skipping the fact it does handle scientific notation in format of ±xE±y) to some form of SIMO + MISO system. Instead of keeping it SISO and introducing localizer function for stringfloats. Obviously the localization is step from 1960 to 2000, but... 

What "fixed format" means there is that it returns a result in a function-choosen "fixed" notation (or rather, "general" notation) instead of having to specify it in the call.

You are probably right, it depends how one do interpret that fixed format.