Convert string with Key-Value pairs to TStrings with Key-Value pairs.

[Bart]

Hi,

Trying not to re-invent the wheel here.

I have a string as input;
It consists of multiple Key-Value pairs.
They are in the form:
Key="Value" (so, values are quoted, keys aren't)
There will never be a double quote inside a Value.

The order in which the Key-Value pairs appear in the string may vary with time.

I want to extract the Key-Value pairs into some type of record.

I've done so using multiple Pos() and Copy() calls.
The problem here may be that you can have Pairs like:
TYPE="Foo"
NEWTYPE="Bar"
And then Pos('TYPE=', AString) will find either of the above two, depending on order or startpos (last param in Pos()).

My goal here is to not be dependant on the order in which the Key-Value pairs appear in the string.

Notice: speed is not important here.
I'm parsing strings derived from executing other programs, which takes orders of magnitude more time than string parsing.

Since TStrings already has a mecahnism to read Values from Keys, I want to use that, since it's been tested and reliable.

So:
How do I parse the input string into in TStrings?

I've come up with this for now:

Code: Pascal  [Select][+][-]

1. procedure SplitLine(Line: String; List: TStrings);
2. var
3.   Len, i, PStart: Integer;
4.   Pair, Key, Value: String;
5. begin
6.   List.Clear;
7.   i := 1;
8.   Len := Length(Line);
9.   if (Len = 0) then
10.     Exit;
11.   Pair := '';
12.   while (i <= Len) do
13.   begin
14.     while (i < Len) and (Line[i] = #32)  do Inc(i);            //skip whitespace
15.     PStart := i;                                               //first non-white space: must be start of Key
16.     if (Line[i] in ['"','=']) then                             //Key cannot start with those
17.       Exit;
18.     while (i < Len) and not (Line[i] in ['=','"']) do Inc(i);  //find equal sign, we cannot find a quote before the equal sign as it cannot be part of e Key
19.     if (Line[i] <> '=') then
20.       Exit;
21.     Inc(i);
22.     if (i > Len) or (Line[i] <> '"') then                      //next char must be a doublequote
23.       Exit;
24.     Key := TrimRight(Copy(Line, PStart, i-PStart));            //don't copy the starting doublequote, allow for trailing spaces
25.     //writeln('Key=',Key);
26.     Inc(i);
27.     PStart := i;
28.     while (i < Len) and (Line[i] <> '"') do Inc(i);            //find closing doublequote  (there cannot be quotes inside Value, they will be escaped like \x<hexvalue>
29.     if (i > Len) or (Line[i] <> '"') then                      //no closing double quote -> error
30.       Exit;
31.     Value := Copy(Line, PStart, i-PStart);                     //don't copy the closing doublequote
32.     writeln('Value=',Value);
33.     Pair := Key + Value;
34.     //writeln('Pair=[',Pair,']');
35.     List.Add(Pair);
36.     Inc(i);
37.   end;
38. end;

It does some limited checking of false input.
In the end it would not really matter if some invalid Key-Value pair ends up in then TStrings, as long as all good pairs are included.

Questions:

Am I re-inventing the wheel here (converting the string to TStrings)?
Any better solutions?

Mind you: I prefer verbose code over cryptic but faster code, and I prefer strings over pchar's..

Bart

[wp]

I have a string as input;
It consists of multiple Key-Value pairs.

How are the key-value pairs in the input string separated?

[Lansdowne]

If I've understood your question, isn't it simply

Code: Pascal  [Select][+][-]

1. List.Add(Line);

And then to retrieve the value stored at Key you would call

Code: Pascal  [Select][+][-]

1. S:=List.Values[Key];
2. {then proceed to strip the " " from the Value string}

Of course this does not cater for multiple lines with the same Key.

EDITED I missed the part where "Line" contains multiple pairs.
So it's a little more involved, maybe:

Code: Pascal  [Select][+][-]

1. begin
2. While Length(Line)>0 do begin
3.   I:=0;
4.   Repeat
5.     Inc(I)
6.   Until (I>Length(Line)or(Line[I]='"'));  //first double quote
7.   Repeat
8.     Inc(I)
9.   Until (I>Length(Line)or(Line[I]='"'));  // second double quote
10.   List.Add(Copy(Line,1,I));
11.   Delete(Line,1,I+1);
12. end;
13. end;
14. { the above assumes Line is always formed according to the specification.
15. Plus it retains the " " on the Value part. }
16.  

[jamie]

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. Var
3.   A:TStringArray;
4.   S:String;
5. begin
6.   Memo1.Clear;
7.   S := 'Key="value" Key2="Value2"';
8.   A := S.Split([' '],'"');
9.   memo1.Lines.AddStrings(A);
10. end;                                        
11.  

Hows that?

Removing the quotes isn't an issue either.

Jamie

[jamie]

Hear is my rendition of a function to do so.

it should work; I just tested it with = inside the "...." to make sure it knows the difference.

Code: Pascal  [Select][+][-]

1. function GetPairsFromString(S: string): TStringArray;
2. var
3.   QuoteChar: char = '"';
4.   QuoteOpened: boolean = False;
5.   Index: integer = 0;
6.   Index2: integer;
7.   EqualFound: boolean = False;
8.   Key, Value: string;
9. begin
10.   Key := '';
11.   Value := '';
12.   repeat
13.     Inc(Index);
14.     if Index > Length(S) then Break;
15.     if S[Index] = QuoteChar then
16.     begin
17.       QuoteOpened := not QuoteOpened;
18.       If (Not QuoteOpened)and(equalFound) Then
19.        Begin
20.         SetLength(Result, Length(Result)+1);
21.         Result[Length(result)-1]:=Key+'='+'"'+Value+'"';
22.         Key := '';
23.         value := '';
24.         EqualFound := False;
25.        end;
26.       Continue;
27.     end;
28.     if (QuoteOpened) and (EqualFound) then
29.     begin
30.       Value := Value + S[Index];
31.       Continue;
32.     end;
33.     if (S[Index] = '=')and(Not QuoteOpened) then
34.     begin
35.       EQualFound := True;
36.       Key := '';
37.       Index2 := Index - 1;
38.       While (Index2>0)and(S[index2]=' ') do dec(Index2); //Remove spaces to the left of =
39.       while (Index2 > 0) and (S[Index2] <> ' ') do
40.       begin
41.         Key.Insert(0, S[index2]);
42.         Dec(Index2);
43.       end;
44.       continue;
45.     end;
46.   until Index >= Length(S);
47. end;
48.  
49. procedure TForm1.Button1Click(Sender: TObject);
50. var
51.   A: TStringArray;
52.   S: string;
53. begin
54.   Memo1.Clear;
55.   S := 'Key="value=" Key2="Value2" K3y3="=value3"';
56.   //A := S.Split([' '], '"');
57.   memo1.Lines.AddStrings(GetPairsFromString(S));
58. end;                                                            
59.  

[wp]

Quick and dirty, using the Split helper as jamie already had suggested, but such that both (multiple) spaces, semicolons, CR, LF can be allowed as separators between the key-value pairs:

Code: Pascal  [Select][+][-]

1. const
2.   InputStr = 'FirstType="T1"; Type="T2" NewType="Very new T2"' + LineEnding +
3.              'Test="Testdata"     LastType="LT"';
4.  
5. procedure TForm1.Button1Click(Sender: TObject);
6. var
7.   pairs, keyAndValue: TStringArray;
8.   i: Integer;
9. begin
10.   pairs := InputStr.Split([' ', ';', #13, #10], '"');
11.   for i := 0 to High(pairs) do
12.   begin
13.     if pairs[i]<>'' then
14.     begin
15.       keyAndValue := pairs[i].Split('=');
16.       Memo1.Lines.Add('Key=' + keyAndValue[0] + ' ---> Value=' + keyAndValue[1].DequotedString('"'));
17.     end;
18.   end;
19. end;      

[Thaddy]

Not complete yet, but close and simple:

Code: Pascal  [Select][+][-]

1. {$mode objfpc}{$H+}
2. uses sysutils,classes;
3.  
4. type
5.   TMyRec = record
6.   a,b:string;
7.   end;
8.  
9. var
10.   L:TStringlist;
11.   R:array of TMyRec;
12.   I:integer;
13. begin
14.   L :=TStringlist.Create;
15.   L.Add('k1="v 1"');
16.   L.Add('k2=v2');
17.   L.Add('k3=v3');
18.   Setlength(R,L.Count);
19.   for I := 0 to High(R) do
20.   begin
21.     R[i].a := L.names[i];
22.     R[i].b := L.Values[R[i].a].DequotedString('"');
23.     //verify:
24.     writeln(R[i].a,' ' ,R[i].b);
25.   end;
26.   readln;
27. end.

I suppose I have to adapt it a bit for strings like:
"whatever is in this string there may be multiple key="value" types, yes the string continues, butt we need to extract any key="value + something", may be we should use a regex, though"

Am I right in interpreting Bart' s question like above?
So look for left and right of Pos('=') but allow for right whitespace in left and subs. left whitespace in right? where right starts and ends with double quotes? I can adapt the above quite easy if that is really what you mean.
Extracting to record is already covered above. Just needs some more preprocessing.

[Bart]

Thanks for all the input.

@wp: since Values can contain spaces, wouldn't your use of split also split on the space inside a Value?
Key-Value pairs are separated by 1 or more spaces (no line-endings).
I read the using LoadFromStream(Proc.Output) where Ptoc is a TProcess.

@Thaddy: I don't want to drag in a dependancy on regexpression unit.
Also, as they say: you have one problem, you solve with a regex, now you have 2 problems.
(My knowledge/experience with regex's is minimal, so I wouldn't be able to maintain it if something changed.)

I'll do some testing with some of the examples using String.Split().

Bart

[Thausand]

@Bart:
https://www.freepascal.org/docs-html/rtl/sysutils/tstringhelper.split.html

If AQuoteStart and AQuoteEnd are supplied, then no splitting will be performed between AQuoteStart and AQuoteEnd characters.

wp have use overload split and have one quote. That is mean have AQuoteStart and AQuoteEnd same.

Make test for check if true and see

[Lansdowne]

@wp: since Values can contain spaces, wouldn't your use of split also split on the space inside a Value?

I thought so too.

Thanks for all the input.

I'll do some testing with some of the examples using String.Split().

Bart

I am not sure how Split can do the job.
Split on '=' is unnecessary because the Pascal Names[] Values[] does that for you.
Split on ' ' won't work because as just stated, you will split on spaces that are within, not between, records.

So my example says:

• Step one char at a time through the string.
• When you have encountered two " characters, you have completed one name-value pair.
• Store that and continue.

You could easily modify my code to not store the two " chars, but just as easy to keep them until you need to extract the value later in your code.
You could also check, when encountering the first " char, check that the char just before that is '=', if not then abort as you have a malformed string.
The other thing my code doesn't do is a use of Trim() to remove excess white space from the name and value.
So should be

Code: Pascal  [Select][+][-]

1. List.Add(Trim(Copy(Line,1,I)));

[cdbc]

Hi
Hmmm, just spitballing here...
@Bart: Wouldn't it be possible to slightly alter the parser from CSV loading/import, to trigger on spaces instead of commas, I mean it's written to cater for quoted strings/fields already, thus taking that off of your plate?!?
...Or you could write a quick'n'dirty "Finite Automata" yourself  ...state-machines can be quick 
Regards Benny

[jamie]

The reason for the quote char to be indicated is because it will not use that space as a marker so that should not happen, that is a split to take place within a quoted string.

But there is the possibility if there are some parameters that are quoted with spaces around it with no "=" on the left, the results would be a string with no Key=value indication.

 The function I provided actually searches for the '=' character and then traverses left for the Key, sets a couple of flags and while the quote is opened, builds the Value part. That should avoid that.

The only issue there, is if for some reason the value part isn't within a "....". That too can be rectified with a build of an unquoted string in that case so when the next space is detected it will assume that as the value.

Depends on what you want, I guess.

Jamie

[Lansdowne]

Hi
Hmmm, just spitballing here...
@Bart: Wouldn't it be possible to slightly alter the parser from CSV loading/import, to trigger on spaces instead of commas, I mean it's written to cater for quoted strings/fields already, thus taking that off of your plate?!?
...Or you could write a quick'n'dirty "Finite Automata" yourself  ...state-machines can be quick 
Regards Benny

Hmm, that might work if we are sure there are never spaces around the = sign (or in the Name part!).
But doesn't the CSV routine scan the string char by char anyway, in which case there is no time benefit.
And we don't have any reason to assume this project requires ultra quick processing.

[Bart]

@wp:
Your code doesn't handle a malformed string like

Code: Pascal  [Select][+][-]

1. 'bar="def"  Foo"="'

I'll spit out:

Code: [Select]

Key=bar Value=def
Key=Foo" Value="
Also, it does not allow for spaces before the equal sign (it crashes, because Pairs[i].Split('=') returns an array with length=1).

@Lansdowne
Your code doesn't compile ;-)
It also does not handle a sapce before the equal sign.

Handling spaces before or after the equal sign wasn't part of the "job description" though.

Bart

[jamie]

@bart

I just inserted one line of code to remove the spaces to the left of the "=" The spaces to the right of the "=" already
get removed unless they are found inside the quote or at least they should anyways.

 I did this online, didn't local test it but the change was simple with a comment.


Jamie