Recent

Author Topic: Read JSON names  (Read 2919 times)

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Read JSON names
« on: May 23, 2022, 09:57:39 am »
TJSONData deals with values, not names. There doesn't seems to be an easy way to get the name of the top object and a list of the names of the children. It is assumed you already know them, and know what type of data structure (Value, Array, Object) it is.

Like, if I have this JSON:

Code: Text  [Select][+][-]
  1. {
  2.         "SomeJSON": {
  3.                 "SomeName": "SomeValue",
  4.                 "AnObject": {
  5.                         "AnotherName": 42,
  6.                         "AnArray": [
  7.                                 "Bla",
  8.                                 "Omg",
  9.                                 "Zoef"
  10.                         ]
  11.                 }
  12.         }
  13. }

I would like to find out that the top object is named "SomeJSON" and has two children. And that the first child is named "SomeName" and has a value, but no children. And that the second child is named "AnObject" and has two children. Etc.

What is the simplest way do do that?

AlexTP

  • Hero Member
  • *****
  • Posts: 2386
    • UVviewsoft
Re: Read JSON names
« Reply #1 on: May 23, 2022, 10:31:53 am »
I know that fcl-JSON has a method to enumerate all names in the given path (e.g. path '/' for the root). I don't remember it.

avk

  • Hero Member
  • *****
  • Posts: 752
Re: Read JSON names
« Reply #2 on: May 23, 2022, 12:23:05 pm »
Somehow, nothing simpler than this comes to mind:
Code: Pascal  [Select][+][-]
  1. {$push}{$B-}
  2. function SuitedJson(aJson: TJsonData): Boolean;
  3. var
  4.   Node: TJsonData;
  5. begin
  6.   if (aJson = nil) or (aJson.JSONType <> jtObject) then
  7.     exit(False);
  8.   Node := aJson.FindPath('SomeJSON');
  9.   if (Node = nil) or (Node.Count <> 2) then
  10.     exit(False);
  11.   Node := aJson.FindPath('SomeJSON.SomeName');
  12.   if (Node = nil) or not(Node.JSONType in ValueJSONTypes) then
  13.     exit(False);
  14.   Node := aJson.FindPath('SomeJSON.AnObject');
  15.   Result := (Node <> nil) and (Node.JSONType = jtObject) and (Node.Count = 2);
  16. end;
  17. {$pop}
  18.  

bytebites

  • Hero Member
  • *****
  • Posts: 632
Re: Read JSON names
« Reply #3 on: May 23, 2022, 01:01:28 pm »
Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. uses
  4.   fpjson,
  5.   jsonparser;
  6.  
  7. var
  8.   j: TJSONData;
  9.   s: string;
  10.   depth: integer = -1;
  11.  
  12.   procedure print(obj: tjsondata);
  13.   var
  14.     it: TJSONEnum;
  15.   begin
  16.     depth := depth + 1;
  17.     for it in obj do
  18.     begin
  19.       Write(space(depth * 4), it.key, ' ');
  20.       if TJSONData(it.Value).Count > 0 then
  21.       begin
  22.         writeln;
  23.         print(it.Value);
  24.       end
  25.       else
  26.         writeln(it.Value.AsString);
  27.     end;
  28.   end;
  29.  
  30. begin
  31.   s := ' {        "SomeJSON": {                "SomeName": "SomeValue",                "AnObject": {                        "AnotherName": 42,                        "AnArray": [                                "Bla",                                "Omg",                                "Zoef"                        ]                }        }}';
  32.   j := GetJSON(s);
  33.   print(j);
  34. end.    

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Read JSON names
« Reply #4 on: May 23, 2022, 01:08:22 pm »
Ah, the key of a TJSONEnum. I'll remember that.

But in the mean time, I've written my own JSON deserializer. Far easier than YAML.

Thanks for the suggestions!

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Read JSON names
« Reply #5 on: May 23, 2022, 01:11:20 pm »
I would like to find out that the top object is named "SomeJSON" and has two children.

Just to be clear: in your specific example the top object has no name and has a single member SomeJSON (the top could after all also be an array for example).

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Read JSON names
« Reply #6 on: May 23, 2022, 01:28:28 pm »
I would like to find out that the top object is named "SomeJSON" and has two children.

Just to be clear: in your specific example the top object has no name and has a single member SomeJSON (the top could after all also be an array for example).
True. I treat that as the "document", as the other formats (YAML, XML, Ini, Text) don't have such an extra wrapper. They are removed automatically, because the tokenizer strips them and calls itself if there was only one element:

Code: Pascal  [Select][+][-]
  1. function TNode.SplitJSON(ThisText: string; var IsArr: Boolean): TStringArray;
  2. var
  3.   sl: TStringList;
  4.   s: string;
  5.   c: Char;
  6.   b: Boolean;
  7.   i, Inside, Start: Integer;
  8. begin
  9.   IsArr := False;
  10.   SetLength(Result, 0);
  11.  
  12.   if Empty(ThisText) then Exit;
  13.  
  14.   sl := TSTringList.Create;
  15.   ThisText := Trim(ThisText);
  16.   Start := 1;
  17.   Inside := 0;
  18.  
  19.   for i := 1 to Length(ThisText) do
  20.   begin
  21.     c := ThisText[i];
  22.     if (c = '{') or (c = '[') then Inc(Inside)
  23.     else if (c = '}') or (c = ']') then Dec(Inside);
  24.     if ((c = ',') or (i = Length(ThisText))) and (Inside = 0) then
  25.     begin
  26.       s := Trim(System.Copy(ThisText, Start, i - Start + 1));
  27.       Start := i + 1;
  28.       if s[Length(s)] = ',' then
  29.         s := Trim(System.Copy(s, 1, Length(s) - 1));
  30.       sl.Add(s);
  31.     end;
  32.   end;
  33.  
  34.   if sl.Count = 1 then
  35.   begin
  36.     s := sl[0];
  37.     if (s[1] = '[') Then IsArr := True;
  38.     if ((s[1] = '[') or (s[1] = '{')) and
  39.       ((s[Length(s)] = ']') or (s[Length(s)] = '}')) then
  40.     begin
  41.       s := Trim(System.Copy(s, 2, Length(s) - 2));
  42.       sl.AddStrings(SplitJSON(s, b), True);
  43.     end;
  44.   end;
  45.   Result := sl.ToStringArray;
  46.  
  47.   sl.Free;
  48. end;


Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: Read JSON names
« Reply #7 on: May 23, 2022, 07:16:28 pm »
Hey SymbolicFrank,

TJSONData deals with values, not names. There doesn't seems to be an easy way to get the name of the top object and a list of the names of the children. It is assumed you already know them, and know what type of data structure (Value, Array, Object) it is.

If what you're stating here would be found true, then you could never aspire to write a JSON viewer.
Not being able to determine the type of node and/or the existence or lack of child nodes would be the death of any viewer that wouldn't know the structure before hand.

Let me ease your pain by giving you the example of such viewer that takes care of Node's names, types and children: Lazarus JSON Viewer

Pay attention to this procedure, that is called recursively on the root node of a JSON file(File: src/forms/ljv.forms.main.pas):
Code: Pascal  [Select][+][-]
  1. procedure TfrmMain.UpdateTreeFromNode(const ANode: PVirtualNode;
  2.   const AJSONData: TJSONData; const APath: UTF8String);

That's all the info you need to get at all the information contained on, either, a TJSONData, or the more specific incarnations like TJSONObject, TJSONArray, TJSONString, TJSONNumber, etc...

Cheers,
Gus

Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: Read JSON names
« Reply #8 on: May 23, 2022, 07:30:51 pm »
Hey SymbolicFrank,

in your code, lines 23 and 23, the way you put it may induce in issues

Code: Pascal  [Select][+][-]
  1. function TNode.SplitJSON(ThisText: string; var IsArr: Boolean): TStringArray;
  2. var
  3.   sl: TStringList;
  4.   s: string;
  5.   c: Char;
  6.   b: Boolean;
  7.   i, Inside, Start: Integer;
  8. begin
  9.   IsArr := False;
  10.   SetLength(Result, 0);
  11.  
  12.   if Empty(ThisText) then Exit;
  13.  
  14.   sl := TSTringList.Create;
  15.   ThisText := Trim(ThisText);
  16.   Start := 1;
  17.   Inside := 0;
  18.  
  19.   for i := 1 to Length(ThisText) do
  20.   begin
  21.     c := ThisText[i];
  22.     if (c = '{') or (c = '[') then Inc(Inside)
  23.     else if (c = '}') or (c = ']') then Dec(Inside);
  24.     if ((c = ',') or (i = Length(ThisText))) and (Inside = 0) then
  25.     begin
  26.       s := Trim(System.Copy(ThisText, Start, i - Start + 1));
  27.       Start := i + 1;
  28.       if s[Length(s)] = ',' then
  29.         s := Trim(System.Copy(s, 1, Length(s) - 1));
  30.       sl.Add(s);
  31.     end;
  32.   end;
  33.  
  34.   if sl.Count = 1 then
  35.   begin
  36.     s := sl[0];
  37.     if (s[1] = '[') Then IsArr := True;
  38.     if ((s[1] = '[') or (s[1] = '{')) and
  39.       ((s[Length(s)] = ']') or (s[Length(s)] = '}')) then
  40.     begin
  41.       s := Trim(System.Copy(s, 2, Length(s) - 2));
  42.       sl.AddStrings(SplitJSON(s, b), True);
  43.     end;
  44.   end;
  45.   Result := sl.ToStringArray;
  46.  
  47.   sl.Free;
  48. end;

If the string contains, for example, "{[]]", you'll have an Inside=0 at the end, but the JSON is not valid.

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Read JSON names
« Reply #9 on: May 24, 2022, 10:14:43 am »
Hey SymbolicFrank,

in your code, lines 23 and 23, the way you put it may induce in issues

If the string contains, for example, "{[]]", you'll have an Inside=0 at the end, but the JSON is not valid.

Cheers,
Gus

Good point. My code does expect the input to be valid JSON.

And to answer your previous post, this is my deserializer:

Code: Pascal  [Select][+][-]
  1. procedure TNode.FromJSON(ThisText: string);
  2. var
  3.   sa: TStringArray;
  4.   s: string;
  5.   IsArr: Boolean;
  6. begin
  7.   if Empty(ThisText) then Exit;
  8.  
  9.   sa := SplitJSON(ThisText, IsArr);
  10.   if Length(sa) = 1 then
  11.   begin
  12.     Deserialize(sa[0], nfJSON);
  13.     SetLength(sa, 0);
  14.     s := '';
  15.     if IsString then s := Trim(VarToStr(MyValue));
  16.     if (Length(s) > 0) and ((s[1] = '[') or (s[1] = '{')) then
  17.     begin
  18.       sa := SplitJSON(MyValue, IsArr);
  19.       if IsArr then MyKind := nkArray;
  20.       MyValue := Null;
  21.     end;
  22.   end;
  23.  
  24.     for s in sa do if not Empty(s) then
  25.       MyChildren.Add(TNode.Deserialize(Self, s, nfJSON));
  26.  
  27.   SetLength(sa, 0);
  28. end;

This line:

Code: Pascal  [Select][+][-]
  1. MyChildren.Add(TNode.Deserialize(Self, s, nfJSON));

is the recursion and this one

Code: Pascal  [Select][+][-]
  1. Deserialize(sa[0], nfJSON);

assigns the name and value to the node.

Ok, improvements can be made. The tokenizer uses multiple passes. And indeed, if the JSON is invalid, it breaks. But it's pretty simple and it works well.
« Last Edit: May 24, 2022, 10:18:12 am by SymbolicFrank »

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: Read JSON names
« Reply #10 on: May 24, 2022, 10:51:01 am »
Hey SymbolicFrank,

Ok, improvements can be made. The tokenizer uses multiple passes. And indeed, if the JSON is invalid, it breaks. But it's pretty simple and it works well.

Ok, while I do applaud your efforts in building a JSON parser, I now need to ask: Why re-invent the wheel?

The fpjson unit comes with Free Pascal, is maintained by the core devs, it's updated and current and it's been battle tested in the wild.
It even has a set of unit testing for it. Do you have unit testing for your JSON de-serializer code?
So I'm baffled as to why you'd want to roll out your own?

Is it because you don't understand how it works, so you feel uncomfortable using it and so you roll your own, to which you're more familiar, comfortable with?

I don't get it, I'm sorry. I really am sorry for not understanding your approach.

Is this the reason that you're also battling with TFPHTTPClient on the other thread?

If this is the case, I encourage you to stop. Any coder worth it's salt has to be able to use the black boxes that have been in use for many years and have the battle scars to prove it.
Re-inventing the wheel each time you don't understand a black box will get you completely off schedule and then your manager is at your throat that the client ain't gonna pay for any extra time.

Just some friendly advice here.

Unless of course this is purely in the open source domain and then your time is yours to waste as you see fit.

But if you ever go back to an enterprise context, you're in trouble with this re-invent the wheel attitude ;)

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Read JSON names
« Reply #11 on: May 24, 2022, 11:24:19 am »
Hi Gus,

See the first post why the fpjson unit didn't work out. I do use it in another part of the application I'm building, but it won't work for deserializing.

To explain what I'm doing: I have to communicate with a large REST API (thousands of functions with often large data structures). That's a lot of work to implement. And it changes frequently. They do have description files, in RAML and OpenAPI. That's both YAML. So, it seems to be best to parse that and generate a big node tree, that contains all the functions, enums and data structures, and the references between them. Which is what I did. Which wasn't easy, because almost every text is valid YAML. It's very unstructured.

You get the parameters and body belonging to a function (by name), fill them in and call the function. Those parts are serialized and send to the webservice, the response is turned into another node tree. Which you then can process. I'm thinking of adding a DataSet interface as well.

That way, it is fully dynamic and I don't have to make (or generate) a class for each data structure, and can automate calling them.

And as it turns out, I've written most of that myself. There is no FP YAML, RAML or OpenAPI parser that I know of. Or extended, serializable nodes. I do use library functions and objects, of course. TSTringlist, some containers from Generics.Collections, Variants, LazUTF8 and fphttpclient. And I follow all the conventions, my classes look the same and have the same methods as the default ones.

For the other side of the application, I made some custom variants of default Lazarus components, like TDbf. And a large library of helper functions, of course. I did post a few bugfixes on the bugtracker, and I have more, but the last one is not accepted yet, so I have forked my own libraries.

That's how you do it, right?

And treating components as black boxes is fine, if they do what you want them to do. If not, you need another one or write your own.

Greetings, Frank

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: Read JSON names
« Reply #12 on: May 24, 2022, 12:15:05 pm »
Hey Frank,

To explain what I'm doing: I have to communicate with a large REST API (thousands of functions with often large data structures). That's a lot of work to implement. And it changes frequently. They do have description files, in RAML and OpenAPI. That's both YAML. So, it seems to be best to parse that and generate a big node tree, that contains all the functions, enums and data structures, and the references between them. Which is what I did. Which wasn't easy, because almost every text is valid YAML. It's very unstructured.

You get the parameters and body belonging to a function (by name), fill them in and call the function. Those parts are serialized and send to the webservice, the response is turned into another node tree. Which you then can process. I'm thinking of adding a DataSet interface as well.

That way, it is fully dynamic and I don't have to make (or generate) a class for each data structure, and can automate calling them.

And as it turns out, I've written most of that myself. There is no FP YAML, RAML or OpenAPI parser that I know of. Or extended, serializable nodes. I do use library functions and objects, of course. TSTringlist, some containers from Generics.Collections, Variants, LazUTF8 and fphttpclient. And I follow all the conventions, my classes look the same and have the same methods as the default ones.

For the other side of the application, I made some custom variants of default Lazarus components, like TDbf. And a large library of helper functions, of course. I did post a few bugfixes on the bugtracker, and I have more, but the last one is not accepted yet, so I have forked my own libraries.

Ahh, yes, now it all makes more sense and I do need to apologise for some of my earlier comments about re-inventing the wheel.

But I still don't understand why you made a JSON de-serialiser and then you mention YAML and not JSON. Cuz, yeah, there is no fpyaml unit, eheheh

Another thing is that when you have the response in JSON containing a huge data tree, it's already in tree format and you can query it either via an XPath alike string, or iterating through all the TJSONData nodes. Which again makes me ask, why your own JSON de-serializer that does not stack up to fpjson?
But nevermind, this is just me being super confused due to lack of data.

My personal option when dealing with JSON is to have proper Object Pascal classes to contain the data and can serialise/de-serialise themselves into/from JSON.
This, obviously, kinda doubles or triples the work, but I like the flexibility of a TObject better than the granularity of TJSONData.
So obviously, just a matter of taste, nothing else.

That's how you do it, right?

Yeap, this is exactly how one goes about it !!

And treating components as black boxes is fine, if they do what you want them to do. If not, you need another one or write your own.

Yes, I agree that when you've exhausted the options provided by the black box and find it lacking, you then either extend it or create your own.
But before you made that lengthy and quite nice explanation of your tribulations, it sounded like you explored 10% of the black box and then you proceeded to whine that it didn't work.

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Read JSON names
« Reply #13 on: May 24, 2022, 12:32:48 pm »
Hi Gus,

The description is YAML, so I have to parse that to know what functions (paths) are available and what data structures and parameters they expect. I need to know the names of them and their type, and their allowed values (like with an enum).

Then, all of that gets serialized, some to text and the body to JSON. Then it gets sent to the REST function, which returns a JSON document. Which I then turn into a bunch of nodes. And therefore, I need to know their names.


And yes, often this gets mapped with a code generator. But then you need one of those (and there is none that I know of for FP) and be prepared to recompile it every time it changes. If you don't want to do that, you need a dynamic structure you can save to and load from file. Which is what I made.

Greetings, Frank

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: Read JSON names
« Reply #14 on: May 24, 2022, 02:01:50 pm »
Hey Frank,

//...
Which is what I made.

Ahhh, yes, this now makes perfect sense :)

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

 

TinyPortal © 2005-2018