Print Page - A new design for a JSON Parser

Programming => General => Topic started by: sysrpl on August 26, 2019, 09:12:55 am

Title: A new design for a JSON Parser
Post by: sysrpl on August 26, 2019, 09:12:55 am

I know the FCL already has a capable JSON parser, but I am writing some Amazon web service interfacing projects and wanted a smaller easier to use JSON parser to assist. I've create a new design for a JSON parser that is pretty small, yet powerful.

If your interested, I've posted the code under GPLv3 and a write up of my thought process and the workflow of using an single small class to work with JSON:

https://www.getlazarus.org/json/

Any and all feedback is welcome.

Title: Re: A new design for a JSON Parser
Post by: k1ng on August 26, 2019, 10:57:01 am

Hey,
nice work! Would be nice to see a speed comparison with LkJSON :)

Title: Re: A new design for a JSON Parser
Post by: sysrpl on August 26, 2019, 11:51:17 am

I am unsure how fast or slow it is, but I didn't design it for speed. That's not to say it's slow, but it's meant to be small, with powerful features, and just one class / unit.

With regard to speed, I am creating 1 pascal object for every node. If I wanted to make it fast I would getmem for many objects at once, and neither create nor destroy them. Instead I would put or get object memory from that pool and not heap allocation / deallocation an object for each node parsed.

That said, would it really be worth it? Do your programs spend most of their time parsing JSON? Are you writing a heavy traffic outward facing service that parses JSON frequently?

If this is the case and speed / scalability is a concern then you probably want to switch to nodejs which is optimized for heavy traffic and parallelization, and is based on JSON to boot. Many smart engineers have designed nodejs for exactly this use case.

Title: Re: A new design for a JSON Parser
Post by: marcov on August 26, 2019, 12:03:31 pm

(the pooling functionality in the fcl-XML unit has also been designed out over time, to sensitive, maintenance wise)

Title: Re: A new design for a JSON Parser
Post by: k1ng on August 26, 2019, 01:06:07 pm

Quote from: sysrpl on August 26, 2019, 11:51:17 am

I am unsure how fast or slow it is, but I didn't design it for speed. That's not to say it's slow, but it's meant to be small, with powerful features, and just one class / unit.

With regard to speed, I am creating 1 pascal object for every node. If I wanted to make it fast I would getmem for many objects at once, and neither create nor destroy them. Instead I would put or get object memory from that pool and not heap allocation / deallocation an object for each node parsed.

I'm not sure how LkJSON works internally but I assume it also creates objects for each node as you refer to them via Field identifier.

Code: Pascal [Select][+]

js := TlkJSONObject.Create();
js := TlkJSON.ParseText(jsonstr) as TlkJSONObject;
 
if js.Field['name'].Field['surname'].SelfType <> jsNull then
  surname := String(js.Field['name'].Field['surname'].Value);

So for me it seems both are working more or less the same just with a different in usage when getting values. For the latter I'd prefer your version as one don't need some typecast. Using AsString etc is more common in recent Delphi/FPC.
It was just a suggestion because I think more users would use your library if it's also faster/comparable to LkJSON, just with a better syntax. So it'd be another pro to try your version ;)

Quote from: sysrpl on August 26, 2019, 11:51:17 am

That said, would it really be worth it? Do your programs spend most of their time parsing JSON? Are you writing a heavy traffic outward facing service that parses JSON frequently?

If this is the case and speed / scalability is a concern then you probably want to switch to nodejs which is optimized for heavy traffic and parallelization, and is based on JSON to boot. Many smart engineers have designed nodejs for exactly this use case.

No, personally I don't need much JSON parsing but others may do. E.g. if your library is 100x times slower than LkJSON (I don't know if there are any other Delphi+FPC JSON Parsers) many people wouldn't use your version as the only 'pro' would be the different syntax but as you only need to write code once...

Title: Re: A new design for a JSON Parser
Post by: minesadorada on August 26, 2019, 01:15:50 pm

Good work sysrpl. I like a design based on simplicity of use and clear syntax.
Thank you for your effort.

Title: Re: A new design for a JSON Parser
Post by: marcov on August 26, 2019, 01:25:35 pm

Quote from: k1ng on August 26, 2019, 01:06:07 pm

I don't know if there are any other Delphi+FPC JSON Parsers

FPC comes with its own, fcl-json

Title: Re: A new design for a JSON Parser
Post by: sysrpl on August 26, 2019, 03:54:19 pm

king,

Code: Pascal [Select][+]

js := TlkJSONObject.Create();
js := TlkJSON.ParseText(jsonstr) as TlkJSONObject;
 
if js.Field['name'].Field['surname'].SelfType <> jsNull then
  surname := String(js.Field['name'].Field['surname'].Value);

The equivalent version of with my library would be:

Code: Pascal [Select][+]

N := TJsonNode.Create;
if N.TryParse(S) and (N.Find('name/surname') <> nil) then
  SurName := N.Find('name/surname').AsString;

With regards to speed, I am considering an experiment for my own curiosity. Here is how and what I would test.

1) Time parsing a large JSON structure thousands of times.
2) Remove the TJsonNode create during the parsing, internally overwriting the same node over and over again and repeat the same test.
3) Remove the internal TList and add, and repeat the test again yet again and note the time.

This should give me a good base line to understand how much time it take the FPC to parse JSON with my library, first as it is now, second as it would be with some type of object pooling, and third with a fixed size list shared among all nodes.

If the times show a marked difference in speed, then adding pooling and a shared list might be a worthwhile enhancement. Also, I may test against uLkJSON. I've look at its source code and I'll be curious to see the speed difference.

Thank you for your replies.

Title: Re: A new design for a JSON Parser
Post by: serbod on August 26, 2019, 04:40:17 pm

Another implementation of JSON serialization:

https://github.com/serbod/dbitems/blob/master/datastorage.pas
https://github.com/serbod/dbitems/blob/master/JsonStorage.pas

IDataStorage (TDataStorage) - abstract item similar to Variant, that can store any value/list/dictionary. Using as Interface allows automatic free unused items by refcount.

TDataSerializerJson - serialize/deserialize items to/from JSON.

TDataSerializerBencode - Bencode serializer/deserializer. Fast, compact and human-readable. Used in torrent files, for example.

Title: Re: A new design for a JSON Parser
Post by: BeniBela on August 26, 2019, 05:54:32 pm

My internettools (http://benibela.de/sources_en.html#internettools) can do JSON, too:

The above task could be solved as:

Code: Pascal [Select][+]

  xsurname := query('json($_1)/name/surname', [jsonstr]);
  if not xsurname.isUndefined then
    surname := xsurname.toString;
 

It is the opposite of being small. If jsonstr was an url, it would be downloaded from the internet; and you can use it with same syntax for HTML

Title: Re: A new design for a JSON Parser
Post by: avra on August 26, 2019, 08:53:14 pm

Quote from: sysrpl on August 26, 2019, 09:12:55 am

https://www.getlazarus.org/json/

I could not connect https because of invalid certificate. I could not connect http because OpenDNS flagged site as malware.

Title: Re: A new design for a JSON Parser
Post by: Leledumbo on August 27, 2019, 08:38:24 pm

I see a few improvements over fcl-json interface. First is that a lot fcl-json methods accepts or returns TJSONData, the top most generic JSON value representation. This will require downcasting to actual type everytime the real value will be used. Second, you node is designed with parent node access, so moving around the tree is possible without the need to keep parent node reference. Other than those, they're pretty much equal. I just with the root node doesn't have to be explicitly created (well, a little wrapper function similar to fcl-json's GetJSON is rather easy to make).

Title: Re: A new design for a JSON Parser
Post by: heejit on August 27, 2019, 10:37:22 pm

If possible please add your library into Online package manager
it help this library available to many user.

Title: Re: A new design for a JSON Parser
Post by: sysrpl on August 28, 2019, 01:36:03 am

I appreciate the feedback

Leledumbo,

I just thought I'd explain the Find(Path) syntax, as it relates to your mentioning of the Parent node.

Code: Pascal [Select][+]

AnyNode.Find('/'); // returns the root node
AnyNode.Find('search/for/name'); // returns a node 3 levels from the current node
AnyNode.Find('/search/for/name'); // returns a node 3 levels from the root node
 

There is also a NodeByName property which does not try to evaluate a path.

Code: Pascal [Select][+]

AnyNode.Find('/search/for/name'); // returns a node directly under the current name with a name of "/search/for/name"

In other words if S contains:

{
"test": {
},
"stuff": {
"enabled": true,
"/search/for/name": "you've found me"
}
}

Then ...

Code: Pascal [Select][+]

N := TJsonNode.Create;
N.Value := S;
N := N.Find('stuff');
WriteLn(N.NodeByName['/search/for/name'].AsString);
N.Root.Free;

Outputs:

you've found me

Title: Re: A new design for a JSON Parser
Post by: sysrpl on August 28, 2019, 02:00:09 am

More usage examples:

Each node can parse / load / save JSON text at any level. That is you can compose a document like so:

Code: Pascal [Select][+]

N := TJsonNode.Create;
N.Add('employees').LoadFromFile('employees.json');
N := N.Add('contacts');
N.LoadFromFile('contacts.json');
N.Add('emergency').LoadFromFile('emergency.json');
WriteLn(N.Value);
WriteLn(N.Root.Value);

Would write out first:

{
.. contact nodes
"emergency": {
.. emergency nodes
}
}

Then write out second:

{
"employees": {
.. employee nodes
},
"contacts": {
.. contact nodes
"emergency": {
.. emergency nodes
}
}
}

So in this way my library allows you to load, parse, or set the JSON value of any node at any level as if it were the root node. The difference being is that child nodes just append to more nodes which ultimate fall under the same root.

The only requirement is this is that the JSON for the root node must be in the form of an object {} or array []. Child node JSON can be object {}, array [], boolean true/false, null null, number 123, or string "hello world" (note the double quotes).

To use NON JSON with nodes, such as 'hello world' (single quotes), use the type safe properties:

AsObject
AsArray
AsBoolean
AsNull
AsNumber
AsString

Title: Re: A new design for a JSON Parser
Post by: Leledumbo on August 29, 2019, 02:15:34 am

Quote from: sysrpl on August 28, 2019, 01:36:03 am

Code: Pascal [Select][+][-]
...
AnyNode.Find('search/for/name'); // returns a node 3 levels from the current node
AnyNode.Find('/search/for/name'); // returns a node 3 levels from the root node

I think I missed the difference between the two. From your example, you go for the second after setting N to stuff node and in this case, stuff node is the root? Then what is current node here? If instead you go for the first, what will you get?

Title: Re: A new design for a JSON Parser
Post by: sysrpl on August 29, 2019, 06:02:41 am

It the same as if you were typing a file system path. If your path string brings with a forward slash, then the path identities an item starting at the root of the files system. If it does not start with a forward slash, then the path evaluates from the current directory.

So for example:

NodeA.Find('/preferences/pallets/inspector/visible').AsBoolean;
NodeA.Find('visible').AsBoolean;

The first line would search the JSON starting at the root, even if NodeA is not the root.

The second line would search for an item called "visible" directly under NodeA.

So the way it works is you can Find from the root level at any node if the path string begins with "/". This is how files paths works, it's how XPath works, and I may eventually make Find support more of XPath.

Title: Re: A new design for a JSON Parser
Post by: Leledumbo on August 29, 2019, 01:01:04 pm

Quote from: sysrpl on August 29, 2019, 06:02:41 am

It the same as if you were typing a file system path. If your path string brings with a forward slash, then the path identities an item starting at the root of the files system. If it does not start with a forward slash, then the path evaluates from the current directory.

If that's so, recalling your example:

Code: Pascal [Select][+]

N := N.Find('stuff');
WriteLn(N.NodeByName['/search/for/name'].AsString);
 

Shouldn't output "you've found me", instead 'search/for/name' or '/stuff/search/for/name' should, but the latter doesn't care whether N points to 'stuff' node or not. Am I right or something is missing here?

Title: Re: A new design for a JSON Parser
Post by: krexon on September 25, 2019, 11:05:48 am

@sysrpl I use your parser, because fpJsonparser can't parse JSON, when there are duplicate keys.
Everything works fine, but ...

There is a key with price (always with 4 decimal places, but last 2 digits are zero), ie.

Code: Pascal [Select][+]

'price1': 0.9900
'price2': 1.9900

I get this price using such code:

Code: Pascal [Select][+]

p1 := n.Find('price1').AsNumber // p1: double
p2 := n.Find('price2').AsNumber // p2: double
ShowMessage(FloatToStr(p1+p2));

At 2 computers (Windows 10) I didn't have any problems, but at one computer (Windows 10) sometimes above code shows 0 instead 2.98

I modified code to check if parser gets proper JSON value:

Code: Pascal [Select][+]

p1 := n.Find('price1').AsNumber // p1: double
p2 := n.Find('price2').AsNumber // p2: double
pj1 := n.Find('price1').AsJSON // pj1: string
pj2 := n.Find('price2').AsJSON // pj2: string
ShowMessage(FloatToStr(p1+p2) + LineEnding + p1 + LineEnding + p2));

Then it shows:
0
0.9900
1.9900
Everything works fine after restarting app. Problem occures again after some time :(
So It seems that getting value AsNumber is broken

Title: Re: A new design for a JSON Parser
Post by: lainz on December 10, 2019, 12:59:13 am

Thanks for this Library.

A thing I found that maybe can be improved is 'AsString', maybe you can do some defaults when the value is Double? Like doing automatically value.ToString?

As well maybe you can provide AsInteger? Doing internally a trunc(value)...?

Title: Re: A new design for a JSON Parser
Post by: marcov on December 10, 2019, 12:49:58 pm

krexon: do those computers have different locale systems?

Title: Re: A new design for a JSON Parser
Post by: Hansaplast on December 23, 2019, 01:48:00 pm

Quote from: sysrpl on August 26, 2019, 09:12:55 am

I know the FCL already has a capable JSON parser, but I am writing some Amazon web service interfacing projects and wanted a smaller easier to use JSON parser to assist. I've create a new design for a JSON parser that is pretty small, yet powerful.

I've been tinkering with several JSON parsers, and just wanted to express my gratitude for this fast and very easy to use parser. It works great for my purposes.

Title: Re: A new design for a JSON Parser
Post by: mboxmas on January 21, 2020, 07:48:43 pm

Hi sysrpl,

For the following JSON data

{
"attr1": "value1",
"attr2": {
"attr21": "value21",
"attr22": "value22"
}
}

why the construct

for C in N do
WriteLn(C.Value);

for attrib1 yields only the value, but for attr2 yields the attribute-value pair?

Title: Re: A new design for a JSON Parser
Post by: soerensen3 on January 21, 2020, 09:37:55 pm

Is it just me or is your page getlazarus always redirecting to youtube?
Maybe the site has been hacked?

Title: Re: A new design for a JSON Parser
Post by: GAN on January 21, 2020, 10:12:55 pm

Quote from: soerensen3 on January 21, 2020, 09:37:55 pm

Is it just me or is your page getlazarus always redirecting to youtube?
Maybe the site has been hacked?

Yes, redirecting to youtube.
I checked the site:

Connecting to https://www.getlazarus.org
Exception: Unexpected response status code: 500
Status: 500

Title: Re: A new design for a JSON Parser
Post by: Hansaplast on January 22, 2020, 12:43:33 pm

Redirection here as well - sure seems "hacked" ...

For anyone is interested (I hope sysrpl doesn't mind);
I still have a copy of the JSONTools source and the documentation that I had found on getlazarus.org (saved as RTF).
This is a copy of 12/23/2019.

I'm using this version in one of my recent projects to read JSON files - I love its simplicity and its very good performance.

Title: Re: A new design for a JSON Parser
Post by: Thaddy on January 22, 2020, 01:54:24 pm

getlazarus has never been an official source.

Title: Re: A new design for a JSON Parser
Post by: soerensen3 on January 27, 2020, 02:04:33 pm

I really like it. Especially the possibility to walk nodes from child to parent and to use absolute paths is nice.
However I'm missing some features.

- You do not have the formatjson method to format the output with spaces which makes the output less readable.
- A GetJSON function would be nice which is trivial to implement.
- There is no possibility to add existing nodes to the tree (None I could find).

Title: Re: A new design for a JSON Parser
Post by: lainz on January 27, 2020, 05:14:07 pm

Quote from: Thaddy on January 22, 2020, 01:54:24 pm

getlazarus has never been an official source.

Well, yes for jsontools that is what we're talking about

But the source is at github
https://github.com/sysrpl/JsonTools/blob/master/jsontools.pas

Title: Re: A new design for a JSON Parser
Post by: GDean on July 11, 2020, 08:04:24 am

Quote from: sysrpl on August 26, 2019, 09:12:55 am

Any and all feedback is welcome.

Great work sysrpl. Checked speed and it was at least 30% faster than fpjson in my middle tier app. Some 1.6 seconds to return a string in 100,000 loops running on a i7 3770k.

Given my middle tier is parsing a lot of json data from vue front end, Its speed does make a lot of difference.

I have swapped over to yours :)

Thanks Glen

Title: Re: A new design for a JSON Parser
Post by: Awkward on July 11, 2020, 08:31:52 am

Yes, JSONTools is good but looks like unfinished and abandoned :(

Title: Re: A new design for a JSON Parser
Post by: alantelles on August 13, 2020, 07:17:04 am

Quote from: sysrpl on August 26, 2019, 09:12:55 am

I know the FCL already has a capable JSON parser, but I am writing some Amazon web service interfacing projects and wanted a smaller easier to use JSON parser to assist. I've create a new design for a JSON parser that is pretty small, yet powerful.

If your interested, I've posted the code under GPLv3 and a write up of my thought process and the workflow of using an single small class to work with JSON:

https://www.getlazarus.org/json/

Any and all feedback is welcome.

I used your parser to be the json to dictionary parser for my UltraGen language. Thanks! it was easy to use your parser to make the conversion.

https://github.com/alantelles/ultragen (https://github.com/alantelles/ultragen)

Thanx!

Title: Re: A new design for a JSON Parser
Post by: VTwin on October 15, 2020, 01:22:43 am

I have been using json to store program preferences, recently switching from xml. I have been working with a user from Costa Rica who reports floating point errors in Windows 10. I can not reproduce the issue myself, even when internationalizing to Costa Rica on my computer.

The error seems to have started in the version that changes over from xml to json, possibly a coincidence, but a clue I am following up.

In poking into the fpjson code I see it uses FloatToStr and TryStrToFloat which internationalize, assuming DefaultFormatSettings is initialized.

Getting to my question. It seems logical to me that json should use a standard float format that can be read regardless of location, such as output by the Str and Val functions. Is that addressed by any standard?

EDIT

I guess this answers my question:

https://www.json.org/json-en.html

I assume fpjson follows this convention by using the appropriate TFormatSettings? I'll poke around some more, but I'd appreciate confirmation if someone knows the answer.

Title: Re: A new design for a JSON Parser
Post by: BeniBela on October 15, 2020, 10:19:18 pm

Quote from: VTwin on October 15, 2020, 01:22:43 am

In poking into the fpjson code I see it uses FloatToStr and TryStrToFloat which internationalize, assuming DefaultFormatSettings is initialized.

Floating point numbers are really broken

Never use FloatToStr. Besides the format settings, it is printing 15 digit numbers, which is not enough to encode a double precisely. Use Str directly

Title: Re: A new design for a JSON Parser
Post by: Jvan on October 17, 2020, 03:55:02 am

How to get only the value of a json pair?

Code: Pascal [Select][+]

ShowMessage(myJson.Find('Data/SubData').AsJson);
 

but I get this:

Quote

"SubData":{...}

And what I want is:

Quote

{...}

Title: Re: A new design for a JSON Parser
Post by: VTwin on October 17, 2020, 05:11:13 pm

Quote from: BeniBela on October 15, 2020, 10:19:18 pm

Quote from: VTwin on October 15, 2020, 01:22:43 am

In poking into the fpjson code I see it uses FloatToStr and TryStrToFloat which internationalize, assuming DefaultFormatSettings is initialized.

Floating point numbers are really broken

Never use FloatToStr. Besides the format settings, it is printing 15 digit numbers, which is not enough to encode a double precisely. Use Str directly

Thanks for the reply. I was surprised to see FloatToStr and TryStrToFloat used in fpjson. I suspect this is causing the problem, but was having trouble pinning it down. I have been bitten by these before when trying to internationalize. I will likely go back to the hand-rolled xml code I was using previously.

Title: Re: A new design for a JSON Parser
Post by: VTwin on October 17, 2020, 08:05:42 pm

Actually in fpjson I see:

Code: Pascal [Select][+]

function TJSONString.GetAsFloat: TJSONFloat;
 
Var
  C : Integer;
 
begin
  Val(FValue,Result,C);
  If (C<>0) then
    If Not TryStrToFloat(FValue,Result) then
      Raise EConvertError.CreateFmt(SErrInvalidFloat,[FValue]);
end;

and

Code: Pascal [Select][+]

procedure TJSONString.SetAsFloat(const AValue: TJSONFloat);
begin
  FValue:=FloatToStr(AValue);
end;

So Val is called before trying TryStrToFloat, however FloatToStr is used instead of Str.

JsonTools uses StrToFloatDef and FloatToStr.

EDIT

I see that

FloatToStr(Value);

is equivalent to:

FloatToStrF(Value, ffGeneral,15, 0);

so may not be a problem, even if Str might be a better choice.

Title: Re: A new design for a JSON Parser
Post by: BeniBela on October 17, 2020, 11:20:58 pm

There is no good choice

Val/Str is also broken

If you parse '1.421085474167199E-14' with Val/TryStrToFloat you get a double that Str prints as 1.4210854741671992E-014

But that is an abbreviation, the actual value of the double is 1.42108547416719915783983492945642068425800459696706212753269937820732593536376953125E-014

Which is wrong!

Because the next smaller double is 1.421085474167198842295472841051698519566578483852570258250125334598124027252197265625E-014

Which is closer to 1.421085474167199E-14 than former double, so that is the one it should convert to

-

And FloatToStr makes an even bigger mess out of this by returning '1.4210854741672E-14'

Which is nowhere close to the doubles above

And is converted by Val and Str back to 1.4210854741672001E-014 (which is in this case correct)

Title: Re: A new design for a JSON Parser
Post by: VTwin on October 18, 2020, 05:38:32 pm

:o

Do you know of bug reports, or third party tools?

Title: Re: A new design for a JSON Parser
Post by: Hansaplast on October 19, 2020, 11:06:24 am

Quote from: VTwin on October 18, 2020, 05:38:32 pm

Do you know of bug reports, or third party tools?

For JSONTools, if that is what you are referring to, I believe you can report bugs here: JSONTools Github Issues (https://github.com/sysrpl/JsonTools/issues).

Title: Re: A new design for a JSON Parser
Post by: VTwin on October 20, 2020, 01:17:21 am

Quote from: Hansaplast on October 19, 2020, 11:06:24 am

Quote from: VTwin on October 18, 2020, 05:38:32 pm
Do you know of bug reports, or third party tools?

For JSONTools, if that is what you are referring to, I believe you can report bugs here: JSONTools Github Issues (https://github.com/sysrpl/JsonTools/issues).

I'd prefer to just use fpjson, rather than depend on an additional library.

Title: Re: A new design for a JSON Parser
Post by: BeniBela on October 20, 2020, 07:23:25 pm

Quote from: VTwin on October 18, 2020, 05:38:32 pm

Do you know of bug reports, or third party tools?

here: https://bugs.freepascal.org/view.php?id=29531

Title: Re: A new design for a JSON Parser
Post by: VTwin on October 20, 2020, 10:03:21 pm

Quote from: BeniBela on October 20, 2020, 07:23:25 pm

Quote from: VTwin on October 18, 2020, 05:38:32 pm

Do you know of bug reports, or third party tools?

here: https://bugs.freepascal.org/view.php?id=29531

Excellent! Thank you for looking into this issue.

Lazarus

Programming => General => Topic started by: sysrpl on August 26, 2019, 09:12:55 am