Recent

Author Topic: A new design for a JSON Parser  (Read 42846 times)

sysrpl

  • Sr. Member
  • ****
  • Posts: 315
    • Get Lazarus
A new design for a JSON Parser
« on: August 26, 2019, 09:12:55 am »
I know the FCL already has a capable JSON parser, but I am writing some Amazon web service interfacing projects and wanted a smaller easier to use JSON parser to assist. I've created a new design for a JSON parser that is pretty small, yet powerful.

If you're interested, I've posted the code under GPLv3 and a write up of my thought process and the workflow of using a single small class to work with JSON:

https://www.getlazarus.org/json/

Any and all feedback is welcome.

Update: Aug 1 2021

I've posted an update that fixes an issue with escaped double quote characters and adds some convenience methods. You may read more about the update at the page linked above.
« Last Edit: August 03, 2021, 08:45:51 am by sysrpl »

k1ng

  • New Member
  • *
  • Posts: 37
Re: A new design for a JSON Parser
« Reply #1 on: August 26, 2019, 10:57:01 am »
Hey,
nice work! Would be nice to see a speed comparison with LkJSON :)

sysrpl

  • Sr. Member
  • ****
  • Posts: 315
    • Get Lazarus
Re: A new design for a JSON Parser
« Reply #2 on: August 26, 2019, 11:51:17 am »
I am unsure how fast or slow it is, but I didn't design it for speed. That's not to say it's slow, but it's meant to be small, with powerful features, and just one class / unit.

With regard to speed, I am creating 1 pascal object for every node. If I wanted to make it fast I would getmem for many objects at once, and neither create nor destroy them. Instead I would put or get object memory from that pool and not heap allocation / deallocation an object for each node parsed.

That said, would it really be worth it? Do your programs spend most of their time parsing JSON? Are you writing a heavy traffic outward facing service that parses JSON frequently?

If this is the case and speed / scalability is a concern then you probably want to switch to nodejs which is optimized for heavy traffic and parallelization, and is based on JSON to boot. Many smart engineers have designed nodejs for exactly this use case.
« Last Edit: August 26, 2019, 11:53:33 am by sysrpl »

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: A new design for a JSON Parser
« Reply #3 on: August 26, 2019, 12:03:31 pm »
(the pooling functionality in the fcl-XML unit has also been designed out over time, to sensitive, maintenance wise)

k1ng

  • New Member
  • *
  • Posts: 37
Re: A new design for a JSON Parser
« Reply #4 on: August 26, 2019, 01:06:07 pm »
I am unsure how fast or slow it is, but I didn't design it for speed. That's not to say it's slow, but it's meant to be small, with powerful features, and just one class / unit.

With regard to speed, I am creating 1 pascal object for every node. If I wanted to make it fast I would getmem for many objects at once, and neither create nor destroy them. Instead I would put or get object memory from that pool and not heap allocation / deallocation an object for each node parsed.
I'm not sure how LkJSON works internally but I assume it also creates objects for each node as you refer to them via Field identifier.

Code: Pascal  [Select][+][-]
  1. js := TlkJSONObject.Create();
  2. js := TlkJSON.ParseText(jsonstr) as TlkJSONObject;
  3.  
  4. if js.Field['name'].Field['surname'].SelfType <> jsNull then
  5.   surname := String(js.Field['name'].Field['surname'].Value);

So for me it seems both are working more or less the same just with a different in usage when getting values. For the latter I'd prefer your version as one don't need some typecast. Using AsString etc is more common in recent Delphi/FPC.
It was just a suggestion because I think more users would use your library if it's also faster/comparable to LkJSON, just with a better syntax. So it'd be another pro to try your version ;)

That said, would it really be worth it? Do your programs spend most of their time parsing JSON? Are you writing a heavy traffic outward facing service that parses JSON frequently?

If this is the case and speed / scalability is a concern then you probably want to switch to nodejs which is optimized for heavy traffic and parallelization, and is based on JSON to boot. Many smart engineers have designed nodejs for exactly this use case.
No, personally I don't need much JSON parsing but others may do. E.g. if your library is 100x times slower than LkJSON (I don't know if there are any other Delphi+FPC JSON Parsers) many people wouldn't use your version as the only 'pro' would be the different syntax but as you only need to write code once...

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: A new design for a JSON Parser
« Reply #5 on: August 26, 2019, 01:15:50 pm »
Good work sysrpl.  I like a design based on simplicity of use and clear syntax.
Thank you for your effort.
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: A new design for a JSON Parser
« Reply #6 on: August 26, 2019, 01:25:35 pm »
I don't know if there are any other Delphi+FPC JSON Parsers

FPC comes with its own, fcl-json

sysrpl

  • Sr. Member
  • ****
  • Posts: 315
    • Get Lazarus
Re: A new design for a JSON Parser
« Reply #7 on: August 26, 2019, 03:54:19 pm »
king,

Code: Pascal  [Select][+][-]
  1. js := TlkJSONObject.Create();
  2. js := TlkJSON.ParseText(jsonstr) as TlkJSONObject;
  3.  
  4. if js.Field['name'].Field['surname'].SelfType <> jsNull then
  5.   surname := String(js.Field['name'].Field['surname'].Value);

The equivalent version of with my library would be:

Code: Pascal  [Select][+][-]
  1. N := TJsonNode.Create;
  2. if N.TryParse(S) and (N.Find('name/surname') <> nil) then
  3.   SurName := N.Find('name/surname').AsString;

With regards to speed, I am considering an experiment for my own curiosity. Here is how and what I would test.

1) Time parsing a large JSON structure thousands of times.
2) Remove the TJsonNode create during the parsing, internally overwriting the same node over and over again and repeat the same test.
3) Remove the internal TList and add, and repeat the test again yet again and note the time.

This should give me a good base line to understand how much time it take the FPC to parse JSON with my library, first as it is now, second as it would be with some type of object pooling, and third with a fixed size list shared among all nodes.

If the times show a marked difference in speed, then adding pooling and a shared list might be a worthwhile enhancement. Also, I may test against uLkJSON. I've look at its source code and I'll be curious to see the speed difference.

Thank you for your replies.

serbod

  • Full Member
  • ***
  • Posts: 142
Re: A new design for a JSON Parser
« Reply #8 on: August 26, 2019, 04:40:17 pm »
Another implementation of JSON serialization:

https://github.com/serbod/dbitems/blob/master/datastorage.pas
https://github.com/serbod/dbitems/blob/master/JsonStorage.pas

IDataStorage (TDataStorage) - abstract item similar to Variant, that can store any value/list/dictionary. Using as Interface allows automatic free unused items by refcount.

TDataSerializerJson - serialize/deserialize items to/from JSON.

TDataSerializerBencode - Bencode serializer/deserializer. Fast, compact and human-readable. Used in torrent files, for example.

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: A new design for a JSON Parser
« Reply #9 on: August 26, 2019, 05:54:32 pm »
My internettools can do JSON, too:

The above task could be solved as:

Code: Pascal  [Select][+][-]
  1.   xsurname := query('json($_1)/name/surname', [jsonstr]);
  2.   if not xsurname.isUndefined then
  3.     surname := xsurname.toString;
  4.  

It is the opposite of being small. If jsonstr was an url, it would be downloaded from the internet; and you can use it with same syntax for HTML

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: A new design for a JSON Parser
« Reply #10 on: August 26, 2019, 08:53:14 pm »
https://www.getlazarus.org/json/
I could not connect https because of invalid certificate. I could not connect http because OpenDNS flagged site as malware.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Leledumbo

  • Hero Member
  • *****
  • Posts: 8746
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: A new design for a JSON Parser
« Reply #11 on: August 27, 2019, 08:38:24 pm »
I see a few improvements over fcl-json interface. First is that a lot fcl-json methods accepts or returns TJSONData, the top most generic JSON value representation. This will require downcasting to actual type everytime the real value will be used. Second, you node is designed with parent node access, so moving around the tree is possible without the need to keep parent node reference. Other than those, they're pretty much equal. I just with the root node doesn't have to be explicitly created (well, a little wrapper function similar to fcl-json's GetJSON is rather easy to make).

heejit

  • Full Member
  • ***
  • Posts: 245
Re: A new design for a JSON Parser
« Reply #12 on: August 27, 2019, 10:37:22 pm »
If possible please add your library into Online package manager
it help this library available to many user.

sysrpl

  • Sr. Member
  • ****
  • Posts: 315
    • Get Lazarus
Re: A new design for a JSON Parser
« Reply #13 on: August 28, 2019, 01:36:03 am »
I appreciate the feedback

Leledumbo,

I just thought I'd explain the Find(Path) syntax, as it relates to your mentioning of the Parent node.

Code: Pascal  [Select][+][-]
  1. AnyNode.Find('/'); // returns the root node
  2. AnyNode.Find('search/for/name'); // returns a node 3 levels from the current node
  3. AnyNode.Find('/search/for/name'); // returns a node 3 levels from the root node
  4.  

There is also a NodeByName property which does not try to evaluate a path.

Code: Pascal  [Select][+][-]
  1. AnyNode.Find('/search/for/name'); // returns a node directly under the current name with a name of "/search/for/name"

In other words if S contains:

{
  "test": {
  },
  "stuff": {
    "enabled": true,
    "/search/for/name": "you've found me"
  }
}

Then ...

Code: Pascal  [Select][+][-]
  1. N := TJsonNode.Create;
  2. N.Value := S;
  3. N := N.Find('stuff');
  4. WriteLn(N.NodeByName['/search/for/name'].AsString);
  5. N.Root.Free;

Outputs:

you've found me
« Last Edit: August 28, 2019, 01:38:20 am by sysrpl »

sysrpl

  • Sr. Member
  • ****
  • Posts: 315
    • Get Lazarus
Re: A new design for a JSON Parser
« Reply #14 on: August 28, 2019, 02:00:09 am »
More usage examples:

Each node can parse / load / save JSON text at any level. That is you can compose a document like so:

Code: Pascal  [Select][+][-]
  1. N := TJsonNode.Create;
  2. N.Add('employees').LoadFromFile('employees.json');
  3. N := N.Add('contacts');
  4. N.LoadFromFile('contacts.json');
  5. N.Add('emergency').LoadFromFile('emergency.json');
  6. WriteLn(N.Value);
  7. WriteLn(N.Root.Value);

Would write out first:

{
  .. contact nodes
  "emergency": {
    .. emergency nodes
  }
}

Then write out second:

{
  "employees": {
    .. employee nodes
  },
  "contacts": {
    .. contact nodes
    "emergency": {
      .. emergency nodes
    }
  }
}

So in this way my library allows you to load, parse, or set the JSON value of any node at any level as if it were the root node. The difference being is that child nodes just append to more nodes which ultimate fall under the same root.

The only requirement is this is that the JSON for the root node must be in the form of an object {} or array []. Child node JSON can be object {}, array [], boolean true/false, null null, number 123, or string "hello world" (note the double quotes).

To use NON JSON with nodes, such as 'hello world' (single quotes), use the type safe properties:

AsObject
AsArray
AsBoolean
AsNull
AsNumber
AsString

« Last Edit: August 28, 2019, 02:56:48 am by sysrpl »

 

TinyPortal © 2005-2018