Recent

Author Topic: A new design for a JSON Parser  (Read 42694 times)

Awkward

  • Full Member
  • ***
  • Posts: 133
Re: A new design for a JSON Parser
« Reply #30 on: July 11, 2020, 08:31:52 am »
Yes, JSONTools is good but looks like unfinished and abandoned :(

alantelles

  • New Member
  • *
  • Posts: 21
Re: A new design for a JSON Parser
« Reply #31 on: August 13, 2020, 07:17:04 am »
I know the FCL already has a capable JSON parser, but I am writing some Amazon web service interfacing projects and wanted a smaller easier to use JSON parser to assist. I've create a new design for a JSON parser that is pretty small, yet powerful.

If your interested, I've posted the code under GPLv3 and a write up of my thought process and the workflow of using an single small class to work with JSON:

https://www.getlazarus.org/json/

Any and all feedback is welcome.

I used your parser to be the json to dictionary parser for my UltraGen  language. Thanks! it was easy to use your parser to make the conversion.

https://github.com/alantelles/ultragen

Thanx!
# compulsive coder
FPC 3.2.0, Lazarus 2.0.10

VTwin

  • Hero Member
  • *****
  • Posts: 1215
  • Former Turbo Pascal 3 user
Re: A new design for a JSON Parser
« Reply #32 on: October 15, 2020, 01:22:43 am »
I have been using json to store program preferences, recently switching from xml. I have been working with a user from Costa Rica who reports floating point errors in Windows 10. I can not reproduce the issue myself, even when internationalizing to Costa Rica on my computer.

The error seems to have started in the version that changes over from xml to json, possibly a coincidence, but a clue I am following up.

In poking into the fpjson code I see it uses FloatToStr and TryStrToFloat which internationalize, assuming DefaultFormatSettings is initialized.

Getting to my question. It seems logical to me that json should use a standard float format that can be read regardless of location, such as output by the Str and Val functions. Is that addressed by any standard?

EDIT

I guess this answers my question:

https://www.json.org/json-en.html

I assume fpjson follows this convention by using the appropriate TFormatSettings? I'll poke around some more, but I'd appreciate confirmation if someone knows the answer.
« Last Edit: October 15, 2020, 04:42:47 pm by VTwin »
“Talk is cheap. Show me the code.” -Linus Torvalds

Free Pascal Compiler 3.2.2
macOS 12.1: Lazarus 2.2.6 (64 bit Cocoa M1)
Ubuntu 18.04.3: Lazarus 2.2.6 (64 bit on VBox)
Windows 7 Pro SP1: Lazarus 2.2.6 (64 bit on VBox)

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: A new design for a JSON Parser
« Reply #33 on: October 15, 2020, 10:19:18 pm »

In poking into the fpjson code I see it uses FloatToStr and TryStrToFloat which internationalize, assuming DefaultFormatSettings is initialized.

Floating point numbers are really broken


Never use FloatToStr. Besides the format settings, it is printing 15 digit numbers, which is not enough to encode a double precisely. Use Str directly

Jvan

  • Full Member
  • ***
  • Posts: 181
Re: A new design for a JSON Parser
« Reply #34 on: October 17, 2020, 03:55:02 am »
How to get only the value of a json pair?

Code: Pascal  [Select][+][-]
  1. ShowMessage(myJson.Find('Data/SubData').AsJson);
  2.  
but I get this:
Quote
"SubData":{...}
And what I want is:
Quote
{...}

VTwin

  • Hero Member
  • *****
  • Posts: 1215
  • Former Turbo Pascal 3 user
Re: A new design for a JSON Parser
« Reply #35 on: October 17, 2020, 05:11:13 pm »

In poking into the fpjson code I see it uses FloatToStr and TryStrToFloat which internationalize, assuming DefaultFormatSettings is initialized.

Floating point numbers are really broken


Never use FloatToStr. Besides the format settings, it is printing 15 digit numbers, which is not enough to encode a double precisely. Use Str directly

Thanks for the reply. I was surprised to see FloatToStr and TryStrToFloat used in fpjson. I suspect this is causing the problem, but was having trouble pinning it down. I have been bitten by these before when trying to internationalize. I will likely go back to the hand-rolled xml code I was using previously.
“Talk is cheap. Show me the code.” -Linus Torvalds

Free Pascal Compiler 3.2.2
macOS 12.1: Lazarus 2.2.6 (64 bit Cocoa M1)
Ubuntu 18.04.3: Lazarus 2.2.6 (64 bit on VBox)
Windows 7 Pro SP1: Lazarus 2.2.6 (64 bit on VBox)

VTwin

  • Hero Member
  • *****
  • Posts: 1215
  • Former Turbo Pascal 3 user
Re: A new design for a JSON Parser
« Reply #36 on: October 17, 2020, 08:05:42 pm »
Actually in fpjson I see:

Code: Pascal  [Select][+][-]
  1. function TJSONString.GetAsFloat: TJSONFloat;
  2.  
  3. Var
  4.   C : Integer;
  5.  
  6. begin
  7.   Val(FValue,Result,C);
  8.   If (C<>0) then
  9.     If Not TryStrToFloat(FValue,Result) then
  10.       Raise EConvertError.CreateFmt(SErrInvalidFloat,[FValue]);
  11. end;

and

Code: Pascal  [Select][+][-]
  1. procedure TJSONString.SetAsFloat(const AValue: TJSONFloat);
  2. begin
  3.   FValue:=FloatToStr(AValue);
  4. end;

So Val is called before trying TryStrToFloat, however FloatToStr is used instead of Str.

JsonTools uses StrToFloatDef and FloatToStr.

EDIT

I see that

FloatToStr(Value);

is equivalent to:

FloatToStrF(Value, ffGeneral,15, 0);

so may not be a problem, even if Str might be a better choice.
« Last Edit: October 17, 2020, 08:48:10 pm by VTwin »
“Talk is cheap. Show me the code.” -Linus Torvalds

Free Pascal Compiler 3.2.2
macOS 12.1: Lazarus 2.2.6 (64 bit Cocoa M1)
Ubuntu 18.04.3: Lazarus 2.2.6 (64 bit on VBox)
Windows 7 Pro SP1: Lazarus 2.2.6 (64 bit on VBox)

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: A new design for a JSON Parser
« Reply #37 on: October 17, 2020, 11:20:58 pm »
There is no good choice

Val/Str is also broken

If you parse '1.421085474167199E-14' with Val/TryStrToFloat you get a double that Str prints as  1.4210854741671992E-014

But that is an abbreviation, the actual value of the double is  1.42108547416719915783983492945642068425800459696706212753269937820732593536376953125E-014

Which is wrong!

Because the next smaller double is 1.421085474167198842295472841051698519566578483852570258250125334598124027252197265625E-014

Which is closer to 1.421085474167199E-14 than former double, so that is the one it should convert to


-

And FloatToStr makes an even bigger mess out of this by returning '1.4210854741672E-14'

Which is nowhere close to the doubles above

And is converted by Val and Str back to 1.4210854741672001E-014 (which is in this case correct)


VTwin

  • Hero Member
  • *****
  • Posts: 1215
  • Former Turbo Pascal 3 user
Re: A new design for a JSON Parser
« Reply #38 on: October 18, 2020, 05:38:32 pm »
 :o

Do you know of bug reports, or third party tools?
“Talk is cheap. Show me the code.” -Linus Torvalds

Free Pascal Compiler 3.2.2
macOS 12.1: Lazarus 2.2.6 (64 bit Cocoa M1)
Ubuntu 18.04.3: Lazarus 2.2.6 (64 bit on VBox)
Windows 7 Pro SP1: Lazarus 2.2.6 (64 bit on VBox)

Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: A new design for a JSON Parser
« Reply #39 on: October 19, 2020, 11:06:24 am »
Do you know of bug reports, or third party tools?


For JSONTools, if that is what you are referring to, I believe you can report bugs here: JSONTools Github Issues.

VTwin

  • Hero Member
  • *****
  • Posts: 1215
  • Former Turbo Pascal 3 user
Re: A new design for a JSON Parser
« Reply #40 on: October 20, 2020, 01:17:21 am »
Do you know of bug reports, or third party tools?


For JSONTools, if that is what you are referring to, I believe you can report bugs here: JSONTools Github Issues.

I'd prefer to just use fpjson, rather than depend on an additional library.
“Talk is cheap. Show me the code.” -Linus Torvalds

Free Pascal Compiler 3.2.2
macOS 12.1: Lazarus 2.2.6 (64 bit Cocoa M1)
Ubuntu 18.04.3: Lazarus 2.2.6 (64 bit on VBox)
Windows 7 Pro SP1: Lazarus 2.2.6 (64 bit on VBox)

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: A new design for a JSON Parser
« Reply #41 on: October 20, 2020, 07:23:25 pm »

VTwin

  • Hero Member
  • *****
  • Posts: 1215
  • Former Turbo Pascal 3 user
Re: A new design for a JSON Parser
« Reply #42 on: October 20, 2020, 10:03:21 pm »

Do you know of bug reports, or third party tools?

here: https://bugs.freepascal.org/view.php?id=29531

Excellent! Thank you for looking into this issue.
“Talk is cheap. Show me the code.” -Linus Torvalds

Free Pascal Compiler 3.2.2
macOS 12.1: Lazarus 2.2.6 (64 bit Cocoa M1)
Ubuntu 18.04.3: Lazarus 2.2.6 (64 bit on VBox)
Windows 7 Pro SP1: Lazarus 2.2.6 (64 bit on VBox)

abouchez

  • Full Member
  • ***
  • Posts: 110
    • Synopse
Re: A new design for a JSON Parser
« Reply #43 on: July 25, 2021, 06:49:39 pm »
If you want a fast JSON parser for FPC, you may try what mORMot 2 offers.

Some numbers, parsing a JSON array of 8000 objects, for a bit more than 1MB:

Code: [Select]
  - JSON benchmark: 100,267 assertions passed  1.31s
     IsValidUtf8() in 16.63ms, 1.1 GB/s
     IsValidJson(RawUtf8) in 24.78ms, 790.8 MB/s
     IsValidJson(PUtf8Char) in 23.22ms, 843.9 MB/s
     JsonArrayCount(P) in 23.26ms, 842.7 MB/s
     JsonArrayCount(P,PMax) in 22.74ms, 862 MB/s
     JsonObjectPropCount() in 9.28ms, 1.1 GB/s
     TDocVariant in 140.43ms, 139.6 MB/s
     TDocVariant dvoInternNames in 156.73ms, 125 MB/s
     TOrmTableJson GetJsonValues in 24.98ms, 345.1 MB/s
     TOrmTableJson expanded in 37.36ms, 524.7 MB/s
     TOrmTableJson not expanded in 20.96ms, 411.2 MB/s
     fpjson in 810.40ms, 10.6 MB/s
In short, mORMot 2 JSON parser is from 13 times to 50 times faster than fpjson - and I guess JSON tools.
« Last Edit: July 25, 2021, 06:53:26 pm by abouchez »

avk

  • Hero Member
  • *****
  • Posts: 752
Re: A new design for a JSON Parser
« Reply #44 on: July 26, 2021, 08:12:00 am »
Have you tried your parser with this test suite?

 

TinyPortal © 2005-2018