Recent

Author Topic: [SOLVED] Reading JSON data from a file  (Read 2396 times)

maurobio

  • Sr. Member
  • ****
  • Posts: 301
  • Ecology is everything.
    • GitHub
[SOLVED] Reading JSON data from a file
« on: August 27, 2021, 12:48:25 am »
Dear ALL,

I have a very simple JSON file as follows:

Code: Pascal  [Select][+][-]
  1. {
  2.   "address1": "Departamento de Botanica",
  3.   "address2": "CP 6109",
  4.   "author": "J. Y. Tamashiro, R. Rodrigues, G. J. Shepherd",
  5.   "city": "Campinas",
  6.   "country": "Brasil",
  7.   "dataset1": "Flora de Santa Genebra.xls",
  8.   "dataset2": "",
  9.   "datetime": "2014-11-21T22:25:41.406Z",
  10.   "description": "Levantamento da Reserva Municipal de Santa Genebra, Campinas",
  11.   "elevation": "750",
  12.   "email": "george@unicamp.br",
  13.   "fax": "",
  14.   "funding": "FAPESP",
  15.   "institution": "UNICAMP",
  16.   "latitude": "-23.185",
  17.   "locality": "Serra de Santa Genebra",
  18.   "longitude": "-46.898",
  19.   "method": "Parcela",
  20.   "phone": "",
  21.   "province": "Campinas",
  22.   "role": "Professor",
  23.   "size": "10 x 10 m",
  24.   "state": "Sao Paulo",
  25.   "title": "Flora de Santa Genebra",
  26.   "uf": "SP",
  27.   "website": "www.unicamp.br",
  28.   "zip": "13083-970"
  29. }
  30.  

I want to read and parse this file using the FPC routines for handling JSON data. However, despite finding plenty of examples of getting JSON data from webservers and internal constant strings, I have not been able to find just a single example of reading JSON data from a text file!

Here is my sample code, which is not working:

Code: Pascal  [Select][+][-]
  1. program testJSON;
  2.  
  3. {$APPTYPE CONSOLE}
  4. {$mode objfpc}{$H+}
  5.  
  6. uses
  7.     Classes, SysUtils, JSONParser, fpJSON;
  8.        
  9. var
  10.   rawJSON: string;
  11.   Data: TJSONArray;  
  12.   DataArrayItem: TJSONObject;
  13.   i: integer;
  14.   title, author: string;
  15.  
  16. function ReadStream(fnam: string): string;
  17. var
  18.   strm: TFileStream;
  19.   n: longint;
  20.   txt: string;
  21. begin
  22.   txt := '';
  23.   strm := TFileStream.Create(fnam, fmOpenRead or fmShareDenyWrite);
  24.   try
  25.     n := strm.Size;
  26.     SetLength(txt, n);
  27.     strm.Read(txt[1], n);
  28.   finally
  29.     strm.Free;
  30.   end;
  31.   Result := txt;
  32. end;
  33.        
  34. begin
  35.   rawJSON := ReadStream('Data.json');
  36.   Data := TJSONArray(GetJSON(rawJSON));
  37.   for i := 1 to Data.Count - 1 do
  38.   begin
  39.     DataArrayItem := Data.Objects[i];
  40.     title :=  DataArrayItem['title'].AsString;
  41.     author :=  DataArrayItem['author'].AsString;
  42.     WriteLn(title + ',' + author);
  43.   end;
  44. end.
  45.  
   

Could anyone give me a hand?

Thanks in advance!

With best wishes,

With best regards,
« Last Edit: August 27, 2021, 01:43:59 pm by maurobio »
UCSD Pascal / Burroughs 6700 / Master Control Program
Lazarus 2.0.12 - FPC 3.2.0 on GNU/Linux Mint 19, GNU/Linux Lubuntu 18.04, Windows XP SP3, Windows 7 Professional, Windows 10 Home

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 757
  • Professional amateur ;-P
Re: Reading JSON data from a file
« Reply #1 on: August 27, 2021, 01:52:45 am »
Hey Maurobio,

First of all let me criticise that piece of JSON:
  • All fields should not be of type STRING!!
  • All fields that have an empty string, should be NULL unless the empty string is in itself valuable
  • All double/float fields should not be a STRING, please unquote them
  • All integer fields should not be a STRING, please unquote them

This is not me bashing that piece of JSON, I'm just saying that you're wasting the capabilities of JSON when you make everything a string.
AND of course, if this is JSON that you consume from a 3rd Party, please create an ISSUE on their customer support platform making them aware of it.

Well, now that I've got that out of my chest, here's some code that will read your file and then print the values(Full project attached).

I've included 2 ways of doing it:
  • When you know the key name of the fields
  • When you iterate through the key/value pair of the object

Code: Pascal  [Select][+][-]
  1. program ReadJSON;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   Classes, SysUtils, fpjson, jsonparser;
  7.  
  8. const
  9.   cFilename = 'input.json';
  10.  
  11. var
  12.   fsJSONFile: TFileStream;
  13.   jJSONData, jJSONObjectField: TJSONData;
  14.   jJSONObject: TJSONObject;
  15.   index: Integer;
  16.  
  17. begin
  18.   WriteLN('Example of JSON reader for maurbio v0.1');
  19.   WriteLN;
  20.   if FileExists(cFilename) then
  21.   begin
  22.     fsJSONFile:= TFileStream.Create(cFilename, fmOpenRead or fmShareDenyWrite);
  23.     try
  24.       jJSONData:= GetJSON(fsJSONFile);
  25.       try
  26.         if jJSONData.JSONType = jtObject then
  27.         begin
  28.           jJSONObject:= TJSONObject(jJSONData);
  29.  
  30.           // By knowing the fields in advance
  31.           WriteLN('--- When you know the fields: ---');
  32.           WriteLN('address1: ':15, jJSONObject.Strings['address1']);
  33.           WriteLN('address2: ':15, jJSONObject.Strings['address2']);
  34.           WriteLN('author: ':15, jJSONObject.Strings['author']);
  35.           WriteLN('city: ':15, jJSONObject.Strings['city']);
  36.           WriteLN('country: ':15, jJSONObject.Strings['country']);
  37.           WriteLN('dataset1: ':15, jJSONObject.Strings['dataset1']);
  38.           WriteLN('dataset2: ':15, jJSONObject.Strings['dataset2']);
  39.           WriteLN('datetime: ':15, jJSONObject.Strings['datetime']);
  40.           WriteLN('description: ':15, jJSONObject.Strings['description']);
  41.           WriteLN('elevation: ':15, jJSONObject.Strings['elevation']);
  42.           WriteLN('email: ':15, jJSONObject.Strings['email']);
  43.           WriteLN('fax: ':15, jJSONObject.Strings['fax']);
  44.           WriteLN('funding: ':15, jJSONObject.Strings['funding']);
  45.           WriteLN('institution: ':15, jJSONObject.Strings['institution']);
  46.           WriteLN('latitude: ':15, jJSONObject.Strings['latitude']);
  47.           WriteLN('locality: ':15, jJSONObject.Strings['locality']);
  48.           WriteLN('longitude: ':15, jJSONObject.Strings['longitude']);
  49.           WriteLN('method: ':15, jJSONObject.Strings['method']);
  50.           WriteLN('phone: ':15, jJSONObject.Strings['phone']);
  51.           WriteLN('province: ':15, jJSONObject.Strings['province']);
  52.           WriteLN('role: ':15, jJSONObject.Strings['role']);
  53.           WriteLN('size: ':15, jJSONObject.Strings['size']);
  54.           WriteLN('state: ':15, jJSONObject.Strings['state']);
  55.           WriteLN('title: ':15, jJSONObject.Strings['title']);
  56.           WriteLN('uf: ':15, jJSONObject.Strings['uf']);
  57.           WriteLN('website: ':15, jJSONObject.Strings['website']);
  58.           WriteLN('zip: ':15, jJSONObject.Strings['zip']);
  59.           WriteLN;
  60.  
  61.           // By key/value pair
  62.           WriteLN('--- When you iterate through fields: ---');
  63.           for index:= 0 to Pred(jJSONObject.Count) do
  64.           begin
  65.             jJSONObjectField:= jJSONObject.Items[index];
  66.             WriteLN(jJSONObject.Names[index]+': ':15, jJSONObjectField.AsString);
  67.           end;
  68.         end
  69.         else
  70.         begin
  71.           WriteLN('Was expecting that the JSON contained a single JSON Object.');
  72.         end;
  73.       finally
  74.         // Do NOT forget to free this object or you'll have leaks
  75.         // Since this object contains all the above objects we used
  76.         // no need to free those, just the overall _parent_.
  77.         jJSONData.Free;
  78.       end;
  79.     finally
  80.       fsJSONFile.Free;
  81.     end;
  82.   end
  83.   else
  84.   begin
  85.     WriteLN('Cannot find file ' + cFilename);
  86.   end;
  87. end.

Grande abraço,
Gus
« Last Edit: August 27, 2021, 02:02:18 am by Gustavo 'Gus' Carreno »
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.04 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.04 64b Dark Theme
http://github.com/gcarreno

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1053
Re: Reading JSON data from a file
« Reply #2 on: August 27, 2021, 02:16:53 am »
hello,
you can use also findPath function of TJsonData :
Code: Pascal  [Select][+][-]
  1.  var
  2. //...
  3. J: TJSonData;
  4. //...
  5.   J := GetJSON(rawJSON);
  6.   title := J.FindPath('title').AsString;
  7.   author :=  J.FindPath('author').AsString;
  8.   writeln(title + ',' + author);
  9.   readln();      
  10.  

Friendly, J.P
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 757
  • Professional amateur ;-P
Re: Reading JSON data from a file
« Reply #3 on: August 27, 2021, 02:48:07 am »
Hey JP,

you can use also findPath function of TJsonData :

Thank you very much for that cuz I always forget about this option!!

And the good thing about this option is that if the path is not found then it returns nil, and if it does you can avoid an Exception when you use any other TJSONObject member function/procedure/arrays.

So thank you JP for mentioning this ;)

Cheers,
Gus
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.04 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.04 64b Dark Theme
http://github.com/gcarreno

maurobio

  • Sr. Member
  • ****
  • Posts: 301
  • Ecology is everything.
    • GitHub
Re: Reading JSON data from a file
« Reply #4 on: August 27, 2021, 12:45:56 pm »
@Gustavo,

First of all, thank you very much for your criticism of my piece of JSON. It is quite helpful and I will take it into account in my application, where I do intend to use simple JSON like that to store project metadata (with input/edit by users controlled by means of an appropriate form). Anyway, I used that piece just as a sample for presenting my most immediate problem, which was reading the JSON file from disk. Of course, thanks for your very illustrative code, which allowed me to better understanding the issues involved in handling JSON fields.

... and @Jurassic Pork,

Thanks a lot for your tip, which is simple and elegant!

More soon...  ;)

With best wishes,
UCSD Pascal / Burroughs 6700 / Master Control Program
Lazarus 2.0.12 - FPC 3.2.0 on GNU/Linux Mint 19, GNU/Linux Lubuntu 18.04, Windows XP SP3, Windows 7 Professional, Windows 10 Home

maurobio

  • Sr. Member
  • ****
  • Posts: 301
  • Ecology is everything.
    • GitHub
Re: Reading JSON data from a file
« Reply #5 on: August 27, 2021, 01:22:14 pm »
Dear Wizards,

However, the simple solution I tried (see the code below) still does not work....  :-\

Code: Pascal  [Select][+][-]
  1. program testJSON;
  2.  
  3. {$APPTYPE CONSOLE}
  4. {$mode objfpc}{$H+}
  5.  
  6. uses
  7.     Classes, SysUtils, JSONParser, fpJSON;
  8.        
  9. var
  10.   JSONFile: TFileStream;
  11.   JSONData: TJSONData;
  12.   title, author: string;
  13.  
  14. begin
  15.   JSONFile := TFileStream.Create('Data.json', fmOpenRead or fmShareDenyWrite);
  16.   JSONData := GetJSON(JSONFile);
  17.   title := JSONData.FindPath('title').AsString;
  18.   author := JSONData.FindPath('author').AsString;
  19.   WriteLn(title + ',' + author);
  20.   JSONData.Free;
  21.   JSONFile.Free;
  22. end.
   
It gives the following error (because when the file is read, an alien character is inserted in the beginning of the data stream):

Code: Pascal  [Select][+][-]
  1. An unhandled exception occurred at $0042CAF4:
  2. EScannerError: Invalid character at line 1, pos 0: 'ï'
  3.   $0042CAF4
  4.   $0042D7B1
  5.   $0042CAA5
  6.   $00425AE1
  7.   $004269A6

Any hints?

With best wishes,
« Last Edit: August 27, 2021, 01:30:16 pm by maurobio »
UCSD Pascal / Burroughs 6700 / Master Control Program
Lazarus 2.0.12 - FPC 3.2.0 on GNU/Linux Mint 19, GNU/Linux Lubuntu 18.04, Windows XP SP3, Windows 7 Professional, Windows 10 Home

eny

  • Hero Member
  • *****
  • Posts: 1628
Re: Reading JSON data from a file
« Reply #6 on: August 27, 2021, 01:35:38 pm »
It's probably the byte order mark of the file.
https://en.wikipedia.org/wiki/Byte_order_mark
All posts based on: Win10 (Win64); Lazarus 2.0.10 'stable' (x64) unless specified otherwise...

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1053
Re: Reading JSON data from a file
« Reply #7 on: August 27, 2021, 01:36:48 pm »
hello,
if you have a  file editor like notepad++ check the encoding of the file -> if it is utf8 with bom you have extra characters at the beginning of the file.
or use my json data file (attachment)   UTF8 without Bom

Friendly, J.P
« Last Edit: August 27, 2021, 01:42:51 pm by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

maurobio

  • Sr. Member
  • ****
  • Posts: 301
  • Ecology is everything.
    • GitHub
Re: Reading JSON data from a file
« Reply #8 on: August 27, 2021, 01:43:31 pm »
Dear @Jurassic Pork,

Yes, that was it! I changed the codification to UTF8 and it now works fine!

Thanks a lot.

With best wishes,
UCSD Pascal / Burroughs 6700 / Master Control Program
Lazarus 2.0.12 - FPC 3.2.0 on GNU/Linux Mint 19, GNU/Linux Lubuntu 18.04, Windows XP SP3, Windows 7 Professional, Windows 10 Home

eny

  • Hero Member
  • *****
  • Posts: 1628
Re: [SOLVED] Reading JSON data from a file
« Reply #9 on: August 27, 2021, 01:45:12 pm »
You're welcome...
All posts based on: Win10 (Win64); Lazarus 2.0.10 'stable' (x64) unless specified otherwise...

winni

  • Hero Member
  • *****
  • Posts: 2709
Re: [SOLVED] Reading JSON data from a file
« Reply #10 on: August 27, 2021, 01:48:08 pm »
Hi!

Have a look in the unit lconvEncoding:

Code: Pascal  [Select][+][-]
  1. const
  2.   UTF8BOM = #$EF#$BB#$BF;
  3.   UTF16BEBOM = #$FE#$FF;
  4.   UTF16LEBOM = #$FF#$FE;
  5.   UTF32BEBOM = #0#0#$FE#$FF;
  6.   UTF32LEBOM = #$FE#$FF#0#0;
  7. ...
  8. function UTF8BOMToUTF8(const s: string): string;
  9. ...
  10.  

Winni


maurobio

  • Sr. Member
  • ****
  • Posts: 301
  • Ecology is everything.
    • GitHub
Re: [SOLVED] Reading JSON data from a file
« Reply #11 on: August 27, 2021, 04:03:07 pm »
@Winni,

Thanks for this much valuable tip! By using that function I can avoid problems with the file encoding, without the need to edit it beforehand.

With best wishes,
UCSD Pascal / Burroughs 6700 / Master Control Program
Lazarus 2.0.12 - FPC 3.2.0 on GNU/Linux Mint 19, GNU/Linux Lubuntu 18.04, Windows XP SP3, Windows 7 Professional, Windows 10 Home

 

TinyPortal © 2005-2018