[SOLVED] Reading JSON data from a file

[maurobio]

Dear ALL,

I have a very simple JSON file as follows:

Code: Pascal  [Select][+][-]

1. {
2.   "address1": "Departamento de Botanica",
3.   "address2": "CP 6109",
4.   "author": "J. Y. Tamashiro, R. Rodrigues, G. J. Shepherd",
5.   "city": "Campinas",
6.   "country": "Brasil",
7.   "dataset1": "Flora de Santa Genebra.xls",
8.   "dataset2": "",
9.   "datetime": "2014-11-21T22:25:41.406Z",
10.   "description": "Levantamento da Reserva Municipal de Santa Genebra, Campinas",
11.   "elevation": "750",
12.   "email": "george@unicamp.br",
13.   "fax": "",
14.   "funding": "FAPESP",
15.   "institution": "UNICAMP",
16.   "latitude": "-23.185",
17.   "locality": "Serra de Santa Genebra",
18.   "longitude": "-46.898",
19.   "method": "Parcela",
20.   "phone": "",
21.   "province": "Campinas",
22.   "role": "Professor",
23.   "size": "10 x 10 m",
24.   "state": "Sao Paulo",
25.   "title": "Flora de Santa Genebra",
26.   "uf": "SP",
27.   "website": "www.unicamp.br",
28.   "zip": "13083-970"
29. }
30.  

I want to read and parse this file using the FPC routines for handling JSON data. However, despite finding plenty of examples of getting JSON data from webservers and internal constant strings, I have not been able to find just a single example of reading JSON data from a text file!

Here is my sample code, which is not working:

Code: Pascal  [Select][+][-]

1. program testJSON;
2.  
3. {$APPTYPE CONSOLE}
4. {$mode objfpc}{$H+}
5.  
6. uses
7.     Classes, SysUtils, JSONParser, fpJSON;
8.        
9. var
10.   rawJSON: string;
11.   Data: TJSONArray;  
12.   DataArrayItem: TJSONObject;
13.   i: integer;
14.   title, author: string;
15.  
16. function ReadStream(fnam: string): string;
17. var
18.   strm: TFileStream;
19.   n: longint;
20.   txt: string;
21. begin
22.   txt := '';
23.   strm := TFileStream.Create(fnam, fmOpenRead or fmShareDenyWrite);
24.   try
25.     n := strm.Size;
26.     SetLength(txt, n);
27.     strm.Read(txt[1], n);
28.   finally
29.     strm.Free;
30.   end;
31.   Result := txt;
32. end;
33.        
34. begin
35.   rawJSON := ReadStream('Data.json');
36.   Data := TJSONArray(GetJSON(rawJSON));
37.   for i := 1 to Data.Count - 1 do
38.   begin
39.     DataArrayItem := Data.Objects[i];
40.     title :=  DataArrayItem['title'].AsString;
41.     author :=  DataArrayItem['author'].AsString;
42.     WriteLn(title + ',' + author);
43.   end;
44. end.
45.  

   

Could anyone give me a hand?

Thanks in advance!

With best wishes,

With best regards,

[Gustavo 'Gus' Carreno]

Hey Maurobio,

First of all let me criticise that piece of JSON:

• All fields should not be of type STRING!!
• All fields that have an empty string, should be NULL unless the empty string is in itself valuable
• All double/float fields should not be a STRING, please unquote them
• All integer fields should not be a STRING, please unquote them

This is not me bashing that piece of JSON, I'm just saying that you're wasting the capabilities of JSON when you make everything a string.
AND of course, if this is JSON that you consume from a 3rd Party, please create an ISSUE on their customer support platform making them aware of it.

Well, now that I've got that out of my chest, here's some code that will read your file and then print the values(Full project attached).

I've included 2 ways of doing it:

• When you know the key name of the fields
• When you iterate through the key/value pair of the object

Code: Pascal  [Select][+][-]

1. program ReadJSON;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   Classes, SysUtils, fpjson, jsonparser;
7.  
8. const
9.   cFilename = 'input.json';
10.  
11. var
12.   fsJSONFile: TFileStream;
13.   jJSONData, jJSONObjectField: TJSONData;
14.   jJSONObject: TJSONObject;
15.   index: Integer;
16.  
17. begin
18.   WriteLN('Example of JSON reader for maurbio v0.1');
19.   WriteLN;
20.   if FileExists(cFilename) then
21.   begin
22.     fsJSONFile:= TFileStream.Create(cFilename, fmOpenRead or fmShareDenyWrite);
23.     try
24.       jJSONData:= GetJSON(fsJSONFile);
25.       try
26.         if jJSONData.JSONType = jtObject then
27.         begin
28.           jJSONObject:= TJSONObject(jJSONData);
29.  
30.           // By knowing the fields in advance
31.           WriteLN('--- When you know the fields: ---');
32.           WriteLN('address1: ':15, jJSONObject.Strings['address1']);
33.           WriteLN('address2: ':15, jJSONObject.Strings['address2']);
34.           WriteLN('author: ':15, jJSONObject.Strings['author']);
35.           WriteLN('city: ':15, jJSONObject.Strings['city']);
36.           WriteLN('country: ':15, jJSONObject.Strings['country']);
37.           WriteLN('dataset1: ':15, jJSONObject.Strings['dataset1']);
38.           WriteLN('dataset2: ':15, jJSONObject.Strings['dataset2']);
39.           WriteLN('datetime: ':15, jJSONObject.Strings['datetime']);
40.           WriteLN('description: ':15, jJSONObject.Strings['description']);
41.           WriteLN('elevation: ':15, jJSONObject.Strings['elevation']);
42.           WriteLN('email: ':15, jJSONObject.Strings['email']);
43.           WriteLN('fax: ':15, jJSONObject.Strings['fax']);
44.           WriteLN('funding: ':15, jJSONObject.Strings['funding']);
45.           WriteLN('institution: ':15, jJSONObject.Strings['institution']);
46.           WriteLN('latitude: ':15, jJSONObject.Strings['latitude']);
47.           WriteLN('locality: ':15, jJSONObject.Strings['locality']);
48.           WriteLN('longitude: ':15, jJSONObject.Strings['longitude']);
49.           WriteLN('method: ':15, jJSONObject.Strings['method']);
50.           WriteLN('phone: ':15, jJSONObject.Strings['phone']);
51.           WriteLN('province: ':15, jJSONObject.Strings['province']);
52.           WriteLN('role: ':15, jJSONObject.Strings['role']);
53.           WriteLN('size: ':15, jJSONObject.Strings['size']);
54.           WriteLN('state: ':15, jJSONObject.Strings['state']);
55.           WriteLN('title: ':15, jJSONObject.Strings['title']);
56.           WriteLN('uf: ':15, jJSONObject.Strings['uf']);
57.           WriteLN('website: ':15, jJSONObject.Strings['website']);
58.           WriteLN('zip: ':15, jJSONObject.Strings['zip']);
59.           WriteLN;
60.  
61.           // By key/value pair
62.           WriteLN('--- When you iterate through fields: ---');
63.           for index:= 0 to Pred(jJSONObject.Count) do
64.           begin
65.             jJSONObjectField:= jJSONObject.Items[index];
66.             WriteLN(jJSONObject.Names[index]+': ':15, jJSONObjectField.AsString);
67.           end;
68.         end
69.         else
70.         begin
71.           WriteLN('Was expecting that the JSON contained a single JSON Object.');
72.         end;
73.       finally
74.         // Do NOT forget to free this object or you'll have leaks
75.         // Since this object contains all the above objects we used
76.         // no need to free those, just the overall _parent_.
77.         jJSONData.Free;
78.       end;
79.     finally
80.       fsJSONFile.Free;
81.     end;
82.   end
83.   else
84.   begin
85.     WriteLN('Cannot find file ' + cFilename);
86.   end;
87. end.

Grande abraço,
Gus

[Jurassic Pork]

hello,
you can use also findPath function of TJsonData :

Code: Pascal  [Select][+][-]

1.  var
2. //...
3. J: TJSonData;
4. //...
5.   J := GetJSON(rawJSON);
6.   title := J.FindPath('title').AsString;
7.   author :=  J.FindPath('author').AsString;
8.   writeln(title + ',' + author);
9.   readln();      
10.  

Friendly, J.P

[Gustavo 'Gus' Carreno]

Hey JP,

you can use also findPath function of TJsonData :

Thank you very much for that cuz I always forget about this option!!

And the good thing about this option is that if the path is not found then it returns nil, and if it does you can avoid an Exception when you use any other TJSONObject member function/procedure/arrays.

So thank you JP for mentioning this

Cheers,
Gus

[maurobio]

@Gustavo,

First of all, thank you very much for your criticism of my piece of JSON. It is quite helpful and I will take it into account in my application, where I do intend to use simple JSON like that to store project metadata (with input/edit by users controlled by means of an appropriate form). Anyway, I used that piece just as a sample for presenting my most immediate problem, which was reading the JSON file from disk. Of course, thanks for your very illustrative code, which allowed me to better understanding the issues involved in handling JSON fields.

... and @Jurassic Pork,

Thanks a lot for your tip, which is simple and elegant!

More soon... 

With best wishes,

[maurobio]

Dear Wizards,

However, the simple solution I tried (see the code below) still does not work.... 

Code: Pascal  [Select][+][-]

1. program testJSON;
2.  
3. {$APPTYPE CONSOLE}
4. {$mode objfpc}{$H+}
5.  
6. uses
7.     Classes, SysUtils, JSONParser, fpJSON;
8.        
9. var
10.   JSONFile: TFileStream;
11.   JSONData: TJSONData;
12.   title, author: string;
13.  
14. begin
15.   JSONFile := TFileStream.Create('Data.json', fmOpenRead or fmShareDenyWrite);
16.   JSONData := GetJSON(JSONFile);
17.   title := JSONData.FindPath('title').AsString;
18.   author := JSONData.FindPath('author').AsString;
19.   WriteLn(title + ',' + author);
20.   JSONData.Free;
21.   JSONFile.Free;
22. end.

   
It gives the following error (because when the file is read, an alien character is inserted in the beginning of the data stream):

Code: Pascal  [Select][+][-]

1. An unhandled exception occurred at $0042CAF4:
2. EScannerError: Invalid character at line 1, pos 0: 'ï'
3.   $0042CAF4
4.   $0042D7B1
5.   $0042CAA5
6.   $00425AE1
7.   $004269A6

Any hints?

With best wishes,

[eny]

It's probably the byte order mark of the file.
https://en.wikipedia.org/wiki/Byte_order_mark

[Jurassic Pork]

hello,
if you have a  file editor like notepad++ check the encoding of the file -> if it is utf8 with bom you have extra characters at the beginning of the file.
or use my json data file (attachment)   UTF8 without Bom

Friendly, J.P

[maurobio]

Dear @Jurassic Pork,

Yes, that was it! I changed the codification to UTF8 and it now works fine!

Thanks a lot.

With best wishes,

[eny]

You're welcome...

[winni]

Hi!

Have a look in the unit lconvEncoding:

Code: Pascal  [Select][+][-]

1. const
2.   UTF8BOM = #$EF#$BB#$BF;
3.   UTF16BEBOM = #$FE#$FF;
4.   UTF16LEBOM = #$FF#$FE;
5.   UTF32BEBOM = #0#0#$FE#$FF;
6.   UTF32LEBOM = #$FE#$FF#0#0;
7. ...
8. function UTF8BOMToUTF8(const s: string): string;
9. ...
10.  

Winni

[maurobio]

@Winni,

Thanks for this much valuable tip! By using that function I can avoid problems with the file encoding, without the need to edit it beforehand.

With best wishes,