* * *

Author Topic: Running out of memory with FPJSON and  (Read 1166 times)

torbente

  • Full Member
  • ***
  • Posts: 225
Running out of memory with FPJSON and
« on: June 10, 2018, 10:49:17 pm »
Recently i checked my app in the windows task manager, and i get surprised to see that it increases it memory usage very fast (around 5 MB/Min)

I was checking around and i guess this function could be the problem:

Code: Pascal  [Select]
  1. function Read_URL_GET(url:string):string;
  2. begin
  3. with TFPHTTPClient.Create(nil) do
  4.   try
  5.     Read_URL_GET:= Get(url);
  6.     destroy; // I tried with free too
  7.   except
  8.     on E:Exception do
  9.        begin
  10.        Read_URL_GET:= 'ERROR';
  11.        destroy; // tried with free too
  12.        end;
  13.   end;
  14. end;

Or here. (dependant from the above)

Code: Pascal  [Select]
  1. function GetWebData(market):dataconfig;
  2. var
  3.   Data: TJSONData;
  4.   DataArrayItem: TJSONData;
  5.   JSON: TJSONStringType;
  6.   I: Integer;
  7.   ObjName: String;
  8.   Resultado: dataconfig;
  9. Begin
  10. market:= StringReplace(mercado,' ','',[rfReplaceAll, rfIgnoreCase]);
  11. JSON :=Read_URL_GET('https:myweb.com?symbol='+market);
  12. if JSON = 'ERROR' then
  13.    Begin
  14.    Resultado.nombredata := 'ERROR';
  15.    Resultado.valordata := 'ERROR';
  16.    GetWebData:=resultado;
  17.    exit;
  18.    end;
  19. if not IsValidJSON(JSON) then
  20.    Begin
  21.    Resultado.nombredata := 'ERROR';Resultado.valordata := 'ERROR';GetWebData:=resultado;
  22.    exit;
  23.    end;
  24. Data := GetJSON(JSON);
  25. for I := 0 to Data.Count - 1 do
  26. begin
  27.   ObjName := TJSONObject(Data).Names[I];
  28.   if Data.Items[I].JSONType = jtArray then
  29.   begin
  30.     DataArrayItem := TJSONObject(Data.Items[I]);
  31.     if (ObjName = 'pros') then Resultado.nombredata := float_changeformat(TJSONObject(DataArrayItem.Items[0]).Items[0].Value);
  32.     if (ObjName = 'cons') then Resultado.valordata := float_changeformat(TJSONObject(DataArrayItem.Items[0]).Items[0].Value);
  33.   end;
  34. end;
  35. GetWebData:=resultado;
  36. data.Free;
  37. end;

Why the memory overloads with every call to this functions?
Thanks a lot.
« Last Edit: June 11, 2018, 02:40:46 am by torbente »

jamie

  • Hero Member
  • *****
  • Posts: 786
Re: Running out of memory (Windows Task Manager)
« Reply #1 on: June 10, 2018, 11:09:55 pm »
how about running the first function a few times without doing the second one, then check for memory use..

torbente

  • Full Member
  • ***
  • Posts: 225
Re: Running out of memory (Windows Task Manager)
« Reply #2 on: June 11, 2018, 12:12:30 am »
how about running the first function a few times without doing the second one, then check for memory use..

I changed the sescond function to:

Code: Pascal  [Select]
  1. function GetWebData(market):dataconfig;
  2. var
  3.   Resultado: dataconfig;
  4. Begin
  5. Read_URL_GET('https:myweb.com?symbol='+market);
  6. resultado.nombredata := 'ERROR';
  7. resultado.valordata :='ERROR';
  8. GetWebData:=resultado;
  9. end;

A timer executes it every 2 seconds and after 15 minutes the memory usage keeps between 3,7 and 3,8 MB, so it is nothing with Read_URL_GET
Why function GetWebData(market):dataconfig gets too many memory? It is something about JSON data use, but i dont know what.

torbente

  • Full Member
  • ***
  • Posts: 225
Re: Running out of memory with FPJSON and
« Reply #3 on: June 11, 2018, 03:10:07 am »
ok, i found WHERE is the problem, but i still do not understand WHY.

I leave the functions like in the first post, but i just ommited one thing...

Code: Pascal  [Select]
  1. function GetWebData(market):dataconfig;
  2. var
  3.   Data: TJSONData;
  4.   DataArrayItem: TJSONData;
  5.   JSON: TJSONStringType;
  6.   I: Integer;
  7.   ObjName: String;
  8.   Resultado: dataconfig;
  9. Begin
  10. market:= StringReplace(market,' ','',[rfReplaceAll, rfIgnoreCase]);
  11. JSON :=Read_URL_GET('https:myweb.com?symbol='+market);
  12. if JSON = 'ERROR' then
  13.    Begin
  14.    Resultado.nombredata := 'ERROR';
  15.    Resultado.valordata := 'ERROR';
  16.    GetWebData:=resultado;
  17.    exit;
  18.    end;
  19. {if not IsValidJSON(JSON) then
  20.    Begin
  21.    Resultado.nombredata := 'ERROR';Resultado.valordata := 'ERROR';GetWebData:=resultado;
  22.    exit;
  23.    end;}
  24. Data := GetJSON(JSON);
  25. for I := 0 to Data.Count - 1 do
  26. begin
  27.   ObjName := TJSONObject(Data).Names[I];
  28.   if Data.Items[I].JSONType = jtArray then
  29.   begin
  30.     DataArrayItem := TJSONObject(Data.Items[I]);
  31.     if (ObjName = 'pros') then Resultado.nombredata := float_changeformat(TJSONObject(DataArrayItem.Items[0]).Items[0].Value);
  32.     if (ObjName = 'cons') then Resultado.valordata := float_changeformat(TJSONObject(DataArrayItem.Items[0]).Items[0].Value);
  33.   end;
  34. end;
  35. GetWebData:=resultado;
  36. data.Free;
  37. end;

Just skiping the call to IsValidJason all works perfect.

Ans this is the function which makes my app eat slowly all the memory.

Code: Pascal  [Select]
  1. function IsValidJSON (MyJSONstring:string):boolean;
  2. var
  3.   Data: TJSONData;
  4. begin
  5.    Try
  6.    Data := GetJSON(MyJSONstring);
  7.    except
  8.    On E : ejsonparser do
  9.       Begin
  10.       IsValidJSON := false;
  11.       exit;
  12.       end;
  13.    end;
  14. IsValidJSON := true;
  15. end;

 %) %) %) %)

josh

  • Hero Member
  • *****
  • Posts: 602
Re: Running out of memory with FPJSON and
« Reply #4 on: June 11, 2018, 04:01:35 am »
Hi

Without being able to test your code, and my personal dislike for exit and breaks.

I have recoded, not the most effecient way but should be enough to work out where things are going wrong, I have used an OK boolean, which you should be able to monitor easily by inserting breakpoints if something is a miss. I have not tested, to see if it compiles..

Code: Pascal  [Select]
  1. function IsValidJSON (MyJSONstring:string):boolean;
  2. var
  3.   Data: TJSONData;
  4. begin
  5.   result:=true;
  6.   Try
  7.     Data := GetJSON(MyJSONstring);
  8.   except
  9.     On E : ejsonparser do
  10.         Begin
  11.           result:=false;
  12.         end;
  13.   end;
  14. end;
  15.  
  16.  
  17. function GetWebData(market):dataconfig;
  18. var
  19.   Data: TJSONData;
  20.   DataArrayItem: TJSONData;
  21.   JSON: TJSONStringType;
  22.   I: Integer;
  23.   ObjName: String;
  24.   Resultado: dataconfig;
  25.   ok:boolean=true;
  26. Begin
  27.   market:= StringReplace(market,' ','',[rfReplaceAll, rfIgnoreCase]);
  28.   JSON :=Read_URL_GET('https:myweb.com?symbol='+market);
  29.   ok:=JSON<>'ERROR';
  30.   if not OK then  // JSON=ERROR
  31.   Begin
  32.     Resultado.nombredata := 'ERROR';
  33.     Resultado.valordata := 'ERROR';
  34.     GetWebData:=resultado;
  35.   end
  36.   else
  37.   begin // JSON IS OK
  38.     ok:=IsValidJSON(JSON); // CHECK if Valid
  39.     if Not ok then // Not Valid
  40.     begin
  41.       Resultado.nombredata := 'ERROR';
  42.       Resultado.valordata := 'ERROR';
  43.       GetWebData:=resultado;
  44.     end
  45.     else
  46.     begin // isValidJson returns OK; so lets process it.
  47.       Data := GetJSON(JSON);
  48.       for I := 0 to Data.Count - 1 do
  49.       begin
  50.         ObjName := TJSONObject(Data).Names[I];
  51.         if Data.Items[I].JSONType = jtArray then
  52.         begin
  53.           DataArrayItem := TJSONObject(Data.Items[I]);
  54.           if (ObjName = 'pros') then Resultado.nombredata := float_changeformat(TJSONObject(DataArrayItem.Items[0]).Items[0].Value);
  55.           if (ObjName = 'cons') then Resultado.valordata := float_changeformat(TJSONObject(DataArrayItem.Items[0]).Items[0].Value);
  56.         end;
  57.       end;
  58.       GetWebData:=resultado;
  59.       data.Free;
  60.     end;
  61.   end;
  62. end;
  63.  
« Last Edit: June 11, 2018, 04:06:05 am by josh »
Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

josh

  • Hero Member
  • *****
  • Posts: 602
Re: Running out of memory with FPJSON and
« Reply #5 on: June 11, 2018, 05:03:18 am »
ps. Not sure if you need to clear your json variable in your procedure/functions, if you do you could use a procedure like below to clean up before your functions/procedures finish
FreeAndNilStuff(Data);
FreeAndNilStuff(DataArrayItem);
etc.


Code: Pascal  [Select]
  1. procedure FreeAndNilStuff (var obj);
  2. var
  3.   temp:TObject = nil;
  4.   O :TObject absolute obj;
  5. begin
  6.   if O is TObject then
  7.   begin
  8.     temp:= O;
  9.   end
  10.   else
  11.   begin
  12.     freemem(O);
  13.   end;
  14.   O:= nil;
  15.   if temp <> nil then temp.free;
  16. end;            
  17.  
Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6420
Re: Running out of memory with FPJSON and
« Reply #6 on: June 11, 2018, 10:00:38 am »
I don't think that is entirely safe, Josh. The IS check on something that isn't an object might poke around in illegal memory in rare cases.

josh

  • Hero Member
  • *****
  • Posts: 602
Re: Running out of memory with FPJSON and
« Reply #7 on: June 11, 2018, 06:16:08 pm »
Hi Marcov,

Ooops sorry was not aware that the IS TObject would cause an issue.
Do yow know of a particular instance where it might fail?
Or even better do you know a way to correct the procedure so that it does not fail...
Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6420
Re: Running out of memory with FPJSON and
« Reply #8 on: June 11, 2018, 06:22:26 pm »
Hi Marcov,

Ooops sorry was not aware that the IS TObject would cause an issue.
Do yow know of a particular instance where it might fail?

As said, if obj is not an object, the result is unpredictable, and maybe could crash (it depends on if the random memory it interprets is always allocated or not, which depends on memory manager internals)

Quote
Or even better do you know a way to correct the procedure so that it does not fail...

You can't. You try to do something that requires compiletime knowledge of types.

jamie

  • Hero Member
  • *****
  • Posts: 786
Re: Running out of memory with FPJSON and
« Reply #9 on: June 11, 2018, 11:39:23 pm »
The function "IsValidJson" is not freeing the Data object before it exits normally.

Obviously "GetJSON" returns a live object if it does not fire a fault.

But the function IsValidJson is not freeing it before exit, the function only returns a Boolean....



torbente

  • Full Member
  • ***
  • Posts: 225
Re: Running out of memory with FPJSON and
« Reply #10 on: June 12, 2018, 03:53:29 pm »
The function "IsValidJson" is not freeing the Data object before it exits normally.

Obviously "GetJSON" returns a live object if it does not fire a fault.

But the function IsValidJson is not freeing it before exit, the function only returns a Boolean....

IT solves it. Thanks.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus