Recent

Author Topic: Memory leak converting CRC characters (Solved)  (Read 7795 times)

jdp

  • Full Member
  • ***
  • Posts: 144
Memory leak converting CRC characters (Solved)
« on: July 23, 2016, 12:45:30 am »
Hi all.

We have another memory leak. We are busy building an app for the use with a Solar Inverter running on the Raspberry Pi. We have been working with Alfred getting the USB code sorted out. He has been very helpful.

So we send the Inverter a command and it will respond with a string. The string has some CRC characters at the  end. If we convert these chars to ascii we get a major memory leak.

What I have done now to get around it is not to convert those from hex into ascii. It solves the problem but I want to know if there is a better way and if Lazarus can se these character the correct way without the memory growing.

Here is the code that works. If the data are outside of the values I just replace it with an F for now.

Code: Pascal  [Select][+][-]
  1. procedure TForm1.ShowRead1(HidDev: TJvHidDevice; ReportID: Byte;
  2.   const Data: Pointer; Size: Word);
  3.  var
  4.  
  5.    I: integer;
  6.    str: String;
  7.    HexDecimalValue : LongInt;
  8.  
  9. begin
  10.  
  11. str := '';
  12.  
  13.   for I := 0 to Size - 1 do
  14.       begin
  15.  
  16.         str := Format('%.2x', [Cardinal(PChar(Data)[I])]);
  17.         HexDecimalValue := Hex2Dec(Str);
  18.         if (HexDecimalValue > 33) and (HexDecimalValue < 125) then
  19.         begin
  20.            str := PansiChar(Data)[I];
  21.         end
  22.         else
  23.         begin
  24.            str := 'F';
  25.         end;
  26.  
  27.         DecodedString := DecodedString + str;
  28.  
  29.       end;

if we use this line of code on the CRC characters then we get a memory leak.

Code: Pascal  [Select][+][-]
  1. str := PansiChar(Data)[I];

Any help or idea ?

Regards.
« Last Edit: July 24, 2016, 09:54:39 pm by jdp »

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Memory leak converting CRC characters
« Reply #1 on: July 24, 2016, 12:00:24 am »
if we use this line of code on the CRC characters then we get a memory leak.
Code: Pascal  [Select][+][-]
  1. str := PansiChar(Data)[I];
Why would you want to put a CRC-code into a string. You str is a string (which would be UTF-8) but if you're going to but any random Data-byte into it you might even end up with an invalid UTF-8 string (i.e. invalid code-point in UTF-8).

What's wrong with the code that works? You could save the CRC to another integer or something or even convert the Data-byte to a hex-code in the string.
Test$00$12 (or something)

(or convert them to proper UTF-8 characters but they may be nonsense or unreadable anyway.)


jdp

  • Full Member
  • ***
  • Posts: 144
Re: Memory leak converting CRC characters
« Reply #2 on: July 24, 2016, 07:26:25 am »
Hi rvk.

Thanks for the reply.
The data comes back from the inverter as a sting of bytes. It is all valid ascii from 0 to 256.Tthis code converts the byte correctly but we then see the memory go up . If I exclude them the memory does not go up. We dont need the chars but I am just wondering if the is another way to convert hex values to ascii and not get the memory problem.

DonAlfredo

  • Hero Member
  • *****
  • Posts: 1739
Re: Memory leak converting CRC characters
« Reply #3 on: July 24, 2016, 08:37:41 am »
Perhaps this works ?
Code: Pascal  [Select][+][-]
  1. var
  2.   .....
  3.   str2:ansichar;
  4. begin
  5.   .....
  6.   str2 := PansiChar(Data)[I];
  7.   .....
  8.   DecodedString := DecodedString + str2;

Define DecodedString also as ansistring !?

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Memory leak converting CRC characters
« Reply #4 on: July 24, 2016, 10:00:00 am »
You could also try
Code: Pascal  [Select][+][-]
  1. str := AnsiToUTF8(PansiChar(Data)[I]);

But I don't understand something:
Code: Pascal  [Select][+][-]
  1.   for I := 0 to Size - 1 do
  2.       begin
  3.         str := Format('%.2x', [Cardinal(PChar(Data)[I])]);
  4.         HexDecimalValue := Hex2Dec(Str);
  5.         if (HexDecimalValue > 33) and (HexDecimalValue < 125) then
  6.         begin
  7.  
You have a datastream with what? Bytes, characters, cardinals ??
You convert data[0] to a hex value and check if this Hexvalue is >33 and <125 and add the data[0] to the string. Why not just check pByte(Data)[0] if it's between that range?

If the stream consists of just characters you shouldn't convert it to hexvalue until you reach the CRC-characters.
So the if statement should be something like:
Code: Pascal  [Select][+][-]
  1.   for I := 0 to Size - 1 do
  2.       begin
  3.         if (PByte(Data)[0] > 33) and (PByte(Data)[0] < 125) then
  4.         begin
  5.           str := chr(PByte(Data)[0])
  6.         end;
and if you really want the CRC-code as characters (which I still think is useless):
Code: Pascal  [Select][+][-]
  1.   for I := 0 to Size - 1 do
  2.   begin
  3.     DecodedString := DecodedString + AnsiToUtf8(PansiChar(Data)[i]);
  4.   end;

What are you going to do with the resulting string?
If it's just for display or adding to a database it's best to strip the CRC characters and check if the string is complete (with the CRC) and just process the string further.

B.T.W. How is the CRC-part marked. with #00 or something? In that case it would be much simpler to just do this without a complete loop:
Code: Pascal  [Select][+][-]
  1. DecodedString  := AnsiToUtf8(Pansistring(Data));
Is it marked by another integer (like #2 or something) you could change that pByte(Data)[x] to #0 and do the same. (after saving the CRC to use for the string-check later)
« Last Edit: July 24, 2016, 10:18:15 am by rvk »

jdp

  • Full Member
  • ***
  • Posts: 144
Re: Memory leak converting CRC characters
« Reply #5 on: July 24, 2016, 11:06:48 am »
Thanks Alfred. will have a look at that.

Rik to be honest I got this code from somebody else.  I am only starting with Lazarus now. I know what I want to do but don't always know how to do it in Lazarus.

So what I do know is that the data comes from the port as an array of bytes, As the data arrives, not as one string, the ShowRead event fires multiple times and we build up the string as we go (You het 9 bytes at a time as that is the size of the USB buffer). We know we will have the full sting after 500 mili seconds thus we sleep in the tread that is waiting for that data. The value is hex in the bytes, Thus I need to convert the hex to ascii. Then once we have the full string I will split the string into the correct values as needed to do the all the calculations.

So if there is a better way to convert a byte of hex to ascii string then please let me know.

Kind Regards and thanks for all the help thus far. This forum rocks !!!!

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Memory leak converting CRC characters
« Reply #6 on: July 24, 2016, 11:39:37 am »
The value is hex in the bytes, Thus I need to convert the hex to ascii.
Wait... are you getting hex-values in your data?
In that case you shouldn't even be doing the Format() line.

But my guess is you're just getting byte-date. Then forget about "hex"-values entirely. Hex is only a representation of a byte. Like 66 = $42.

You mentioned CRC-data. How do you distinguish the CRC from the actual characters?

I would build up a small buffer of Bytes (array of bytes) and put the bytes in that array. After the timeout you can convert that array to a string (or strip any CRC-characters first and then convert it to a string). That way you don't have to check each character if it's a CRC-character or part of the string.

If you still want to add each character to the string you could do it like this:
Code: Pascal  [Select][+][-]
  1. for I := 0 to Size - 1 do
  2. begin
  3.   DecodedString := DecodedString + AnsiToUtf8(PansiChar(Data)[i]);
  4. end;
But that doesn't take any CRC character into account.
You would still need to know when they come.

Is there any documentation for this protocol (inverter brand and type?).

shobits1

  • Sr. Member
  • ****
  • Posts: 271
  • .
Re: Memory leak converting CRC characters
« Reply #7 on: July 24, 2016, 02:41:09 pm »
Is there any documentation for this protocol (inverter brand and type?).
Or do you have any actual data and how to interpret it.

jdp

  • Full Member
  • ***
  • Posts: 144
Re: Memory leak converting CRC characters
« Reply #8 on: July 24, 2016, 05:26:00 pm »
Hi all. I have a fully working system I developed in VB.Net (http://aicc.org.za/). We are busy porting the code to Lazarus to run it on the Raspberry Pi and Linux. So we know how the protocol works. That is not my problem. Here is a sample, but there are many commands with different return strings of different sizes. If I use serial (RS-232) to communicate with the Inverter then all the data is in plain text ASCII, but if you use USB then it seems that all data is transmitted in hex (not sure if this is always the case. This is my first USB project. Been working with serial for years) 

So here you can see how far we have come with some of the port.

https://www.youtube.com/watch?v=XQ2qNT1iYMI

https://www.youtube.com/watch?v=hv0KiLJT-fY

https://www.youtube.com/watch?v=7dfhbgmH0tg

So the problem is not the comms or how to talk to the Inverter. I need to understand what is the best way to chage hex to ascii so we dont get a memory leak as I am new to Lazarus and don't know why this is happeneing.

Kind Regards.



Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: Memory leak converting CRC characters
« Reply #9 on: July 24, 2016, 06:19:00 pm »
Casting to PAnsiChar will help.
Even assigning to a RawByteString before that.

BUT (big but!):

Note that vb.net has a UNICODE16 string type.... remember that.
Maybe just using {$mode delphiunicode} is enough. Maybe together with a PWideChar cast.
Specialize a type, not a var.

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: Memory leak converting CRC characters
« Reply #10 on: July 24, 2016, 07:19:35 pm »
I don't see "hex" in that image. As I said before hex is a presentation of a number. So my guess is when you send <16>003MD<13> to the device you get something back like:
<4>037111,222222,33,4,5,6666,7777,88,999<crc-code><13>
(^P is decimal 16, ^D is decimal 4 and <cr> is decimal 13 usually)

The ^D, <crc-code> and <cr> can't be converted to a string. You should check if you get ^D0037 before going on to read the string but it's still save to read the complete reply into a buffer until the <cr>. In that case you know you have the whole answer and otherwise you generate a timeout and repeat the command. You can use a array of bytes as buffer.

After that you should (optionally) check the crc against the string you received. Or you could live dangerously and just accept the string as is. You need to strip off that ^D0037 and <crc-code> (the <cr> shouldn't have been but in the buffer because it was an end of string code).

Then you can split the string on the "," and convert the hex (if 11, 22 etc where in hex value) to an integer (or other type).

You could just read the whole string like you do now and strip out the <33 and >125 but when there is incomplete communications this could lead to strange results in your database.

If I use serial (RS-232) to communicate with the Inverter then all the data is in plain text ASCII, but if you use USB then it seems that all data is transmitted in hex (not sure if this is always the case.
Data can't be transported as hex. It's ALWAYS transported as bytes. It's how you interpret the bytes that could be different. For instance:
Bytes: $68 $65 $6C $6C $6F is hello in ascii
But bytes: $31 $31 is 11 in ascii and you can translate that to an integer as eleven or integer seventeen if it's a hex-number in ascii :)

But first you need to interpret the result into a string before converting it to numbers. Because that protocol doesn't use straight numbers looking at the "," and the 0037 etc.

jdp

  • Full Member
  • ***
  • Posts: 144
Re: Memory leak converting CRC characters
« Reply #11 on: July 24, 2016, 08:12:32 pm »
Rik you are correct. The data is just bytes. It is not hex. I did convert with

Code: Pascal  [Select][+][-]
  1. DecodedString := DecodedString + AnsiToUtf8(PansiChar(Data)[i]);

And it converts correctly. I now have something to work with and will see if I can fix it.

Thanks again.

 

TinyPortal © 2005-2018