DecodeStringBase64() - could not understand output

[ertank]

Hello,

I have following Base64 encoded text:

Code: [Select]

7EFBZP6kPw==
It should normally decode into following bytes hex/decimal (as to my check on online base64 decoding to hex pages):

Code: [Select]

hex: ec 41 41 64 fe a4 3f
dec: 236 65 65 100 254 164 63

I could not manage to get that decoded data using a code like following:

Code: Pascal  [Select][+][-]

1. uses
2.   Base64;
3.  
4. var
5.   ct1: array [1..7] of Char;  // Using AnsiChar do not help either
6. begin
7.   ct1 := DecodeStringBase64('7EFBZP6kPw==');
8.   // ct1 end up with following information in debug value check
9.   // #195 #172 A A d #197 #159
10. end;
11.  

I am using following fpc and lazarus versions:

Code: [Select]

fpc 3.1.1 from trunk SVN revision: 38798
lazarus 1.8.3 from branches/fixes_1_8 SVN revision: 57681

I appreciate any help as I stuck to get that right.

Thanks & regards,
Ertan

[taazz]

Code: Pascal  [Select][+][-]

1. var
2.   vData :String;
3.   vStr  :String;
4.   vCntr :Integer;
5. begin
6.   vData := DecodeStringBase64('7EFBZP6kPw==');
7.   for vCntr := 1 to Length(vData) do
8.     vStr :=  vStr + IntToHex(Byte(vData[vCntr]),2)+',';
9.   SetLength(vStr,Length(vStr)-1);
10.   WriteLn(vStr);
11. end;

[ertank]

Hello taazz,

I get following output using sample code:

Code: [Select]

C3,AC,41,41,64,C5,9F,C2,A4,3F
That output is 10 bytes long and it is also wrong.

[ertank]

More tests show me that Indy base64 routines gives same result as online tool:

Code: Pascal  [Select][+][-]

1. uses
2.   IdGlobal,
3.   IdCoder,
4.   IdCoderMIME;
5.  
6. { TForm1 }
7.  
8. procedure TForm1.Button1Click(Sender: TObject);
9. var
10.   Bytes: TIdBytes;
11.   TempString: string;
12.   I: Integer;
13. begin
14.   Bytes := TIdDecoderMIME.DecodeBytes('7EFBZP6kPw==');
15.   TempString := EmptyStr;
16.   for I := Low(Bytes) to High(Bytes) do
17.     TempString := TempString + IntToHex(Bytes[I], 2) + ' ';
18.   ShowMessage(TempString);
19. end;
20.  

Message displayed is:

Code: [Select]

EC 41 41 64 FE A4 3F


[Thaddy]

Code: Pascal  [Select][+][-]

1. program untitled;
2. {$ifdef fpc}{$mode delphi}{$H+}{$endif}
3. uses base64,sysutils;
4. var
5.   vData :AnsiString; // <---
6.   vStr  :AnsiString; // <---
7.   vCntr :Integer;
8. begin
9.   vData := DecodeStringBase64('7EFBZP6kPw==');
10.   for vCntr := 1 to Length(vData) do
11.     vStr :=  vStr + IntToHex(Byte(vData[vCntr]),2)+',';
12.   SetLength(vStr,Length(vStr)-1);
13.   WriteLn(vStr);
14. end.

IOW: utf8 strings f*cked up (again)....This happens all the time, that's why I use fully qualified string types when using Lazarus. (And I strongly advise everybody to do just that...)

Code: Bash  [Select][+][-]

1. EC,41,41,64,FE,A4,3F
2.  
3.  
4. ------------------
5. (program exited with code: 0)
6.  

Same code but with "string" and a Lazarus GUI app gives the other result.

[taazz]

sorry I know that it does not help but here are screen shots of the above code producing correct results in windows 7 64bit with lazarus 1.4.4 and 18.2
Have you tried adding -dDisableUTF8RTL on your projects custom defines?

[ertank]

IOW: utf8 strings f*cked up (again)....This happens all the time, that's why I use fully qualified string types when using Lazarus. (And I strongly advise everybody to do just that...)

Same code but with "string" and a Lazarus GUI app gives the other result.

Hi Thaddy,

If I make a console application with your code its result is correct. However, on my system same code used in GUI with "mode delphi" and using AnsiString result is wrong. Result is still wrong if I define -dDisableUTF8RTL in project custom options. Moreover, I get 10 characters long result in vData variable after decoding takes place.

I also see that DecodeStringBase64() accepts only string and there is no overloaded AnsiString version of it. Not sure if that is relevant though.

My test with lazarus/trunk result is also wrong.

Is it possible you too try in a GUI application using same code of yours and check if result is still correct?

Thanks.

[taazz]

I'm guessing that has something to do with your source code encoding then try
1) create an external file (from notepad for example) with two lines

Code: Pascal  [Select][+][-]

1. const
2.   cData = '7EFBZP6kPw==';
3.  

and save it as data.inc somewhere in your unit add a {$I Data.inc} preferably somewhere in your implementation section before the call to your procedure that does the decoding and pass the cData const to the function instead, also try a simple writeln/showmessage(cdata) to see that everything looks ok.
here are a couple procedures that try to by pass the strings and return an array of bytes instead see if those make any difference.

Code: Pascal  [Select][+][-]

1. type
2.   TByteArray = array of Byte;
3.  
4. function DecodeBase64(const s:string;Strict:boolean=false):TByteArray;overload;
5. var
6.   Instream  : TStringStream;
7.   Outstream : TMemoryStream;
8.   Decoder   : TBase64DecodingStream;
9. begin
10.   Instream:=TStringStream.Create(s);
11.   try
12.     Outstream:=TMemoryStream.Create;
13.     try
14.       if strict then
15.         Decoder:=TBase64DecodingStream.Create(Instream,bdmStrict)
16.       else
17.         Decoder:=TBase64DecodingStream.Create(Instream,bdmMIME);
18.       try
19.          Outstream.CopyFrom(Decoder,Decoder.Size);
20.          SetLength(Result,Outstream.Size);
21.          Outstream.Position := 0;
22.          Move(Outstream.Memory^,Result[0], Outstream.Size);
23.          //Result:=Outstream.DataString;
24.       finally
25.         Decoder.Free;
26.       end;
27.     finally
28.       Outstream.Free;
29.     end;
30.   finally
31.     Instream.Free;
32.   end;
33. end;
34.  
35. function DecodeBase64(const s;const aSize:LongInt; Strict:boolean=false):TByteArray;overload;
36. var
37.   Instream  : TMemoryStream;
38.   Outstream : TMemoryStream;
39.   Decoder   : TBase64DecodingStream;
40. begin
41.   Instream:=TMemoryStream.Create;//(s);
42.   try
43.     Outstream:=TMemoryStream.Create;
44.     try
45.       Instream.WriteBuffer(s,aSize);
46.       Instream.Position := 0;
47.       if strict then
48.         Decoder:=TBase64DecodingStream.Create(Instream,bdmStrict)
49.       else
50.         Decoder:=TBase64DecodingStream.Create(Instream,bdmMIME);
51.       try
52.          Outstream.CopyFrom(Decoder,Decoder.Size);
53.          SetLength(Result,Outstream.Size);
54.          Outstream.Position := 0;
55.          Move(Outstream.Memory^,Result[0], Outstream.Size);
56.          //Result:=Outstream.DataString;
57.       finally
58.         Decoder.Free;
59.       end;
60.     finally
61.       Outstream.Free;
62.     end;
63.   finally
64.     Instream.Free;
65.   end;
66. end;
67.  
68. procedure TForm1.Button1Click(Sender :TObject);
69. //replace the const section with {$I data.inc} for testing
70. const
71.   cData = '7EFBZP6kPw==';
72. var
73.   vData :TByteArray;
74.   vStr  :String;
75.   vCntr :Integer;
76. begin
77.   vData := DecodeBase64(cData[1],Length(cData));//expected result EC,41,41,64,FE,A4,3F
78.   for vCntr := low(vData) to High(vData) do
79.     vStr :=  vStr + IntToHex(Byte(vData[vCntr]),2)+',';
80.   SetLength(vStr,Length(vStr)-1);
81.   ShowMessage(vStr);
82. end;
83.  

[ertank]

I'm guessing that has something to do with your source code encoding then try
1) create an external file (from notepad for example) with two lines
and save it as data.inc somewhere in your unit add a {$I Data.inc} preferably somewhere in your implementation section before the call to your procedure that does the decoding and pass the cData const to the function instead, also try a simple writeln/showmessage(cdata) to see that everything looks ok.
here are a couple procedures that try to by pass the strings and return an array of bytes instead see if those make any difference.

I assure you that is not related with my source code encoding. I checked it and newly created project file source encoding is UTF8. My former code source encoding is also UTF8. Moreover, my former code was working fine until I updated my fpc and lazarus versions yesterday.

Your new functions did work just fine. They still work if I use "{$mode objfpc}{$H+}" and they continue to work if I remove "-dDisableUTF8RTL" compiler parameter. I suppose they are good because they return Bytes instead of string.

I will make some more tests to see if it is all good.

Thank you.

[engkin]

I could be wrong, but I think this is the same bug.

To confirm see what value you get for TEncoding.Default.EncodingName. If not UTF8 then call TEncoding.FreeEncodings.

DefaultSystemCodePage is changed by Lazarus to 65001, that's UTF8, while TEncoding.Default holds the OS's encoding (Some ANSI encoding on Windows). TStringSteam uses TEncoding.Default.

Calling TEncoding.FreeEncodings should reset TEncoding.Default to DefaultSystemCodePage.

[Xor-el]

I can confirm such issues and it only occurs in gui programs.
In console app, it works fine.
To circumvent such issues, I wrote a library that works fine in both cases.
Here is it
https://github.com/Xor-el/SimpleBaseLib4Pascal

[ertank]

I could be wrong, but I think this is the same bug.

To confirm see what value you get for TEncoding.Default.EncodingName. If not UTF8 then call TEncoding.FreeEncodings.

DefaultSystemCodePage is changed by Lazarus to 65001, that's UTF8, while TEncoding.Default holds the OS's encoding (Some ANSI encoding on Windows). TStringSteam uses TEncoding.Default.

Calling TEncoding.FreeEncodings should reset TEncoding.Default to DefaultSystemCodePage.

I read TEncoding.Default.EncodingName = windows-1254
it is an exception if I try to read same value after calling TEncoding.FreeEncodings

[Thaddy]

As others stated it only occurs in a GUI app.
That means that - in this case - you simply should qualify string as AnsiString.
There is really no reason to touch codepage code when working with base64 encoding/decoding.

It is simply a recurring issue because of Lazarus UTF8 string encoding.
See my demo. (And yes, that also works with Cyrillic languages)

[engkin]

@Xor-el and @ertank, thanks for confirming.

@ertank, I can't test but try:

Code: Pascal  [Select][+][-]

1.       TEncoding.FreeEncodings;
2.       e := TEncoding.GetEncoding(DefaultSystemCodePage);

@Thaddy and @Xor-el, not just GUI apps. Any app, console or GUI, that uses LazUTF8 unit or changes DefaultSystemCodePage to something different from TEncoding.Default will have this bug.

[Thaddy]

@engkin
Yes, I know.