[SOLVED] How to write string to TBytesStream in UCS2 encoding?

[artem101]

How to convert string to ucs2 encoding and write it to TBytesStream?

Code: Pascal  [Select][+][-]

1. procedure send_reply(msg: string);
2. var
3.   s: string;
4.   strm: TBytesStream;
5. begin
6.   s := 'Это строка';
7.   strm := TBytesStream.Create;
8.  
9.   try
10.     strm.WriteString(s); // how to write in UCS2 encoding ???
11.     strm.SaveToFile('file.txt');
12.  
13.   finally
14.     strm.Free;
15.   end;
16. end;      

[dsiders]

How to convert string to ucs2 encoding and write it to TBytesStream?

Code: Pascal  [Select][+][-]

1. procedure send_reply(msg: string);
2. var
3.   s: string;
4.   strm: TBytesStream;
5. begin
6.   s := 'Это строка';
7.   strm := TBytesStream.Create;
8.  
9.   try
10.     strm.WriteString(s); // how to write in UCS2 encoding ???
11.     strm.SaveToFile('file.txt');
12.  
13.   finally
14.     strm.Free;
15.   end;
16. end;      

Perhaps?

Code: Pascal  [Select][+][-]

1. procedure send_reply(msg: string);
2. var
3.   s: string;
4.   strm: TBytesStream;
5.   in, out: TBytes;
6. begin
7.   s := 'Это строка';
8.   in := BytesOf(s);
9.   out := TEncoding.Convert(TEncoding.UTF8, TEncoding.{BigEndian}Unicode, in);
10.   strm := TBytesStream.Create(out);
11.  
12.   try
13.     strm.SaveToFile('file.txt');
14.   finally
15.     strm.Free;  
16.   end;
17. end;
18.  

[theo]

Why not use a TStringStream?
It is a direct descendant of TBytesStream.

Code: Pascal  [Select][+][-]

1.   var
2.     s: string;
3.     strm: TStringStream;
4.   begin
5.     s := 'Это строка';
6.     strm := TStringStream.Create(UTF8Decode(s));
7.     try
8.       strm.SaveToFile('file.txt');
9.     finally
10.       strm.Free;
11.     end;
12.   end;  

[artem101]

Why not use a TStringStream?

Adding string in constructor is not suits me - I also to need to add some binary data before and after string. I tried to use writeunicodestring method with utf8decode, but it adds mess to the stream:

Code: Pascal  [Select][+][-]

1. var
2.   s: string;
3.   strm: TStringStream;
4. begin
5.   s := 'Это строка';
6.  
7.   strm := TStringStream.create;
8.   strm.writebyte(length(s));
9.   strm.writebyte($00); // add some bytes before
10.   strm.writeunicodestring(utf8decode(s));
11.   strm.writebyte($00); // and after string
12.  
13.   print_stream(strm);
14.   strm.free;

end;
Bytes:

13 00 3F 3F 3F 20 3F 3F 3F 3F 3F 3F 3F 20 3F 3F 3F 3F 3F 3F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 41 80 27 00 00 00 00 00 B8 32 D9 7C 4D 7F 00 00 00 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 00 3F 3F 3F 20 3F 3F 3F 3F 3F 3F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 41 80 2B 00 00 00 00 00 F8 32 D9 7C 4D 7F 00 00 78 32 D9 7C 4D 7F 00 00 00 00 00 00 00 00 00 00 00 00 ...

Demo: https://onlinegdb.com/6sWiXYg25

[theo]

This?
You can also use a TBytesStream in this case.

Code: Pascal  [Select][+][-]

1. var
2.   ws: widestring;
3.   strm: TStringStream;
4. begin
5.   ws := utf8decode('Это строка');
6.   strm := TStringStream.Create;
7.   strm.writebyte($02); // add some bytes before
8.   strm.writebyte($03);
9.   strm.writeBuffer(WS[1], Length(ws) * 2);
10.   strm.writebyte($04); // and after string
11.   strm.writebyte($05);
12.   strm.SaveToFile('filestr.txt');
13.   strm.Free;
14. end;    

[artem101]

This?
You can also use a TBytesStream in this case.

Code: Pascal  [Select][+][-]

1. var
2.   ws: widestring;
3.   strm: TStringStream;
4. begin
5.   ws := utf8decode('Это строка');
6.   strm := TStringStream.Create;
7.   strm.writebyte($02); // add some bytes before
8.   strm.writebyte($03);
9.   strm.writeBuffer(WS[1], Length(ws) * 2);
10.   strm.writebyte($04); // and after string
11.   strm.writebyte($05);
12.   strm.SaveToFile('filestr.txt');
13.   strm.Free;
14. end;    

This is what I need. It works. Thanks.

[Remy Lebeau]

I tried to use writeunicodestring method with utf8decode, but it adds mess to the stream

Something appears to be very messed up with TStringStream.WriteUnicodeString(). If the stream is empty, it doesn't write anything at all. And if the stream is not empty, it writes WAY too many bytes (4098 instead of 22).

Also, FYI using BytesOf() won't work, because it will not return UCS2/UTF16-encoded bytes when passed a decoded UnicodeString.

This works for me:

Code: Pascal  [Select][+][-]

1. var
2.   s: string;
3.   us: UnicodeString;  
4.   out: TBytes;
5.   strm: TBytesStream;
6. begin
7.   s := 'Это строка';
8.      
9.   strm := TBytesStream.create;
10.   strm.WriteWord(Length(s)); // add some bytes before
11.   // either:
12.   us := UTF8Decode(s);
13.   strm.WriteBuffer(PByte(us)^, Length(us)*2);
14.   // or:
15.   out := TEncoding.Unicode.GetBytes(UTF8Decode(s));
16.   strm.WriteBuffer(PByte(out)^, Length(out));
17.   //
18.   strm.WriteByte($00); // and after string
19.      
20.   print_stream(strm);
21.   strm.free;

[Remy Lebeau]

Code: Pascal  [Select][+][-]

1. var
2.   ws: widestring;
3. begin
4.   ws := utf8decode('Это строка');

UTF8Decode() returns a UnicodeString, not a WideString. Use WideString only when you are writing code that interacts with ActiveX/COM, otherwise use UnicodeString for anything else.

[artem101]

returns a UnicodeString, not a WideString

What a difference?

[LeP]

returns a UnicodeString, not a WideString

What a difference?

https://forum.lazarus.freepascal.org/index.php/topic,22464.0.html

[theo]

Use WideString only when you are writing code that interacts with ActiveX/COM, otherwise use UnicodeString for anything else.

As a general recommendation, you're right.
But I don't think it makes much of a difference in this example.

[Remy Lebeau]

But I don't think it makes much of a difference in this example.

The data is effectively the same, but the memory management is very different. For trivial examples, sure the difference is negligible. But in production code, it can matter a lot. Best to teach proper practices.