IMAP Search command & unicode in Synapse

[paweld]

@rvk: Thank for the trick, hadn't thought of that.

[rvk]

@rvk: Thank for the trick, hadn't thought of that.

I only wonder if that also works for real other (extended) utf-8 characters.

The #$FC (as in ü) seems to be just ascii (1 character) encoding.
https://www.garykessler.net/library/ascii.html

So what would happen with other real utf-8 characters.
(guess I need to type some Chinese characters to test this  )

[rvk]

@rvk: Thank for the trick, hadn't thought of that.

I've got a better method.

I noticed the Charset for Header class was set to cp1252 on my system.
Forcing it to UTF-8, before the DecodeMessage, will give you the result directly in UTF-8 

Code: Pascal  [Select][+][-]

1.           imap.FetchMess(StrToInt(MessageList[i]), Message);
2.           MimeMess.Lines.Text := Message.Text;
3.           MimeMess.Header.CharsetCode := UTF_8;  // was cp1252
4.           MimeMess.DecodeMessage;
5.           Memo1.Lines.Add('---------');
6.           Memo1.Lines.Add('Subject: ' + MimeMess.Header.Subject);
7.           Memo1.Lines.Add('From: ' + MimeMess.Header.From);
8.           Memo1.Lines.Add('Date: ' + FormatDateTime('yyyy-mm-dd hh:nn:ss', MimeMess.Header.Date));

In that case extended UTF-8 characters are translated correctly too.
Internally mime encoding is used, but first translating it to cp1252 and then to Utf-8 will loose characters.
Letting it directly translate to utf-8 will fix that.

[paweld]

@rvk Thank you very much. When sending an email, I always set the charset to UTF8, but I didn't think that when reading it, you have to too. That's the trick to solve the problem.

[Rebell]

With me it finds messages without any problem, there is only a problem with decoding messages - there are ? instead of non-ascii characters.

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button8Click(Sender: TObject);
2. var
3.   imap: TIMAPSend;
4.   MessageList, Message: TStringList;
5.   MimeMess: TMimeMess;
6.   i: Integer;
7. begin
8.   Memo1.Lines.Clear;
9.   MessageList := TStringList.Create;
10.   imap := TIMAPSend.Create;
11.   imap.FullSSL := True;
12.   imap.AutoTLS := True;
13.   imap.Username := maillogin;
14.   imap.Password := mailpasswd;
15.   imap.TargetHost := mailhost;
16.   imap.TargetPort := mailport;
17.   if imap.Login then
18.   begin
19.     if imap.SelectFolder('INBOX') then
20.     begin
21.       if imap.SearchMess('SUBJECT "möglich"', MessageList) then
22.       begin
23.         Memo1.Lines.Add('Number of messages found: ' + IntToStr(MessageList.Count));
24.         Memo1.Lines.Add('Mesaage list:');
25.         Message := TStringList.Create;
26.         MimeMess := TMimeMess.Create;
27.         for i := 0 to MessageList.Count - 1 do
28.         begin
29.           Message.Clear;
30.           imap.FetchMess(StrToInt(MessageList[i]), Message);
31.           MimeMess.Lines.Text := Message.Text;
32.           MimeMess.DecodeMessage;
33.           Memo1.Lines.Add('---------');
34.           Memo1.Lines.Add('Subject: ' + MimeMess.Header.Subject);
35.           Memo1.Lines.Add('From: ' + MimeMess.Header.From);
36.           Memo1.Lines.Add('Date: ' + FormatDateTime('yyyy-mm-dd hh:nn:ss', MimeMess.Header.Date));
37.         end;
38.         Message.Free;
39.         MimeMess.Free;
40.       end
41.       else
42.         ShowMessage('Message not found!')
43.     end
44.     else
45.       ShowMessage('IMAP CHANGE FOLDER ERROR: ' + imap.ResultString);
46.     imap.Logout;
47.   end
48.   else
49.     ShowMessage('IMAP LOGIN ERROR: ' + imap.ResultString);
50.   imap.Free;
51.   MessageList.Free;
52. end;    
53.  

Your code give me same results, Nothing found ( Maybe something in My Synapse version , but i tried few ones.. And now using the latest one from trunk.
Can you share me your Synapse files please ? I will try

[rvk]

Your code give me same results, Nothing found ( Maybe something in My Synapse version , but i tried few ones.. And now using the latest one from trunk.
Can you share me your Synapse files please ? I will try

I've just used the synapse 40.1 from the Online Package Manager in Lazarus.

It could also be that you imap server is configured differently.

What happens if you just use "lich" as search term?
Does it find something?
And if it does, how is the encoding you get back (if you look in the debugger)?

[Rebell]

Your code give me same results, Nothing found ( Maybe something in My Synapse version , but i tried few ones.. And now using the latest one from trunk.
Can you share me your Synapse files please ? I will try

I've just used the synapse 40.1 from the Online Package Manager in Lazarus.

It could also be that you imap server is configured differently.

What happens if you just use "lich" as search term?
Does it find something?
And if it does, how is the encoding you get back (if you look in the debugger)?

Yes, it finds.. It do not find only when special chars in request.

Moreover, i see that it change this chars in requests in anyway...
"ü" it change to "u"
"ß" it change to "?"

[Rebell]

So i think trouble on my side, on Synapse side, that Synapse cant send normal request in UTF-8

[rvk]

Moreover, i see that it change this chars in requests in anyway...
"ü" it change to "u"
"ß" it change to "?"

Yeah, then it's not going to find anything 

What version of Lazarus and OS are you using?

If I look in Synapse I don't see it is changing the UTF-8 characters anywhere.
Not in TIMAPSend.SearchMess and not in SendString.

So i think trouble on my side, on Synapse side, that Synapse cant send normal request in UTF-8

For us it does (same version).
So my guess is that it is your Lazarus version.

[Rebell]

Moreover, i see that it change this chars in requests in anyway...
"ü" it change to "u"
"ß" it change to "?"

Yeah, then it's not going to find anything 

What version of Lazarus and OS are you using?

If I look in Synapse I don't see it is changing the UTF-8 characters anywhere.
Not in TIMAPSend.SearchMess and not in SendString.

So i think trouble on my side, on Synapse side, that Synapse cant send normal request in UTF-8

For us it does (same version).
So my guess is that it is your Lazarus version.

I'm using Rad Studio 11

[rvk]

I'm using Rad Studio 11

That's Delphi 

You are on a Lazarus/FPC forum here   

Delphi doesn't do UTF-8. It does unicode/widestring/UTF-16 or something.

[Rebell]

Understand. Thank you very much ! I will continue my search )

[rvk]

In TIMAPSend.IMAPcommand there is a call to FSock.SendString.
In FSock.SendString it expects an AnsiString.

For Lazarus/FPC this can contain a UTF-8 string and it will send it just fine.

For Delphi (in which Value is a Unicodestring/Widestring, so 2 bytes per char) it will convert automatically to ansistring (NOT UTF-8) where you will loose characters.

You might want to try to convert to utf-8 before the call but I'm not sure how imapsend.pas will handle that (because there it is still a "string").

You can also hack imapsend.pas TIMAPSend.IMAPcommand to convert the value there from unicode to utf-8 in the expected ansistring.

[rvk]

I've just tried this in Delphi and it seems to work:

Code: Pascal  [Select][+][-]

1. var
2.   S: AnsiString;
3. //...
4.   S := Utf8Encode('┴'); // upside T
5.   if imap.SearchMess('SUBJECT "' +S + '"', MessageList) then

So converting your Widestring to a UTF-8 in AnsiString and passing it back to SearchMess should work.
(The AnsiString is again translated to Widestring, but the characters stay in the 255 range and are passed as such to the AnsiString again in SendString.)

[Rebell]

I've just tried this in Delphi and it seems to work:

Code: Pascal  [Select][+][-]

1. var
2.   S: AnsiString;
3. //...
4.   S := Utf8Encode('┴'); // upside T
5.   if imap.SearchMess('SUBJECT "' +S + '"', MessageList) then

So converting your Widestring to a UTF-8 in AnsiString and passing it back to SearchMess should work.
(The AnsiString is again translated to Widestring, but the characters stay in the 255 range and are passed as such to the AnsiString again in SendString.)

Uhuhuh ! It works ! Thank you so much !