Recent

Author Topic: IMAP Search command & unicode in Synapse  (Read 2623 times)

paweld

  • Hero Member
  • *****
  • Posts: 852
Re: IMAP Search command & unicode in Synapse
« Reply #15 on: October 03, 2023, 10:17:56 am »
@rvk: Thank for the trick, hadn't thought of that.
Best regards / Pozdrawiam
paweld

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #16 on: October 03, 2023, 10:22:40 am »
@rvk: Thank for the trick, hadn't thought of that.
I only wonder if that also works for real other (extended) utf-8 characters.

The #$FC (as in ü) seems to be just ascii (1 character) encoding.
https://www.garykessler.net/library/ascii.html

So what would happen with other real utf-8 characters.
(guess I need to type some Chinese characters to test this  :D )

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #17 on: October 03, 2023, 10:38:05 am »
@rvk: Thank for the trick, hadn't thought of that.
I've got a better method.

I noticed the Charset for Header class was set to cp1252 on my system.
Forcing it to UTF-8, before the DecodeMessage, will give you the result directly in UTF-8  :D

Code: Pascal  [Select][+][-]
  1.           imap.FetchMess(StrToInt(MessageList[i]), Message);
  2.           MimeMess.Lines.Text := Message.Text;
  3.           MimeMess.Header.CharsetCode := UTF_8;  // was cp1252
  4.           MimeMess.DecodeMessage;
  5.           Memo1.Lines.Add('---------');
  6.           Memo1.Lines.Add('Subject: ' + MimeMess.Header.Subject);
  7.           Memo1.Lines.Add('From: ' + MimeMess.Header.From);
  8.           Memo1.Lines.Add('Date: ' + FormatDateTime('yyyy-mm-dd hh:nn:ss', MimeMess.Header.Date));

In that case extended UTF-8 characters are translated correctly too.
Internally mime encoding is used, but first translating it to cp1252 and then to Utf-8 will loose characters.
Letting it directly translate to utf-8 will fix that.
« Last Edit: October 03, 2023, 10:40:46 am by rvk »

paweld

  • Hero Member
  • *****
  • Posts: 852
Re: IMAP Search command & unicode in Synapse
« Reply #18 on: October 03, 2023, 10:48:31 am »
@rvk Thank you very much. When sending an email, I always set the charset to UTF8, but I didn't think that when reading it, you have to too. That's the trick to solve the problem.
Best regards / Pozdrawiam
paweld

Rebell

  • New Member
  • *
  • Posts: 13
Re: IMAP Search command & unicode in Synapse
« Reply #19 on: October 03, 2023, 11:05:03 am »
With me it finds messages without any problem, there is only a problem with decoding messages - there are ? instead of non-ascii characters.
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button8Click(Sender: TObject);
  2. var
  3.   imap: TIMAPSend;
  4.   MessageList, Message: TStringList;
  5.   MimeMess: TMimeMess;
  6.   i: Integer;
  7. begin
  8.   Memo1.Lines.Clear;
  9.   MessageList := TStringList.Create;
  10.   imap := TIMAPSend.Create;
  11.   imap.FullSSL := True;
  12.   imap.AutoTLS := True;
  13.   imap.Username := maillogin;
  14.   imap.Password := mailpasswd;
  15.   imap.TargetHost := mailhost;
  16.   imap.TargetPort := mailport;
  17.   if imap.Login then
  18.   begin
  19.     if imap.SelectFolder('INBOX') then
  20.     begin
  21.       if imap.SearchMess('SUBJECT "möglich"', MessageList) then
  22.       begin
  23.         Memo1.Lines.Add('Number of messages found: ' + IntToStr(MessageList.Count));
  24.         Memo1.Lines.Add('Mesaage list:');
  25.         Message := TStringList.Create;
  26.         MimeMess := TMimeMess.Create;
  27.         for i := 0 to MessageList.Count - 1 do
  28.         begin
  29.           Message.Clear;
  30.           imap.FetchMess(StrToInt(MessageList[i]), Message);
  31.           MimeMess.Lines.Text := Message.Text;
  32.           MimeMess.DecodeMessage;
  33.           Memo1.Lines.Add('---------');
  34.           Memo1.Lines.Add('Subject: ' + MimeMess.Header.Subject);
  35.           Memo1.Lines.Add('From: ' + MimeMess.Header.From);
  36.           Memo1.Lines.Add('Date: ' + FormatDateTime('yyyy-mm-dd hh:nn:ss', MimeMess.Header.Date));
  37.         end;
  38.         Message.Free;
  39.         MimeMess.Free;
  40.       end
  41.       else
  42.         ShowMessage('Message not found!')
  43.     end
  44.     else
  45.       ShowMessage('IMAP CHANGE FOLDER ERROR: ' + imap.ResultString);
  46.     imap.Logout;
  47.   end
  48.   else
  49.     ShowMessage('IMAP LOGIN ERROR: ' + imap.ResultString);
  50.   imap.Free;
  51.   MessageList.Free;
  52. end;    
  53.  

Your code give me same results, Nothing found ( Maybe something in My Synapse version , but i tried few ones.. And now using the latest one from trunk.
Can you share me your Synapse files please ? I will try

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #20 on: October 03, 2023, 11:08:10 am »
Your code give me same results, Nothing found ( Maybe something in My Synapse version , but i tried few ones.. And now using the latest one from trunk.
Can you share me your Synapse files please ? I will try
I've just used the synapse 40.1 from the Online Package Manager in Lazarus.

It could also be that you imap server is configured differently.

What happens if you just use "lich" as search term?
Does it find something?
And if it does, how is the encoding you get back (if you look in the debugger)?

Rebell

  • New Member
  • *
  • Posts: 13
Re: IMAP Search command & unicode in Synapse
« Reply #21 on: October 03, 2023, 11:24:43 am »
Your code give me same results, Nothing found ( Maybe something in My Synapse version , but i tried few ones.. And now using the latest one from trunk.
Can you share me your Synapse files please ? I will try
I've just used the synapse 40.1 from the Online Package Manager in Lazarus.

It could also be that you imap server is configured differently.

What happens if you just use "lich" as search term?
Does it find something?
And if it does, how is the encoding you get back (if you look in the debugger)?

Yes, it finds.. It do not find only when special chars in request.

Moreover, i see that it change this chars in requests in anyway...
"ü" it change to "u"
"ß" it change to "?"



Rebell

  • New Member
  • *
  • Posts: 13
Re: IMAP Search command & unicode in Synapse
« Reply #22 on: October 03, 2023, 11:28:26 am »
So i think trouble on my side, on Synapse side, that Synapse cant send normal request in UTF-8

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #23 on: October 03, 2023, 11:29:10 am »
Moreover, i see that it change this chars in requests in anyway...
"ü" it change to "u"
"ß" it change to "?"
Yeah, then it's not going to find anything  ;)

What version of Lazarus and OS are you using?

If I look in Synapse I don't see it is changing the UTF-8 characters anywhere.
Not in TIMAPSend.SearchMess and not in SendString.

So i think trouble on my side, on Synapse side, that Synapse cant send normal request in UTF-8
For us it does (same version).
So my guess is that it is your Lazarus version.

Rebell

  • New Member
  • *
  • Posts: 13
Re: IMAP Search command & unicode in Synapse
« Reply #24 on: October 03, 2023, 11:33:05 am »
Moreover, i see that it change this chars in requests in anyway...
"ü" it change to "u"
"ß" it change to "?"
Yeah, then it's not going to find anything  ;)

What version of Lazarus and OS are you using?

If I look in Synapse I don't see it is changing the UTF-8 characters anywhere.
Not in TIMAPSend.SearchMess and not in SendString.

So i think trouble on my side, on Synapse side, that Synapse cant send normal request in UTF-8
For us it does (same version).
So my guess is that it is your Lazarus version.

I'm using Rad Studio 11

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #25 on: October 03, 2023, 11:39:52 am »
I'm using Rad Studio 11
That's Delphi  %) %) %) %)

You are on a Lazarus/FPC forum here   ;)

Delphi doesn't do UTF-8. It does unicode/widestring/UTF-16 or something.

Rebell

  • New Member
  • *
  • Posts: 13
Re: IMAP Search command & unicode in Synapse
« Reply #26 on: October 03, 2023, 11:49:56 am »
Understand. Thank you very much ! I will continue my search )

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #27 on: October 03, 2023, 11:58:35 am »
In TIMAPSend.IMAPcommand there is a call to FSock.SendString.
In FSock.SendString it expects an AnsiString.

For Lazarus/FPC this can contain a UTF-8 string and it will send it just fine.

For Delphi (in which Value is a Unicodestring/Widestring, so 2 bytes per char) it will convert automatically to ansistring (NOT UTF-8) where you will loose characters.

You might want to try to convert to utf-8 before the call but I'm not sure how imapsend.pas will handle that (because there it is still a "string").

You can also hack imapsend.pas TIMAPSend.IMAPcommand to convert the value there from unicode to utf-8 in the expected ansistring.

« Last Edit: October 03, 2023, 12:16:26 pm by rvk »

rvk

  • Hero Member
  • *****
  • Posts: 5652
Re: IMAP Search command & unicode in Synapse
« Reply #28 on: October 03, 2023, 12:16:34 pm »
I've just tried this in Delphi and it seems to work:
Code: Pascal  [Select][+][-]
  1. var
  2.   S: AnsiString;
  3. //...
  4.   S := Utf8Encode('┴'); // upside T
  5.   if imap.SearchMess('SUBJECT "' +S + '"', MessageList) then

So converting your Widestring to a UTF-8 in AnsiString and passing it back to SearchMess should work.
(The AnsiString is again translated to Widestring, but the characters stay in the 255 range and are passed as such to the AnsiString again in SendString.)


Rebell

  • New Member
  • *
  • Posts: 13
Re: IMAP Search command & unicode in Synapse
« Reply #29 on: October 03, 2023, 12:24:14 pm »
I've just tried this in Delphi and it seems to work:
Code: Pascal  [Select][+][-]
  1. var
  2.   S: AnsiString;
  3. //...
  4.   S := Utf8Encode('┴'); // upside T
  5.   if imap.SearchMess('SUBJECT "' +S + '"', MessageList) then

So converting your Widestring to a UTF-8 in AnsiString and passing it back to SearchMess should work.
(The AnsiString is again translated to Widestring, but the characters stay in the 255 range and are passed as such to the AnsiString again in SendString.)


Uhuhuh ! It works ! Thank you so much !

 

TinyPortal © 2005-2018