IMAP Search command & unicode in Synapse

[Rebell]

Hello, I’m using Synapse, and I can’t figure out how to use Japanese, German, and Russian characters for the search command in IMAP.  I tried playing with UTF8Encode, but in socket logs i see that in anyway characters are not recognized correctly. At best I get the error  "Error in IMAP command SEARCH: Invalid search key".

So, for example. How to use a such requests ?

Code: Pascal  [Select][+][-]

1. Imap.SearchMess('SUBJECT  möglich');

In this case, socket log appear :

Code: Pascal  [Select][+][-]

1. 03896AA0-> S8 SEARCH SUBJECT "moglich"

so "ö" symbol looks in other way.

Another symbols like "ß" just change to "?" .

Please explain to me how to deal with this. What encoding should I use?

[rvk]

Have you tried mime encoding the string.

 ś would become =B6 etc.

You can use EncodeQuotedPrintable() in synacode.

[Rebell]

For some reason this function does not work for me and does not convert the string. I tried to specify an already encoded value as a constant. There was no error in the server’s response, but he didn’t see the letter either. I'm not sure that the IMAP server automatically decrypts this encoding.

Code: Pascal  [Select][+][-]

1. 20231003-030521.547 030D6AA0-> S7 SEARCH FROM "abzuschlie=C3=9Fen"
2.  
3. 20231003-030521.547 030D6AA0HR_WriteCount: 37
4. 20231003-030521.764 030D6AA0HR_CanRead:
5. 20231003-030521.764 030D6AA0HR_ReadCount: 64
6. 20231003-030521.764 030D6AA0<- * SEARCH
7. S7 OK Search completed (0.004 + 0.000 + 0.003 secs).

[rvk]

Code: Pascal  [Select][+][-]

1. 20231003-030521.547 030D6AA0-> S7 SEARCH FROM "abzuschlie=C3=9Fen"
2.  

You are searching FROM there.
Does subject give the same empty result?

Does it return the mime encoding when you ask for a list with subjects?
Something like
a1 FETCH 1:* (FLAGS BODY.PEEK[HEADER.FIELDS (SUBJECT)])

(Not sure if that's the exact correct notation for a list)

[Rebell]

For some reason this function does not work for me and does not convert the string. I tried to specify an already encoded value as a constant. There was no error in the server’s response, but he didn’t see the letter either. I'm not sure that the IMAP server automatically decrypts this encoding.

Code: Pascal  [Select][+][-]

1. 20231003-030521.547 030D6AA0-> S7 SEARCH FROM "abzuschlie=C3=9Fen"
2.  
3. 20231003-030521.547 030D6AA0HR_WriteCount: 37
4. 20231003-030521.764 030D6AA0HR_CanRead:
5. 20231003-030521.764 030D6AA0HR_ReadCount: 64
6. 20231003-030521.764 030D6AA0<- * SEARCH
7. S7 OK Search completed (0.004 + 0.000 + 0.003 secs).

yes, same results, also with message part.. It's just do not see this keyword neither in the headers nor in the body of the letter

[rvk]

I meant how does the encoding of those characters look when you retrieve the mail without converting the raw message in imap?

Do you see real utf-8 or encoded characters with = ?

[Rebell]

For some reason this function does not work for me and does not convert the string. I tried to specify an already encoded value as a constant. There was no error in the server’s response, but he didn’t see the letter either. I'm not sure that the IMAP server automatically decrypts this encoding.

Code: Pascal  [Select][+][-]

1. 20231003-030521.547 030D6AA0-> S7 SEARCH FROM "abzuschlie=C3=9Fen"
2.  
3. 20231003-030521.547 030D6AA0HR_WriteCount: 37
4. 20231003-030521.764 030D6AA0HR_CanRead:
5. 20231003-030521.764 030D6AA0HR_ReadCount: 64
6. 20231003-030521.764 030D6AA0<- * SEARCH
7. S7 OK Search completed (0.004 + 0.000 + 0.003 secs).

yes, same results, also with message part.. It's just do not see this keyword neither in the headers nor in the body of the letter

For example, full log :

Code: Pascal  [Select][+][-]

1. 20231003-033423.642 03766AA0HR_Connect: secureimap.t-online.de:993
2. 20231003-033424.549 03766AA0HR_ReadCount: 120
3. 20231003-033424.549 03766AA0<- * OK [CAPABILITY IMAP4 IMAP4rev1 QUOTA LITERAL+ NAMESPACE UIDPLUS IDLE SORT] IMAP Proxy ready (secureimap.t-online.de)
4.  
5. 20231003-033424.549 03766AA0-> S1 CAPABILITY
6.  
7. 20231003-033424.549 03766AA0HR_WriteCount: 15
8. 20231003-033424.734 03766AA0HR_CanRead:
9. 20231003-033424.734 03766AA0HR_ReadCount: 145
10. 20231003-033424.734 03766AA0<- * CAPABILITY IMAP4 IMAP4rev1 QUOTA LITERAL+ NAMESPACE UIDPLUS IDLE SORT
11. S1 OK Pre-login capabilities listed, post-login capabilities have more
12.  
13. 20231003-033424.734 03766AA0-> S2 LOGIN "myemail@t-online.de" "mypassword"
14.  
15. 20231003-033424.734 03766AA0HR_WriteCount: 45
16. 20231003-033424.951 03766AA0HR_CanRead:
17. 20231003-033424.951 03766AA0HR_ReadCount: 457
18. 20231003-033424.951 03766AA0<- S2 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY METADATA SPECIAL-USE COMPRESS=DEFLATE XEXPIRE QUOTA] Logged in
19.  
20. 20231003-033424.951 03766AA0-> S3 LIST "" *
21.  
22. 20231003-033424.951 03766AA0HR_WriteCount: 14
23. 20231003-033425.168 03766AA0HR_CanRead:
24. 20231003-033425.168 03766AA0HR_ReadCount: 367
25. 20231003-033425.168 03766AA0<- * LIST (\HasChildren) "." INBOX
26. * LIST (\HasNoChildren \UnMarked \Junk) "." INBOX.Spam
27. * LIST (\HasNoChildren \UnMarked \Trash) "." INBOX.Trash
28. * LIST (\HasNoChildren \UnMarked \Sent) "." INBOX.Sent
29. * LIST (\HasNoChildren \UnMarked) "." INBOX.Schule
30. * LIST (\HasNoChildren \UnMarked \Drafts) "." INBOX.Drafts
31. S3 OK List completed (0.003 + 0.000 + 0.002 secs).
32.  
33. 20231003-033425.168 03766AA0-> S4 SELECT "INBOX"
34.  
35. 20231003-033425.168 03766AA0HR_WriteCount: 19
36. 20231003-033425.438 03766AA0HR_CanRead:
37. 20231003-033425.438 03766AA0HR_ReadCount: 495
38. 20231003-033425.438 03766AA0<- * FLAGS (\Answered \Flagged \Deleted \Seen \Draft DTAG_document DTAG_image $HasNoAttachment $HasAttachment)
39. * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft DTAG_document DTAG_image $HasNoAttachment $HasAttachment \*)] Flags permitted.
40. * 6021 EXISTS
41. * 0 RECENT
42. * OK [UNSEEN 2] First unseen.
43. * OK [UIDVALIDITY 1440249986] UIDs valid
44. * OK [UIDNEXT 8647] Predicted next UID
45. * OK [HIGHESTMODSEQ 17827] Highest
46. S4 OK [READ-WRITE] Select completed (0.001 + 0.000 + 0.001 secs).
47.  
48. 20231003-033425.438 03766AA0-> S5 SEARCH SUBJECT "f=c3=bcr sie"
49.  
50. 20231003-033425.438 03766AA0HR_WriteCount: 34
51. 20231003-033425.707 03766AA0HR_CanRead:
52. 20231003-033425.707 03766AA0HR_ReadCount: 64
53. 20231003-033425.707 03766AA0<- * SEARCH
54. S5 OK Search completed (0.019 + 0.000 + 0.018 secs).
55.  
56. 20231003-033425.707 03766AA0-> S6 SELECT "INBOX.Trash"
57.  

[Rebell]

I meant how does the encoding of those characters look when you retrieve the mail without converting the raw message in imap?

Do you see real utf-8 or encoded characters with = ?

Code: Pascal  [Select][+][-]

1. Imap.FetchHeader(ID,MimeMess)
2. MimeMess.MessagePart.TargetCharset:=UTF_8;
3. MimeMess.MessagePart.ConvertCharset:=true;
4. MimeMess.DecodeMessage;
5.  

So after decoding MimeMess.Header.Subject  Gives me :
                  Wichtige Hinweise f_r Sie
Have to be : Wichtige Hinweise für Sie

[rvk]

Does the full log of Imap.FetchHeader show the raw headers?

[Rebell]

Does the full log of Imap.FetchHeader show the raw headers?

Yes it is.

Here is subject in raw: 

Code: Pascal  [Select][+][-]

1. Subject: =?utf-8?q?Wichtige_Hinweise_f=C3=BCr_Sie?=

[rvk]

Try to search with passing the encoding too.

So search
=?utf-8?q?f=C3=BCr_Sie?=

Also in that exact case (case sensitive).

[rvk]

Actually, for me the unencoded utf8 does seem to work in my imap.

a1 FETCH 1:* (FLAGS BODY.PEEK[HEADER.FIELDS (SUBJECT)])
* 1 FETCH (FLAGS (\Seen) BODY[HEADER.FIELDS (SUBJECT)] {106}
Subject: Topic reply: Direct use of OpenSSL without using intermediate libraries (LNet, Indy, Synapse)

)
<Snip>
* 20 FETCH (FLAGS (\Seen \Recent) BODY[HEADER.FIELDS (SUBJECT)] {66}
Subject: Topic reply: IMAP Search command & unicode in Synapse

)
* 21 FETCH (FLAGS (\Seen) BODY[HEADER.FIELDS (SUBJECT)] {41}
Subject: =?UTF-8?Q?Test_=C3=BC_test?=


* 22 FETCH (FLAGS (\Recent) BODY[HEADER.FIELDS (SUBJECT)] {41}
Subject: =?UTF-8?Q?Test_=C3=BC_test?=

)
a1 OK Fetch completed (0.002 + 0.000 + 0.001 secs).
a2 search subject "ü"
* SEARCH 21 22
a2 OK Search completed (0.002 + 0.000 + 0.001 secs).

Done directly via telnet.

[Rebell]

Try to search with passing the encoding too.

So search
=?utf-8?q?f=C3=BCr_Sie?=

Also in that exact case (case sensitive).

Nope, it doesn't find it.

Code: Pascal  [Select][+][-]

1. 20231003-041635.133 03676AA0-> S5 SEARCH SUBJECT "=?utf-8?q?f=C3=BCr_Sie?="
2.  
3. 20231003-041635.133 03676AA0HR_WriteCount: 46
4. 20231003-041635.350 03676AA0HR_CanRead:
5. 20231003-041635.350 03676AA0HR_ReadCount: 64
6. 20231003-041635.350 03676AA0<- * SEARCH
7. S5 OK Search completed (0.032 + 0.000 + 0.031 secs).

But it finds the letter if you specify the part without a special character

Code: Pascal  [Select][+][-]

1. 20231003-042331.400 030A6AA0-> S5 SEARCH SUBJECT "Wichtige Hinweise"
2.  
3. 20231003-042331.400 030A6AA0HR_WriteCount: 39
4. 20231003-042331.663 030A6AA0HR_CanRead:
5. 20231003-042331.663 030A6AA0HR_ReadCount: 69
6. 20231003-042331.663 030A6AA0<- * SEARCH 6012
7. S5 OK Search completed (0.050 + 0.000 + 0.049 secs).

[paweld]

With me it finds messages without any problem, there is only a problem with decoding messages - there are ? instead of non-ascii characters.

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button8Click(Sender: TObject);
2. var
3.   imap: TIMAPSend;
4.   MessageList, Message: TStringList;
5.   MimeMess: TMimeMess;
6.   i: Integer;
7. begin
8.   Memo1.Lines.Clear;
9.   MessageList := TStringList.Create;
10.   imap := TIMAPSend.Create;
11.   imap.FullSSL := True;
12.   imap.AutoTLS := True;
13.   imap.Username := maillogin;
14.   imap.Password := mailpasswd;
15.   imap.TargetHost := mailhost;
16.   imap.TargetPort := mailport;
17.   if imap.Login then
18.   begin
19.     if imap.SelectFolder('INBOX') then
20.     begin
21.       if imap.SearchMess('SUBJECT "möglich"', MessageList) then
22.       begin
23.         Memo1.Lines.Add('Number of messages found: ' + IntToStr(MessageList.Count));
24.         Memo1.Lines.Add('Mesaage list:');
25.         Message := TStringList.Create;
26.         MimeMess := TMimeMess.Create;
27.         for i := 0 to MessageList.Count - 1 do
28.         begin
29.           Message.Clear;
30.           imap.FetchMess(StrToInt(MessageList[i]), Message);
31.           MimeMess.Lines.Text := Message.Text;
32.           MimeMess.DecodeMessage;
33.           Memo1.Lines.Add('---------');
34.           Memo1.Lines.Add('Subject: ' + MimeMess.Header.Subject);
35.           Memo1.Lines.Add('From: ' + MimeMess.Header.From);
36.           Memo1.Lines.Add('Date: ' + FormatDateTime('yyyy-mm-dd hh:nn:ss', MimeMess.Header.Date));
37.         end;
38.         Message.Free;
39.         MimeMess.Free;
40.       end
41.       else
42.         ShowMessage('Message not found!')
43.     end
44.     else
45.       ShowMessage('IMAP CHANGE FOLDER ERROR: ' + imap.ResultString);
46.     imap.Logout;
47.   end
48.   else
49.     ShowMessage('IMAP LOGIN ERROR: ' + imap.ResultString);
50.   imap.Free;
51.   MessageList.Free;
52. end;    
53.  

[rvk]

With me it finds messages without any problem, there is only a problem with decoding messages - there are ? instead of non-ascii characters.

That worked for me too.

I got a string with #$FC for ü back.
Throwing the string through WinCPToUTF8 (from LazUtf8) did fix that.