[SOLVED] CURL throws Error CURLE_REMOTE_FILE_NOT_FOUND but only on newer OS/CURL

[Hartmut]

I hope I picked the right subforum.

My little demo (see below) reads 1 mail-header via IMAP. It runs perfect on Ubuntu 18.04 (with CURL 7.58). But on Ubuntu 22.04 (with CURL 7.81) the same demo throws Error "CURLE_REMOTE_FILE_NOT_FOUND".
Don't know what happens on Windows, because my Windows 7 is too old for Internet activities.

I'm a beginner to CURL and need your help.

Code: Pascal  [Select][+][-]

1. {$mode objfpc}{$H+}
2.  
3. uses sysutils,classes,libcurl;
4.  
5. const
6.    url_imap    = 'imaps://imap.gmx.net'; {wanted URL}
7.    imap_folder = 'INBOX';                {wanted IMAP-Foldername}
8.    user_pw     = 'username:password';    {insert your username:password}
9. {$IFNDEF LINUX}                            {Windows-only: Certificate-Bundle: }
10.    fspec_crt: PChar = 'C:\Progs\Curl\bin\curl-ca-bundle.crt';
11. {$ENDIF}
12.    curl_Verb = true;      {CURL with verbose output?}
13.    FspecTmp = 'curl.tmp'; {temporary file for a mail header}
14.  
15. {$IFDEF LINUX}
16. type size_t = dword; {is on Windows in Unit system}
17. {$ENDIF}
18.  
19. var hCurl: pCurl; {= ^pointer = Curl-Handle}
20.  
21. function DoWrite(buf: Pointer; size: size_t; nmemb: size_t;
22.                  SS: Pointer): size_t; cdecl;
23.    {Helper for function curl_read_file(). Writes 'size*nmemb' bytes from buffer
24.     'buf' into a stream 'SS'. Returns the number of written bytes}
25.    begin
26.    result:=TStream(SS).Write(buf^,size*nmemb);
27.    end;
28.  
29. function curl_read_file(url, user_pw, creq, fspecO: ansistring): CURLcode;
30.    {reads from URL 'url' via CURL and writes the result into file 'fspecO'.
31.     In: - user_pw = Username + Password as '<username>:<password>'.
32.         - creq: optional custom request.
33.     Out: returns as result the CURL-errorcode or 'CURLE_OK'}
34.    var FS: TFileStream;
35.        pC: pChar;
36.        e: CURLcode;
37.    begin
38.    writeln('curl_read_file(): url="' + url + '", creq="' + creq + '"');
39.  
40.    if curl_Verb then                                                 {verbose?}
41.       curl_easy_setopt(hCurl,CURLOPT_VERBOSE,[True]);
42.    curl_easy_setopt(hCurl,CURLOPT_URL,[PChar(url)]);           {set Server-URL}
43.    curl_easy_setopt(hCurl,CURLOPT_USERPWD,[PChar(user_pw)]);    {set user + pw}
44. {$IFNDEF LINUX}
45.    curl_easy_setopt(hCurl,CURLOPT_CAINFO,[fspec_crt]); {set Certificate-Bundle}
46. {$ENDIF}
47.  
48.    if creq <> '' then pC:=PChar(creq)               {optional Custom Request: }
49.                  else pC:=nil;
50.    curl_easy_setopt(hCurl,CURLOPT_CUSTOMREQUEST,[pC]);
51.  
52.    FS:=TFileStream.Create(fspecO,fmCreate);              {output into a file: }
53.    curl_easy_setopt(hCurl,CURLOPT_WRITEFUNCTION,[@DoWrite]);
54.    curl_easy_setopt(hCurl,CURLOPT_WRITEDATA,[Pointer(FS)]);
55.  
56.    e:=curl_easy_perform(hCurl);                          {execute CURL-command}
57.    FS.Free;
58. writeln('Result1=', e);
59.    exit(e);
60.    end;
61.  
62. function Test_read_IMAP(nr: integer): CURLcode;
63.    {reads the Header of Mail number 'nr' into file 'FspecTmp' and returns as
64.     result the CURL-errorcode or 'CURLE_OK'}
65.    var url: ansistring;
66.        e: CURLcode;
67.    begin
68.    url:=url_imap + '/' + imap_folder + '/;UID=' + IntToStr(nr)
69.       + '/;SECTION=HEADER'; {read only the Header}
70.  
71.    hCurl:=curl_easy_init; {initialize CURL}
72.    if not system.Assigned(hCurl) then exit(CURLE_FAILED_INIT);
73.  
74.    try e:=curl_read_file(url,user_pw,'',FspecTmp);
75.    finally
76.        curl_easy_cleanup(hCurl);
77.    end; {try}
78.    exit(e);
79.    end;
80.  
81. var e: CURLcode;
82. begin {main}
83. e:=Test_read_IMAP(1); {reads mail-header from the 1st mail}
84. writeln('Result2=', e);
85. end.

I enabled "verbose CURL output" and attached the 2 outputs with CURL 7.58 and with CURL 7.81 and a diff-screenshot. What I see is, that CURL 7.58 uses TLS 1.2, while CURL 7.81 uses TLS 1.3. Can this be the problem? Or what else?

I use FPC 3.2.0 (but have the same error with FPC 3.2.2). Attached is a compilable demo-project. Thanks in advance.

[Hartmut]

Hmm, no answers after 4 days. Did I pick the wrong subforum? Did I describe my problem not clear enough? Then please ask...

[rvk]

You are hitting 2 different servers.
The imap implementation could be different.

Maybe you should first read out the UID before actually querying the BODY.

Difference in FETCH 1 (BODY) and UID FETCH 1.
https://forums.oracle.com/ords/apexds/post/uid-fetch-1-vs-fetch-1-285-uid-4595

So it's not certain your first message (1) is really UID = 1.
You shouldn't do a curl_read_file directly but first check out what the real output of the command.

(Though I'm not an imap expert)

BTW. I send you a PM.

[Hartmut]

Thank you very much Rik for helping me.

You are hitting 2 different servers. The imap implementation could be different.

I did not realize that before.

Maybe you should first read out the UID before actually querying the BODY.

My real program 1st reads in all UUID's and then selects 1 mail-number. Do you mean that?
But my demo shall read only the header, not the body of a mail.

Difference in FETCH 1 (BODY) and UID FETCH 1.
https://forums.oracle.com/ords/apexds/post/uid-fetch-1-vs-fetch-1-285-uid-4595

Sorry, I don't understand what you want to tell me and what is said in this link. I don't know Java or JavaMail.
What is meant with 'UID'? I only "know" UUID's like in this list:

Code: Text  [Select][+][-]

1. 1 1MmSgj-1pDbi21FIH-00iJVZ
2. 2 1MqfgY-1qfFac3pOi-00chkj
3. 3 1N5CpL-1qIpbz1s4T-010ylP
4. 4 1MAg1q-1pzjcy0FWi-00B8Co
5. 5 1MmFpi-1qbTMv42e8-00h7s9
6. ...

I call the 1st column 'mail number' and the 2nd column 'UUID'. How do you call them?

I'm a beginner to CURL and to all of that mail stuff. Please be more verbose. Thanks a lot.

[rvk]

I'm not sure if you can do it like that (using URL to retrieve a mailbody directory).
Normally you would connect to imap server and issue commands.

For the UID. With FETCH 1 you get the first mail in a box.
But with UID FETCH 1 you get the mail with UID 1.
But as I understand it, UID 1 isn't the same as the first mail.

So the error could be just that mail with UID 1 doesn't exist.

You could do a A001 UID SEARCH ALL to search for all UID's. When you use that UID in the UID FETCH x.

If you want to do this manually you can do this:

Code: Bash  [Select][+][-]

1. openssl s_client -connect imap.gmx.net:993 -quiet
2.  
3. a1 login username password
4.  
5. a2 select inbox
6.  
7. a3 uid search all

You get a list with UIDs (which for me definitely are not in the lower range).

Code: Bash  [Select][+][-]

1. a3 uid search all
2. * SEARCH 364420 364746 364899 365310 365367 365464 365583 365584 365585 365586 365587 365588 365589 365590
3. a3 OK Search completed (0.001 + 0.000 secs).

So you need to use one of those UIDs.

Problem is that I don't know if you can get those UID numbers via an URL.
(you can get them if you connect to the server and issue that command)

BTW. Synapse has some functionality build in to do all this.

[Hartmut]

The problem seems to be solved!

If you want to do this manually you can do this:

Code: Bash  [Select][+][-]

1. openssl s_client -connect imap.gmx.net:993 -quiet
2. a1 login username password
3. a2 select inbox
4. a3 uid search all

You get a list with UIDs (which for me definitely are not in the lower range).

I did this and got this list:

Code: Text  [Select][+][-]

1. a3 uid search all
2. * SEARCH 8168 8706 8707 8708 8709 8710 8711 8712 8713 8714 8715 8716 8717 8718 8719 8720 8721 8722 8723 8724
3. ...
4. 8872 8873 8874 8875 8876 8877
5. a3 OK UID SEARCH completed

An when I passed one of these numbers to my function Test_read_IMAP() it worked successfully.
So rvk was right, that after '/;UID=' a UID from above list has to be passed and not a relative mail-index, like I did.

Because I want to get a mail-header not for a UID, but for a relative mail-index, I found with google the right syntax: I only had to change line 68 in my demo from

Code: Pascal  [Select][+][-]

1. url:=url_imap + '/' + imap_folder + '/;UID=' + IntToStr(nr) + '/;SECTION=HEADER';

to

Code: Pascal  [Select][+][-]

1. url:=url_imap + '/' + imap_folder + ';MAILINDEX=' + IntToStr(nr) + ';SECTION=HEADER';

and now my demo works as I want.

I made a quick test with this change in my bigger real-world-program and it worked. So the problem seems to be solved (needs more deep testing in the next days).

Thanks a lot to rvk for his help. This is a great forum.