Recent

Author Topic: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem  (Read 2356 times)

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2020
  • Former Delphi 1-7, 10.2 user
Over the last few days I've been updating the lNet (Lightweight Networking Library) to compile and run successfully on FreeBSD, macOS and Windows using FPC 3.2.2 or FPC 3.3.1. It now does. macOS happily uses TLS 1.3 (thanks to MIA Forum user Tz) for SMTP;  FreeBSD happily uses TLS 1.2 for SMTP; and Windows 32 and 64 bit happily use TLS 1.2 for SMTP.

I should mention that I've only thoroughly tested the SMTP protocol which was the itch I wanted to scratch because macOS has no framework for SMTP at all and so a third-party library was required.

I have added the Lazarus SMTP client example to the Wiki here.

The updated lNet library is on GitHub here.

Alas, my Linux (Ubuntu 20.10) attempts met with a little less success. If I comment out the TLS handshake (FSMTP.StartTLS;) , the SMTP transaction flies through with no problems. If I leave the TLS handshake in, it succeeds but then the application locks up with the error:

Error on epoll [4]: Interrupted system call

which comes from the Linux only file lib/sys/lepolleventer.inc and never sends the email to my FreeBSD sendmail server which waits patiently to no avail.

I very rarely ever use any Linux and I'll happily concede that I know nothing about its workings. My searching threw up a few similar epoll issues but did not enlighten me sufficiently to come up with a fix.

So, if anyone feels like trying to tame the beast, you'll need to change the only FreeBSD ifdef in lib/lnetssl.pp to Linux because Linux, like FreeBSD, always seems to want to use SSL 2/3 rather than TLS 1.x and fails, so I force it to use TLS 1.2. macOS happily picks TLS 1.3 all by itself.

Have fun :-)

[Updated to add Windows]
« Last Edit: March 20, 2022, 01:09:25 pm by trev »

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: lNet updated for macOS + FreeBSD, Linux epoll problem
« Reply #1 on: March 19, 2022, 01:39:30 pm »
I'm neither sufficiently up to speed on TLS nor a sufficiently good Linux systems programmer to be able to help directly, but out of curiosity: is this based on the master Lnet sources which I think are on Github these days, or on the variant that comes with FPC (utils/fppkg/lnet in the sources)?

The reason I ask is that the latter has some extra Telnet support stuff which I think never made it into the main version.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2020
  • Former Delphi 1-7, 10.2 user
Re: lNet updated for macOS + FreeBSD, Linux epoll problem
« Reply #2 on: March 19, 2022, 02:09:24 pm »
I forked the these  lNet sources on GitHub after dabbling with the FPC ones and then discovering warnings that they were only for FPC development and should be avoided.

Quote
I'm neither sufficiently up to speed on TLS nor a sufficiently good Linux systems programmer

I resemble both aspects of that remark ;-)

I think the TLS handshake is ok as apart from the client saying so, my mail server also says:

Quote
Mar 19 23:46:51 shadow sm-mta[75484]: STARTTLS=server, relay=ubuntu.example.org [192.168.1.26], version=TLSv1.2, verify=NO, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256/256


MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: lNet updated for macOS + FreeBSD, Linux epoll problem
« Reply #3 on: March 19, 2022, 02:46:08 pm »
I find Lnet very difficult internally, and only use it for Telnet... and the next time I need that I'll redo from scratch. I modded the FPC ones to support the Hercules mainframe emulator, which used subcommands and demanded a terminal type that aren't widely implemented.

Apropos TLS, I saw a situation late last month where an accounting package would no longer send invoices via Office-365 which turned out to be something related to TLS version. I mention this because there seems to be an unholy number of security patches being pushed out by the various OS vendors at the moment, and there's always a possibility that the test system which is giving you problems has had its configuration forcibly changed recently.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2020
  • Former Delphi 1-7, 10.2 user
Re: lNet updated for macOS + FreeBSD, Linux epoll problem
« Reply #4 on: March 20, 2022, 07:01:37 am »
I've updated lNet - there's no longer an ifdef for FreeBSD. There's one for Darwin/macOS when setting the TLS preferred order, everything else gets a forced TLS 1.2 and down.

As for the example SMTP client, I forgot to mention that for Linux (GTK2) you need to scatter application.processmessages around liberally (once in each of the On... functions and in the main loop) so that the log is updated - this wasn't needed for macOS (Cocoa) or FreeBSD (GTK2). Weird.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: lNet updated for macOS + FreeBSD, Linux epoll problem
« Reply #5 on: March 20, 2022, 08:58:27 am »
you need to scatter application.processmessages around liberally

Which is one of the reasons I really don't like Lnet: I suspect it predates threads, and I've seen some odd effects when one end or the other of a connection has been dropped unexpectedly which I've not been able to resolve  due to the complexity of the code.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2020
  • Former Delphi 1-7, 10.2 user
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #6 on: March 20, 2022, 10:41:48 am »
The scattering seems confined to Linux. I'm working on Windows at the moment <argh DLL hell> and like macOS and FreeBSD it doesn't require the scattering.

And... it now works on Windows 32 bit and 64 bit (tested on Win 10 only, both 32 bit and 64 bit).
« Last Edit: March 20, 2022, 01:08:04 pm by trev »

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2020
  • Former Delphi 1-7, 10.2 user
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #7 on: March 22, 2022, 08:54:48 am »
With my latest commit, FreeBSD, macOS and Windows now all choose TLS v1.3. Yay.

Linux still chooses TLS v1.2 and dies after the handshake with the epoll error. Something to do with the Linux-specific lib/sys/lepolleventer.inc file obviously needs a tweak. The other operating systems use the BSD/Win inc files and are happy little Vegemites.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #8 on: March 22, 2022, 10:08:01 am »
Error on epoll [4]: Interrupted system call

On Linux, syscalls are not reliable. To avoid deadlock, the kernel can let them fail with EsysEINTR, and all syscalls should be guarded with repeat loops to restart the syscall in such case. See e.g. the write() call in rtl/unix/sysfile.inc  (see below)  and grep for eintr in that directory

Code: Pascal  [Select][+][-]
  1. Function Do_Write(Handle:thandle;Addr:Pointer;Len:Longint):longint;
  2.  
  3. var j : cint;
  4. Begin
  5.   repeat
  6.     Do_Write:=Fpwrite(Handle,addr,len);
  7.     j:=geterrno;
  8.   until (do_write<>-1) or ((j<>ESysEINTR) and (j<>ESysEAgain));
  9.   If Do_Write<0 Then
  10.    Begin
  11.     Errno2InOutRes;
  12.     Do_Write:=0;
  13.    End
  14.   else
  15.    InOutRes:=0;
  16. End;
  17.  

IOW EINTR and EAGAIN are not "real" errors but things that can happen in normal program flow

IIRC this is an scheme used in older Unices, but Linux stuck to it because the supreme Penguin thought it to be elegant in its simplicity.
« Last Edit: March 22, 2022, 10:10:28 am by marcov »

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #9 on: March 22, 2022, 11:10:15 am »
On Linux, syscalls are not reliable. To avoid deadlock, the kernel can let them fail with EsysEINTR, and all syscalls should be guarded with repeat loops to restart the syscall in such case. See e.g. the write() call in rtl/unix/sysfile.inc  (see below)  and grep for eintr in that directory

I'm not arguing in the slightest but I thought the fp...() functions wrapped that retry automatically?

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 14204
  • Probably until I exterminate Putin.
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #10 on: March 22, 2022, 11:14:10 am »
I'm not arguing in the slightest but I thought the fp...() functions wrapped that retry automatically?
These are not wrappers as such at all. They are mostly merely straight header translations to Pascal syntax.
So if it does not work for FPC, it probably does not work on any other compiler for the same platform.
« Last Edit: March 22, 2022, 11:17:49 am by Thaddy »
Specialize a type, not a var.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #11 on: March 22, 2022, 11:45:59 am »
I'm not arguing in the slightest but I thought the fp...() functions wrapped that retry automatically?

I don't know why you thought that:
rtl/linux/ossysc.inc:

Code: Pascal  [Select][+][-]
  1. function Fpwrite(fd: cint;buf:pchar; nbytes : size_t): ssize_t; [public, alias : 'FPC_SYSC_WRITE'];
  2.  
  3. begin
  4.  Fpwrite:=do_syscall(syscall_nr_write,Fd,TSysParam(buf),nbytes);
  5. end;
  6.  

One of the problems of doing it in the above routine, is that if the runtime switches to libc (android?), it is still needed on a higher level.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #12 on: March 22, 2022, 12:04:54 pm »
Thanks both for the clarification re ESysEINTR etc. My suspicion is that I was confused by recollection of very old discussion of the distinction between libc and syscalls, explained succinctly by e.g. the somewhat more recent https://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html (via https://lists.freepascal.org/fpc-pascal/2017-August/051997.html).

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

han

  • Jr. Member
  • **
  • Posts: 96
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #13 on: April 21, 2022, 01:53:46 pm »
>>The updated lNet library is on GitHub here.

I'm glad somebody addressed some problems.  The Visual package still indicates V.0.6.6. Would it not be better to increase the version to 0.6.6a or 0.6.7?

Han

zamronypj

  • Full Member
  • ***
  • Posts: 133
    • Fano Framework, Free Pascal web application framework
Re: lNet updated for macOS, FreeBSD, Windows - Linux epoll problem
« Reply #14 on: April 23, 2022, 12:38:23 am »
@trev

I create issue https://github.com/trevoz/lnet/issues/1 regarding FPC_FULLVERSION number.

Regards,
Fano Framework, Free Pascal web application framework https://fanoframework.github.io
Apache module executes Pascal program like scripting language https://zamronypj.github.io/mod_pascal/
Github https://github.com/zamronypj

 

TinyPortal © 2005-2018