Lazarus

Programming => Operating Systems => Linux => Topic started by: MarkMLl on October 08, 2021, 08:59:53 am

Title: Linux serial comms: a heads-up
Post by: MarkMLl on October 08, 2021, 08:59:53 am
Everybody agrees that the days of the RS232 serial port are past, but serial comms APIs still tend to be used heavily when talking to USB-connected equipment: bench meters and so on, often via an FTDI chip. In the particular case I've got here I've got an HP/IB/GP-IB/IEEE-488 interface that looks like a CDC device.

In conjunction with the standard serial.pp unit, this code may be used to check how much data the OS has buffered prior to a SerRead() etc.:

Code: Pascal  [Select][+][-]
  1. function SerAvailable(handle: TSerialHandle): integer;
  2.  
  3. begin
  4.   if handle < 0 then
  5.     result := -1
  6.   else begin
  7.     if fpIoctl(handle, FIONREAD, @result) <> 0 then
  8.       result := -1
  9.   end
  10. end { SerAvailable } ;
  11.  

However the result appears to be limited to the range 0..4095, and this appears to be a strict limit rather than a truncated number of bits. Obviously this can have implications if trying to deal with large blocks efficiently.

Above on Linux 4.19 x86_64, other platforms untested but probably affected.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: geraldholdsworth on October 18, 2021, 02:01:33 pm
Everybody agrees that the days of the RS232 serial port are past
Not everybody - we regularly install AV equipment which communicates over RS232. OK, it may be past in the PC world, but is still very much still around.
Title: Re: Linux serial comms: a heads-up
Post by: PascalDragon on October 19, 2021, 09:01:23 am
Everybody agrees that the days of the RS232 serial port are past,

At work we use this day in day out for debugging our software (which runs standalone without an OS) on various systems. So I can't agree here. ;)
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 19, 2021, 09:25:18 am
:-) I've got RS-232 interfaces to modems and possibly a UPS, but apart from that I find embedded USB serial chips (including many fake FTDI) much more common. Even the HP-IB interface to my logic analyser and 'scope looks like one.

In any event, it would probably be wisest to assume that this is a general serial API limitation, rather than being driver-specific. And while 4K serial data blocks are probably fairly rare, I noticed this when parsing a data dump arriving from an instrument that didn't use handshaking... the transfer itself is fine, it's only that one ioctl that is a problem.

Edited after consideration: As far as I know it's only that call that's a problem. Buffered incoming data is definitely not being lost, and I presume that somebody would have spotted if Linux syscalls were size-limited.

This only came to light when I wrote some functions to allow backtracking, i.e. to allow me to put data back into a local buffer for a different parser to try. So if you've just put a few bytes back, you might need to be able to find out if you can read a 5K block yet.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: SymbolicFrank on October 19, 2021, 10:19:58 am
You can do: TStringList.WriteToFile("COM1").
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 19, 2021, 11:08:58 am
No I can't.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: af0815 on October 19, 2021, 11:32:14 am
I see at https://www.freepascal.org/docs-html/rtl/baseunix/fpioctl.html and https://man7.org/linux/man-pages/man4/tty_ioctl.4.html the argument is int. So this is not the limitation.

is the limitation for 0..4095 by system ? Can the Hardware/inputbuffer have more than 4095 Bytes ?!




Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 19, 2021, 12:00:00 pm
I see at https://www.freepascal.org/docs-html/rtl/baseunix/fpioctl.html and https://man7.org/linux/man-pages/man4/tty_ioctl.4.html the argument is int. So this is not the limitation.

is the limitation for 0..4095 by system ? Can the Hardware/inputbuffer have more than 4095 Bytes ?!

Only tested on x86_64 (Debian Linux) at the moment, and it appears to be that one specific ioctl: as I've said, no incoming data is lost. If I were making a wild guess I's say that it means "I'm using more than one page in a linked list for this, and you surely don't want me to walk the whole kirottu thing?".

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: SymbolicFrank on October 19, 2021, 12:39:05 pm
No I can't.

MarkMLl

Well, ok, like this:

Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. uses Sysutils, Classes;
  4.  
  5. var
  6.   MyList: TStringList;
  7. begin
  8.   MyList := TStringList.Create;
  9.   MyList.Add('Bla');
  10.   MyList.SaveToFile('COM1');
  11. end.    
  12.  

And only on Windows, if the port is used and opened.

Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 19, 2021, 12:46:25 pm
@SymbolicFrank, with apologies to the mods I'm about to waste far more time on you than you deserve.

Did you bother to read the OP, where was made quite clear it was a Linux problem?

Did you bother to read the OP, where was made quite clear it was a problem with one specific ioctl?

Did you bother to read the OP, where was made quite clear that the ioctl was used to read the amount of input data currently buffered?

If you can't be bothered to read the OP, then I suggest that reading https://monologues.co.uk/004/Owl-Critic.htm would be appropriate.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: SymbolicFrank on October 19, 2021, 12:50:17 pm
Hi Mark,

I did. There was a discussion about RS-232 being past, to which I also contributed.

Yes, the last time I tried to communicate with a specific USB device in Linux, it was definitely non-trivial. And no, I cannot help you with the bug.
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 19, 2021, 01:13:21 pm
Yes, the last time I tried to communicate with a specific USB device in Linux, it was definitely non-trivial.

The communications as such isn't a problem: serial.pp works well for serial devices and in general libusb works well... in both cases subject to access rights.

If I had time to do a bit of maintenance (or more specifically, the required testing on multiple OSes) on serial.pp I'd change it to (a) lock the port by default since that's Windows behaviour and is generally what's wanted and (b) add that "what's buffered?" function if non-Linux unix variants supported it.

Walking the /sys tree is a hassle, since different drivers can put things in slightly different places... cdc_acm appearing to be one specific problem. However I've now checked over the code I've previously posted at e.g. https://github.com/MarkMLl/hp2671/blob/main/hp2671/locatecdcacmport.pas to centralise its handling of different types of chip, and it's definitely possible- albeit arduous- to have a general solution.

I believe that I've mentioned in another thread that FTDI chips have a unique serial number in their EEPROM, which makes them particularly useful since e.g. two ends of a comms link can be easily distinguished during testing. Unfortunately the majority of "FTDI" chips are actually counterfeits, and in at least some cases the standard utilities can't rewrite the configuration... I need to take a debugger to the code involved (it's C, which is not something I'm particularly looking forward to) to see if that's a mere software problem or if the counterfeit firmware had its R/W capability removed after FTDI bricked a whole lot of questionable devices.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 20, 2021, 01:14:55 pm
If I had time to do a bit of maintenance (or more specifically, the required testing on multiple OSes) on serial.pp I'd change it to (a) lock the port by default since that's Windows behaviour and is generally what's wanted and (b) add that "what's buffered?" function if non-Linux unix variants supported it.

Also I've just spotted that (c) SerSetParams() starts off from a zeroed state rather than getting the current line state which is considered to be correct practice. Somehow my system's in a state where by default incoming CR is changed into NL, which sent me looking at initialisation...

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: SymbolicFrank on October 20, 2021, 03:10:18 pm
Yes, locating the right USB device, accessing and locking it is the main problem.

IIRC, long ago, with Unix, you had serial and block devices. You could directly copy to and from them, like they were files. With some restrictions, the serial devices were like pipes. Linux didn't do that at first, although they made an effort to make it like that about a decade ago. The status- and info-files are a great example. And a few years ago they abandoned that idea again and made separate device classes for each type of interface. Flexible, yes, but you need different tools and libraries for each one.

The only thing that survived it all is the Berkeley TCP/IP stack and sockets, which are part of every OS that has internet that I know of.

Ok, the main culprit for the complexity of USB devices is Microsoft, who wanted everything to be their vision of Plug-and-Play. And it is clearly designed by a committee. But I think KISS should be used for all layers, not just the top one.
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 20, 2021, 03:32:54 pm
No, Linux has always had character and block devices: it's a fundamental unix-like feature. It's obviously also always had an IP stack with TCP and UDP on top (I can't say when it stopped badging that as Swansea-derived NET4), and I think it had unix-domain sockets etc. fairly early as well. I'm pretty sure that its serial handling was orthodox, except that setserial had some PC-only facilities which would quite simply have been meaningless on e.g. an Ethernet-connected terminal server.

The two areas where it is rather odd are excessively-heavy use of the /proc tree (compare that with SunOS etc. where it is far more process-oriented) and increasingly-heavy emphasis on named device files being there purely to provide a handle for subsequent ioctl calls. The latter in particular is very much "un-unix" and in particular "un-Plan9", since it moves the APIs progressively further from the "everything is a file" model. Then there are things that have never been in the filesystem namespace at all, like network interfaces.

I'm all for purity and uniformity, although it gets very difficult to imagine how you could implement a rich API like OpenGL as a file. Unless, of course, you reverted to something like Display Postscript... which would probably be redefined these days in terms of XML.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: SymbolicFrank on October 20, 2021, 04:04:32 pm
Yes, I agree with that. We're just looking at it from a different perspective.

In C++, reading through the header-tree in Linux and comparing that to OS-X and Windows, the bottom of the stack is roughly the same, with some small changes in the Apple headers, all the way back to Palo Alto. Higher up, they start to diverge more and more.

The only *NIX-es I've worked with for the last 10 years or so are Linux and OS-X. Together with Windows, they all share a POSIX parent, although the implementations are quite different. For things like sockets and threads, it's best to stick to it anyway, but the implementations of the more modern stuff varies wildly.

The serial and block model are as old as computing, where they all differ is the "everything is a file" view, as you said. And that also changes over time. But even in Windows, at the core you see the same model, where most (but not everything) is a file-like entity in the (virtual) directory tree. But that is only really used by the kernel and hidden as much as possible from users and programmers.
Title: Re: Linux serial comms: a heads-up
Post by: PascalDragon on October 21, 2021, 08:59:59 am
But even in Windows, at the core you see the same model, where most (but not everything) is a file-like entity in the (virtual) directory tree. But that is only really used by the kernel and hidden as much as possible from users and programmers.

It's not file-like, but everything in NT is an object. Including inheritance. Somewhat.
Title: Re: Linux serial comms: a heads-up
Post by: SymbolicFrank on October 21, 2021, 02:11:11 pm
I once wrote a program (in Delphi or FPC/Laz) to communicate with a serial USB device in Windows, and at that level it works. It wasn't even all that complex, but almost completely undocumented. If you manage to find the object in that virtual tree, it behaves like a COM port. So it's mostly obfuscation.

I'll see if I can find that project.
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on October 21, 2021, 02:36:14 pm
I was the last person to do any maintenance on serial.pp, and when I put it on Mantis it included test programs which had run successfully on Linux, Solaris and Windows (but not Mac, which in my case I have not got). It definitely works using USB on Linux- I use it almost continually- and while I can't speak for USB on the other platforms I have no reason to think there would be a problem: it's just the standard serial comms API subject to Windows's strange naming conventions for ports numbered above 9.

MarkMLl
Title: Re: Linux serial comms: a heads-up
Post by: MarkMLl on November 01, 2021, 09:04:22 pm
Code: Pascal  [Select][+][-]
  1. function SerAvailable(handle: TSerialHandle): integer;
  2. ...
  3.     if fpIoctl(handle, FIONREAD, @result) <> 0 then
  4.  

That code is problematic since it makes no explicit check that the amount of data returned by the ioctl() matches the function return type.

I've changed the return type to longint and verified that it's filled by the ioctl(), but doing so has no effect on the underlying problem.

MarkMLl
TinyPortal © 2005-2018