Recent

Author Topic: Basic Data Exchange over home network  (Read 3468 times)

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Basic Data Exchange over home network
« on: May 09, 2021, 08:18:15 pm »
I have a RPi4 project using Ultibo RTL (no OS).

That end of the project is generating a lot of data and I'm not happy with using the SD card to 'import' it to the Mac or same to transport updated run time program to the Pi.

I'd like a bare bones ethernet solution where my Mac can send/receive arbitrary data from the Pi and v-v.  (I'd be writing the code for each machine). 

Call the Mac program Boss and the Pi program Sensor.

I'm not sure if TCP/IP is too much or if UDP is too little.  In the later case I can easily implement checksumming and such to verify the data.

I have no idea how to do this.

Is there an absolutely bare bones FP demo of using UDP or TCP/IP on a local network between two machines?

The local network has a WiFi-modem-router and a switch in between the two machines.  This is all wired, not WiFi.

Thanks.

Update: I've found some code for the Ultibo RTL unit called Winsock2.  It compiled and is allegedly sending a test message 1/second to my Mac from the Pi.  (Local IP and port).

I see mention here of INet, but it won't "uses" on the mac (FPC compiler).... hmm - seems to be for Windows and Linux, not Mac.  !
« Last Edit: May 09, 2021, 09:41:50 pm by AlanTheBeast »
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2687
Re: Basic Data Exchange over home network
« Reply #1 on: May 09, 2021, 10:18:21 pm »
Are you moving packets of data described by a fixed-size record or streams?

MarkMLl

p.s. You've probably seen my considered opinion of LNet elsewhere.
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Re: Basic Data Exchange over home network
« Reply #2 on: May 09, 2021, 10:37:17 pm »
Fixed size packets is fine.  That is to say the packets may vary in size from packet to packet, but not be a stream.

I have not seen your opinion but your tone says it all...
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

PierceNg

  • Full Member
  • ***
  • Posts: 110
Re: Basic Data Exchange over home network
« Reply #3 on: May 10, 2021, 03:58:17 am »
I have a RPi4 project using Ultibo RTL (no OS).
I'd like a bare bones ethernet solution where my Mac can send/receive arbitrary data from the Pi and v-v.  (I'd be writing the code for each machine). 

MQTT https://mqtt.org/ is designed for IoT networking. Here's an implementation for Ultibo: https://github.com/pjde/ultibo-mqtt. Search the forum for threads of MQTT for FPC / Lazarus on the usual 'regular' operating systems.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2687
Re: Basic Data Exchange over home network
« Reply #4 on: May 10, 2021, 09:31:49 am »
Fixed size packets is fine.  That is to say the packets may vary in size from packet to packet, but not be a stream.

In that case either follow @PierceNg and use whatever you can find available, or stick to UDP which is a very thin wrapper over IP which is a very thin wrapper over Ethernet or SLIP. Avoid TCP unless you really need it, it's a rabbit hole.

LNet might have been OK before threads were universally implemented, but is somewhat... abstruse. I use it for Telnet and have added a couple of bits to the variant bundled as a tool with FPC, but there are some nasty bugs in there which have defied resolution and if I need to do more I'll quite simply reimplement Telnet from scratch.

MarkML
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Re: Basic Data Exchange over home network
« Reply #5 on: May 10, 2021, 09:44:37 pm »
MQTT https://mqtt.org/ is designed for IoT networking.

Thanks!

In that case either follow @PierceNg and use whatever you can find available, or stick to UDP which is a very thin wrapper over IP which is a very thin wrapper over Ethernet

Thanks!

Probably get to this on Friday.
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2687
Re: Basic Data Exchange over home network
« Reply #6 on: May 10, 2021, 10:29:47 pm »
The problem with TCP- and I note that MQTT uses it- is that while the trivial protocol isn't too bad in practice it relies on a very large number of "options". There is truly vast scope for error, to the extent that between its inception in the 1970s and (say) 2010 not just some but /most/ implementations had bugs that could be exploited for intrusion or a (D)DoS.

Saying that it's for strictly local use so it doesn't have to be robust is, as I'm quite sure you're aware, no defence.

I don't know what would be involved in an "SCTP-lite", i.e. a minimal implementation for something like Ultibo, but it might be worth reviewing.

Later: sniffing around for "SCTP-Lite", https://www.pdl.cmu.edu/mailinglists/ips/mail/msg03648.html discusses the addition of framing on top of TCP, but this would obviously assume all the baggage of TCP. There's obviously been no movement on that in the intervening 19 (!) years, but that's not to say that there isn't a need for it.

So what have I got against using TCT where it's not needed? These for a start:

https://lcamtuf.coredump.cx/oldtcp/tcpseq.html
https://lcamtuf.coredump.cx/newtcp/

/If/ I had time and incentive to look at this (which I don't), I think I'd set up MQTT on two PCs using SCTP and then investigate what parts of the protocol were actually needed. But https://www.uni-due.de/~be0001/sctplib/index.html or http://www.sctp.de/sctp-download.html would probably be a good start.

MarkMLl
« Last Edit: May 11, 2021, 03:06:33 pm by MarkMLl »
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Re: Basic Data Exchange over home network
« Reply #7 on: May 15, 2021, 01:54:43 pm »
Saying that it's for strictly local use so it doesn't have to be robust is, as I'm quite sure you're aware, no defence.


Not sure what you're alluding to, but what I am doing is a project where I need some very basic communication between some Pi's and my computer.  This is not release code, but tooling for my own needs.  For data 'robustness' I will indeed be implementing a basic checksum such as Fletcher and keeping the blocks small (256 byte or so) with a mechanism to signal bad blocks back to the sender.  Not sure if I'll throw block serial numbering and recovery on top of that.  The files I'll be transporting are pretty small generally.

I could do something basic where the sender sends an arbitrary number of blocks, say 10 and not send anymore until it receives a 'pass' for a block by seq number.  If it receives a pass, then send a new block; if it receives a fail, then send a replacement for the block that failed.  If a timeout expires, then abandon the transfer.

Or, given that the file sizes will be pretty small (several MB at most), just 1 block at a time, repeating blocks if failed and a general timeout.
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2687
Re: Basic Data Exchange over home network
« Reply #8 on: May 15, 2021, 02:14:57 pm »
Saying that it's for strictly local use so it doesn't have to be robust is, as I'm quite sure you're aware, no defence.
Not sure what you're alluding to, but what I am doing is a project where I need some very basic communication between some Pi's and my computer.  This is not release code, but tooling for my own needs.

Come on old chap. You've been knocking around industry for long enough to know that roughly 50% of the software (and much hardware) in circulation is stuff knocked together on those terms, which somebody (else) has then pressed into service where it's publicly visible.

https://www.theregister.com/2020/04/27/who_me/

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Re: Basic Data Exchange over home network
« Reply #9 on: May 15, 2021, 02:27:31 pm »
I have a RPi4 project using Ultibo RTL (no OS).
I'd like a bare bones ethernet solution where my Mac can send/receive arbitrary data from the Pi and v-v.  (I'd be writing the code for each machine). 

MQTT https://mqtt.org/ is designed for IoT networking. Here's an implementation for Ultibo: https://github.com/pjde/ultibo-mqtt. Search the forum for threads of MQTT for FPC / Lazarus on the usual 'regular' operating systems.

In the end that's way over the top of what I want to do.

I really need something very basic on the Mac end to do UDP in Free Pascal.  I'll implement what I need under that.
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Re: Basic Data Exchange over home network
« Reply #10 on: May 15, 2021, 08:52:33 pm »

Come on old chap. You've been knocking around industry for long enough to know that roughly 50% of the software (and much hardware) in circulation is stuff knocked together on those terms, which somebody (else) has then pressed into service where it's publicly visible.

https://www.theregister.com/2020/04/27/who_me/

Amusing story.  But really - all I want is the simplest of procedures or functions that I can use to listen to the other machine and talk to it over UDP.

This is what I have for the Pi to get going that allegedly works (it compiles and is threaded ... whether anything is getting to the Mac... not sure).

What I can't figure out is an equivalent for the Mac side...

Code: Pascal  [Select][+][-]
  1. procedure BasicUDP(p:pointer);
  2. const
  3.  i:integer=0;
  4. var
  5.  buf:string;
  6.  IPAddress:string;
  7.  Winsock2UDPClient:TWinsock2UDPClient;
  8.  
  9. begin
  10. Winsock2UDPClient:=TWinsock2UDPClient.Create;
  11. repeat  IPAddress:=Winsock2UDPClient.LocalAddress; sleep(1000) until ipaddress<>'';
  12.  
  13. Winsock2UDPClient.RemoteHost := '10.7.219.69';
  14. Winsock2UDPClient.RemotePort := StrToInt('61013');
  15. Winsock2UDPClient.Connect;
  16. repeat
  17.   i:=i+1;
  18.   buf:='Test string '+inttostr(i);
  19.   Winsock2UDPClient.SendData(PChar(buf), Length(buf));
  20.   sleep(1000);
  21. until false;
  22. end;
  23.  
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2687
Re: Basic Data Exchange over home network
« Reply #11 on: May 15, 2021, 09:46:06 pm »
I agree that UDP's a good bet subject to a very thin ack/nak protocol layer. Remember that it's a connectionless protocol, so what you need is something like

Code: Pascal  [Select][+][-]
  1. (* The result of this may be subsequently retrieved using InetDomainSocket().
  2.   Assume that any existing socket was deleted when its owner terminated. Create
  3.   a fresh Inet Domain socket and save the handle, on failure save an error code.
  4.   Both success and failure are silent.
  5. *)
  6. procedure CreateInetDomainSocket(port: integer);
  7.  
  8. var
  9.   sockAddr: TSockAddr;
  10.  
  11. begin
  12.  
  13. // TODO : This could usefully announce what IP addresses are being listened on.
  14.  
  15. // Scenario: the host computer is restarted, and comes up with a DHCP-allocated
  16. // address which no longer corresponds to that being assumed by other WatchXxx
  17. // instances assuming it's the supervisor.
  18.  
  19.   Write(StdErr, 'Creating new Internet domain socket (UDP, port ', port, ')... ');
  20.   inetDomainSocketHandle := fpSocket(PF_INET, SOCK_DGRAM, 0);
  21.   try
  22.     sockAddr.sin_family:= AF_INET;
  23.     sockAddr.sin_port:= hToNS(port);
  24.     sockAddr.sin_addr.s_addr:= 0;
  25.     FillChar(sockAddr.sin_zero[0], SIZEOF(sockAddr.sin_zero), #0);
  26.     if fpBind(inetDomainSocketHandle, @sockAddr, SIZEOF(sockAddr)) = 0 then
  27.       WriteLn(StdErr, 'OK')
  28.     else begin
  29.       WriteLn(StdErr, 'failed, might need CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_NET_RAW=p+e');
  30.       inetDomainSocketHandle := -Abs(ErrNo)
  31.     end
  32.   except
  33.     WriteLn(StdErr, 'failed, might need CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_NET_RAW=p+e');
  34.     inetDomainSocketHandle := -Abs(ErrNo)
  35.   end
  36.  
  37. (* Assume that this will be deleted automatically at program termination.       *)
  38.  
  39. end { CreateInetDomainSocket } ;
  40.  

from which you can then read using something like

Code: Pascal  [Select][+][-]
  1.   FillByte(recvBuffer, SizeOf(recvBuffer), 0);
  2.     FillByte(sa, SizeOf(sa), 0);
  3.     saLength := SizeOf(sa);
  4.     messageLength := fpRecvFrom(socket, @recvBuffer, SizeOf(recvBuffer), 0, @sa, @saLength)
  5.     sender := NetAddrToStr(sa.sin_addr) + ':' + IntToStr(NToHS(sa.sin_port));
  6.  

which will normally be inside a thread, or given a timeout using fpSelect().

MarkMLl
« Last Edit: May 15, 2021, 10:16:26 pm by MarkMLl »
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

AlanTheBeast

  • Full Member
  • ***
  • Posts: 192
  • My software never cras....
Re: Basic Data Exchange over home network
« Reply #12 on: May 15, 2021, 10:09:12 pm »
Thanks Mark,

I'll get on that in the morning.
Everyone talks about the weather but nobody does anything about it.
..Samuel Clemens.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2687
Re: Basic Data Exchange over home network
« Reply #13 on: May 16, 2021, 10:16:10 am »
Note that as part of your setup protocol you'll need to check how long UDP datagrams can be. This tends to vary depending on what /exactly/ is in your network path, but in the worst case you'll find them truncated to somewhere around 1.5K and you might usefully consider the situation when the length changes in the middle of a session. Also consider "sliding window" for your acks... you can almost certainly do without it but you can usefully know about it.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018