In need of a class/component that implements a peer in a P2P network

[Gustavo 'Gus' Carreno]

Hey Remy,

I'm in need of a class/component that would implement a peer in a P2P network. Something akin to TIdTPCPeer or the likes.

I think my need for help is in understanding if it would be a good idea to make a descendent of TIdTCPServer and then abuse it with methods for the client side of P2P, or would it be better to derive from some other class I'm unaware exists?

I say this because in the P2P context, the connections will have to have a flag that indicate if the connection is an UP one, or a DOWN one, something I'm guessing does not exist in the current implementation of the TCP Server.

I would love to read all the opinions you have on the possibility of the Indy package implementing and distributing a class that implements a peer in a P2P network.

Many thanks in advance!!

Cheers,
Gus

[MarkMLl]

Could we step back from the language (i.e. class etc.) aspect for a moment and consider what you're actually trying to move around? Also what sort of peer discovery do you need, and is this strictly local or to work across subnetworks?

Historically, I've used UDP broadcasts to locate collaborating programs (e.g. to share clipboard contents); however this was strictly (and intentionally) local since UDP broadcasts aren't normally routed between subnetworks.

MarkMLl

[Gustavo 'Gus' Carreno]

Hey Mark,

Could we step back from the language (i.e. class etc.) aspect for a moment and consider what you're actually trying to move around? Also what sort of peer discovery do you need, and is this strictly local or to work across subnetworks?

Here's my situation:

• I'm in the middle of an evaluation of a complete re-write of a cryptocurrency core node.
• Any cryptocurrency is based in a P2P model with messages as the basis of all the contents flowing through the pipe.
• I've looked before and apart from the various server components/classes that Indy and Synapse have, there is no support for a peer in a P2P network, be it for file downloads or just messages.

What I'm trying to move around is messages. We're thinking that our best bet would be protobuf.

This has to work in the open internet and through NAT.

Historically, I've used UDP broadcasts to locate collaborating programs (e.g. to share clipboard contents); however this was strictly (and intentionally) local since UDP broadcasts aren't normally routed between subnetworks.

The discovery of other nodes is usually not via network discovery, but via a list maintained by all the nodes that make the main network.
In other words: Think more like crypto, less like BitTorrent. This is not a scenario where we need to download files in chunks from various participating nodes.

I would absolutely love to also have the opportunity of having this same thing using UDP connections instead of TCP connections!!
But lets start with TCP first and then see if it can also be done using UDP.

Hope this answers your concerns.

Cheers,
Gus

[Remy Lebeau]

I think my need for help is in understanding if it would be a good idea to make a descendent of TIdTCPServer and then abuse it with methods for the client side of P2P, or would it be better to derive from some other class I'm unaware exists?

TCP makes a clear distinction between client (connect) and server (listen+accept) socket connections. But P2P doesn't. A peer may need to connect out to another peer, or be connected to by another peer. So you have to be prepared to handle both cases.

You are probably looking for the TIdSimpleServer component, which is a simple non-threaded TCP server that accepts only 1 client connection at a time.

For example, TIdFTP uses TIdTCPClient for Passive mode transfers (outbound connect), and uses TIdSimpleServer for Active mode transfers (inbound accept). It chooses which component to use for any given transfer, but the bulk of the transfer logic is the same regardless of which component is used to create the connection.

You can use a similar approach in a P2P scenario.

[MarkMLl]

You can use a similar approach in a P2P scenario.

There's obviously also various possibilities using multicast etc., although that won't necessarily traverse a domestic router. There's also the central question of the extent to which nodes can be allowed to get out of step, and "whose" responsibility it is to get them back in sync.

MarkMLl

[Gustavo 'Gus' Carreno]

Hey Remy,

First of all, let me thank you for your time spent answering my not so well formulated question. I could've made a better job in defining my needs, that's for sure.

TCP makes a clear distinction between client (connect) and server (listen+accept) socket connections. But P2P doesn't. A peer may need to connect out to another peer, or be connected to by another peer. So you have to be prepared to handle both cases.

You're absolutely correct in this.

My naive idea is to have the connect to, or what I call DOWN connections be handled by the usual Server methods of Listen+Accept.
Those would be added to a list that will contain both the UP(initiated) and DOWN(received) connections, with a flag attached to the connection so we know if it's an UP or DOWN one.
Then add some methods to allow for the programmer to feed the component, either a single or multiple addresses for the component to initiate connections, and add them to the common list properly flagged.

One thing I forgot to mention is that in a P2P model in the context of crypto, all connections are forever. These are the type of always on, as long as nothing goes wrong, or until the node quits.
Not sure if this influences anything. In a way, we could kinda compare them to websockets, kinda.

You are probably looking for the TIdSimpleServer component, which is a simple non-threaded TCP server that accepts only 1 client connection at a time. For example, TIdFTP uses TIdTCPClient for Passive mode (outbound) transfers, and TIdSimpleServer for Active mode (inbound) transfers. It chooses which component to use for a transfer, but the bulk of the transfer logic is the same regardless of which component is used to create the connection.

You can use a similar approach in a P2P scenario.

Hummm... Non-threaded is probably not the best solution since a lot of parallel tasks must be performed while the P2P peer code is also doing it's thing.

Nonetheless those are good hints on trying to investigate a solution for my dilemma.

Now, let's say I would take upon myself the task to create such a thing:

• Can you please point me at the best LEGO pieces I could use, mainly in terms of the connection list and the connection data classes?
• Can you please point me at the best implementation of the Listen+Accept code in any of the current Server implementations?
• Can you suggest some bullet points on how I could then apply some thoughtful glue to all this?

Again, so many thanks in advance for any answer you may provide.

Cheers,
Gus

[Gustavo 'Gus' Carreno]

Hey Mark,

There's obviously also various possibilities using multicast etc., although that won't necessarily traverse a domestic router. There's also the central question of the extent to which nodes can be allowed to get out of step, and "whose" responsibility it is to get them back in sync.

From the little that I know, the P2P thang, doesn't need to have this type of node management.

If I'm not mistaken, the assessment of a node's behaviour is made outside the P2P thang.
A lot of this behaviour management is tied to shenanigans that could be considered illegal or attacks to the integrity of the blockchain.

But I do see the possibility of managing the fact that if a node is spamming the network, DDoS like, then some throttling may be done by the P2P thang itself.
These are details that I would probably need to implement once I do have a solid implementation, that works, but then needs the bells and whistles to be added.

Cheers,
Gus

[Remy Lebeau]

Hummm... Non-threaded is probably not the best solution since a lot of parallel tasks must be performed while the P2P peer code is also doing it's thing.

TIdSimpleServer is not threaded because it doesn't need to be threaded. Just as TIdTCPClient is not threaded.  They both service a single connection.  YOU decide what to do with that connection once it has been established.  If you want to thread it yourself, you can do so.

That being said, since TIdSimpleServer is just a single connection, that means if you want multiple peers to be able to connect to you concurrently, then you would have to create multiple TIdSimpleServer instances on separate ports, or else use TIdTCPServer instead on a single port.

Now, let's say I would take upon myself the task to create such a thing:

• Can you please point me at the best LEGO pieces I could use, mainly in terms of the connection list and the connection data classes?
• Can you please point me at the best implementation of the Listen+Accept code in any of the current Server implementations?
• Can you suggest some bullet points on how I could then apply some thoughtful glue to all this?

I'm not really sure what you are asking for.  But I think most of that is likely going to be your responsibility to implement to meet your specific needs.

[Gustavo 'Gus' Carreno]

Hey Remy,

That being said, since TIdSimpleServer is just a single connection, that means if you want multiple peers to be able to connect to you concurrently, then you would have to create multiple TIdSimpleServer instances on separate ports, or else use TIdTCPServer instead on a single port.

Okidokes, this is a good starting point, thanks.

Now, let's say I would take upon myself the task to create such a thing:

• Can you please point me at the best LEGO pieces I could use, mainly in terms of the connection list and the connection data classes?
• Can you please point me at the best implementation of the Listen+Accept code in any of the current Server implementations?
• Can you suggest some bullet points on how I could then apply some thoughtful glue to all this?

I'm not really sure what you are asking for.  But I think most of that is likely going to be your responsibility to implement to meet your specific needs.

My idea was to create a class/component that would fit nicely in the current list of components that Indy currently offers.
I would prefer that all this came from a more experience developer with enough knowledge of the internals of Indy, someone like you, but I'm also aware that this is a big ask and maybe I should take it upon myself.
This is why I'm asking for your guidance in order to use the available LEGO pieces that Indy has in terms of thread safe lists, connection data classes and any other element that would make this an official Indy component.
In the end I would probably do a PR to have it added to the official repository on GitHub.
Hope this clarifies this.

Cheers,
Gus