Forum > Linux

DNS "Query" in netdb uses fpSelect

(1/3) > >>

Molochnik:
While testing network application on Linux with a lot of open sockets/files I encountered with the problem that DNS is resolved using fpSelect which definitely fails when it comes down to Linux with more than 1000 open files. Is it possible to use fpPoll instead?

marcov:

--- Quote from: Molochnik on January 22, 2024, 10:40:38 am ---While testing network application on Linux with a lot of open sockets/files I encountered with the problem that DNS is resolved using fpSelect which definitely fails when it comes down to Linux with more than 1000 open files. Is it possible to use fpPoll instead?

--- End quote ---

Which resolver do you use? It might be complicated since fpselect is portable, and epoll not.

Having 1000 simultaneous DNS requests that are not cached somehow ? What are you, Facebook?  :)

MarkMLl:
A DNS query is just a couple of (usually) UDP messages, so you can any algorithm you like: but not necessarily with standard components and libraries.

I've been in a not dissimilar situation, when I had a minor brainstorm and forgot to close the sockets associated with connectionless UDP messages.

However if you really /do/ have to have this number of sessions open you would be far better off investigating spreading them over a pool of a few score processes. I emphasise *processes* here, since unlike *threads* they don't share memory so the OS would find it easier to spread them over multiple cores and processors.

A few months ago I wrote a port scanner which needed to apply more contextual filtering than was realistic with e.g. Nmap, and found that performance was adequate if I had a handful of threads each looking at a couple of dozen ports. But again I'd emphasise that this was written from the ground up, without any attempt to shoehorn it into existing components or libraries.

MarkMLl

Molochnik:

--- Quote from: marcov on January 22, 2024, 11:13:07 am ---Having 1000 simultaneous DNS requests that are not cached somehow ? What are you, Facebook?  :)

--- End quote ---
No,  :) if you have 1000 sockets already opened then one DNS resolving is enough to crash the whole program with SIGSEGV. FPC DNS resolver is used by Indy, I already replaced its explicit calls to fpSelect, but it uses some more from FPC utility functions.
I am not Facebook, just a small Avaya compe..., no you cant compete with anybody if you program crashes from a small loadup.

marcov:

--- Quote from: Molochnik on January 22, 2024, 02:15:15 pm ---
--- Quote from: marcov on January 22, 2024, 11:13:07 am ---Having 1000 simultaneous DNS requests that are not cached somehow ? What are you, Facebook?  :)

--- End quote ---
No,  :) if you have 1000 sockets already opened then one DNS resolving is enough to crash the whole program with SIGSEGV. FPC DNS resolver is used by Indy, I already replaced its explicit calls to fpSelect, but it uses some more from FPC utility functions.

--- End quote ---

But that should be a different fpselect (the one from the resolver), and each individual fpselect should be able to do 1024 handles, not overall in the system.

So indy socket fpselects and the dns stuff shouldn't be in one select ? ?!?! Or at least not in a FPC code one, since it doesn't know about indy.


--- Quote ---I am not Facebook, just a small Avaya compe..., no you cant compete with anybody if you program crashes from a small loadup.

--- End quote ---

I understand, but I think this is not the main issue. Something else is, and that will need to be found out. (strace?!?)

Navigation

[0] Message Index

[#] Next page

Go to full version