Recent

Author Topic: DNS "Query" in netdb uses fpSelect  (Read 1253 times)

Molochnik

  • Jr. Member
  • **
  • Posts: 76
DNS "Query" in netdb uses fpSelect
« on: January 22, 2024, 10:40:38 am »
While testing network application on Linux with a lot of open sockets/files I encountered with the problem that DNS is resolved using fpSelect which definitely fails when it comes down to Linux with more than 1000 open files. Is it possible to use fpPoll instead?
« Last Edit: January 22, 2024, 10:43:42 am by Molochnik »

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11537
  • FPC developer.
Re: DNS "Query" in netdb uses fpSelect
« Reply #1 on: January 22, 2024, 11:13:07 am »
While testing network application on Linux with a lot of open sockets/files I encountered with the problem that DNS is resolved using fpSelect which definitely fails when it comes down to Linux with more than 1000 open files. Is it possible to use fpPoll instead?

Which resolver do you use? It might be complicated since fpselect is portable, and epoll not.

Having 1000 simultaneous DNS requests that are not cached somehow ? What are you, Facebook?  :)

MarkMLl

  • Hero Member
  • *****
  • Posts: 6763
Re: DNS "Query" in netdb uses fpSelect
« Reply #2 on: January 22, 2024, 02:02:23 pm »
A DNS query is just a couple of (usually) UDP messages, so you can any algorithm you like: but not necessarily with standard components and libraries.

I've been in a not dissimilar situation, when I had a minor brainstorm and forgot to close the sockets associated with connectionless UDP messages.

However if you really /do/ have to have this number of sessions open you would be far better off investigating spreading them over a pool of a few score processes. I emphasise *processes* here, since unlike *threads* they don't share memory so the OS would find it easier to spread them over multiple cores and processors.

A few months ago I wrote a port scanner which needed to apply more contextual filtering than was realistic with e.g. Nmap, and found that performance was adequate if I had a handful of threads each looking at a couple of dozen ports. But again I'd emphasise that this was written from the ground up, without any attempt to shoehorn it into existing components or libraries.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Molochnik

  • Jr. Member
  • **
  • Posts: 76
Re: DNS "Query" in netdb uses fpSelect
« Reply #3 on: January 22, 2024, 02:15:15 pm »
Having 1000 simultaneous DNS requests that are not cached somehow ? What are you, Facebook?  :)
No,  :) if you have 1000 sockets already opened then one DNS resolving is enough to crash the whole program with SIGSEGV. FPC DNS resolver is used by Indy, I already replaced its explicit calls to fpSelect, but it uses some more from FPC utility functions.
I am not Facebook, just a small Avaya compe..., no you cant compete with anybody if you program crashes from a small loadup.
« Last Edit: January 22, 2024, 02:32:10 pm by Molochnik »

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11537
  • FPC developer.
Re: DNS "Query" in netdb uses fpSelect
« Reply #4 on: January 22, 2024, 02:57:36 pm »
Having 1000 simultaneous DNS requests that are not cached somehow ? What are you, Facebook?  :)
No,  :) if you have 1000 sockets already opened then one DNS resolving is enough to crash the whole program with SIGSEGV. FPC DNS resolver is used by Indy, I already replaced its explicit calls to fpSelect, but it uses some more from FPC utility functions.

But that should be a different fpselect (the one from the resolver), and each individual fpselect should be able to do 1024 handles, not overall in the system.

So indy socket fpselects and the dns stuff shouldn't be in one select ? ?!?! Or at least not in a FPC code one, since it doesn't know about indy.

Quote
I am not Facebook, just a small Avaya compe..., no you cant compete with anybody if you program crashes from a small loadup.

I understand, but I think this is not the main issue. Something else is, and that will need to be found out. (strace?!?)

Molochnik

  • Jr. Member
  • **
  • Posts: 76
Re: DNS "Query" in netdb uses fpSelect
« Reply #5 on: January 22, 2024, 04:54:22 pm »
marcov
:) Yes its hard to believe that, i didn't believe it either at first but actually its true. If you opened e.g.1200 sockets and issue "polls" you work normally, but just another single "select" on a single socket will crush your program.

P.S I found the problem with DNS resolving when tried to understand why the program crashes from a single HTTP request
« Last Edit: January 22, 2024, 05:08:55 pm by Molochnik »

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11537
  • FPC developer.
Re: DNS "Query" in netdb uses fpSelect
« Reply #6 on: January 22, 2024, 05:12:00 pm »
I'm not denying your problem, I'm just shedding some doubt on your analysis of it that it is fpselect in the DNS resolver. Unless you have 1200 uncached resolves resulting from your 1200 sockets, like a SMTP bot that has to find SMTP servers for 1200 email addresses.


Thaddy

  • Hero Member
  • *****
  • Posts: 14659
  • Sensorship about opinions does not belong here.
Re: DNS "Query" in netdb uses fpSelect
« Reply #7 on: January 22, 2024, 05:25:42 pm »
It is not fpselect, it is the number of connections that a select can handle per thread as was already mentioned. That is an OS setting/limitation, not a limitation of FreePascal. fpselect is just an alias for select, It is a direct alias, which is an OS call on UNIX/POSIX: use threads and make sure you cache even with threads.

A OS server version install may give you more head-room, but you still need to use threads and caches.
(and distribution of available cores)

I believe the lmit is 1024 connections, at least in the past, but also subject to connection load.
You can't expect a single core CPU to deliver the same number of connections as a rack full of 32 core CPU's with each 64 GB of memory per core.
« Last Edit: January 22, 2024, 05:47:17 pm by Thaddy »
bitrate is always calculated like this:sample rate * bitdepth * number of channels.

Molochnik

  • Jr. Member
  • **
  • Posts: 76
Re: DNS "Query" in netdb uses fpSelect
« Reply #8 on: January 22, 2024, 05:42:25 pm »
marcov
Right now it's exactly the DNS resolver, I spent some time with debugger trying to locate it. Indy uses ResolveName in IdStackUnix. Of course using a resolved IP address directly instead of the Name hides the issue.
« Last Edit: January 22, 2024, 05:54:35 pm by Molochnik »

Thaddy

  • Hero Member
  • *****
  • Posts: 14659
  • Sensorship about opinions does not belong here.
Re: DNS "Query" in netdb uses fpSelect
« Reply #9 on: January 22, 2024, 05:48:38 pm »
No, posts crosses. There is a finite amount of connections per core. Maybe that can be configured, though, but I would not bet on it. It is not the DNS resolver PerSe', because that normally is the front of a server stack and has virtually no load per connection, because it distributes. All about infrastructure. E.g. use the google DNS resolvers that work like that: 8.8.4.4 and 8.8.8.8. Local DNS resolvers can usually not handle big loads, unless the infrastructure is there, like in a proper bank.. :D
« Last Edit: January 22, 2024, 05:54:49 pm by Thaddy »
bitrate is always calculated like this:sample rate * bitdepth * number of channels.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6763
Re: DNS "Query" in netdb uses fpSelect
« Reply #10 on: January 22, 2024, 05:52:11 pm »
It is not fpselect, it is the number of connections that a select can handle per thread as was already mentioned. That is an OS setting/limitation, not a limitation of FreePascal. fpselect is just an alias for select, It is a direct alias, which is an OS call on UNIX/POSIX: use threads and make sure you cache even with threads.

A process- not a thread- has a handle limit, and every open UDP socket consumes a handle. BTDT.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 14659
  • Sensorship about opinions does not belong here.
Re: DNS "Query" in netdb uses fpSelect
« Reply #11 on: January 22, 2024, 05:55:49 pm »
Mark, I should have written it as you did... correct.
bitrate is always calculated like this:sample rate * bitdepth * number of channels.

 

TinyPortal © 2005-2018