504 Gateway Timeout

[Thaddy]

Besides this, there is the past months a different pattern in server load which I don't know yet how to handle.

I think that pattern is like I described above: many users are using A.I. during development and that means that multiple, seemingly inactive connections can be attributed, but not held, by the same user.
In the case of CoPilot, it acknowledged that it works like that, because it can't drop the connection without loosing context. DeepSeek seems to work similar, frankly I suspect all of them work like that if they cooperate with you: e.g. CoPilot also read responses on the forum during a session and sometimes even before I had the chance. It reads with me, not from me, it also reads from the server.
What you get is connection rot, especially with multiple tabs open in a browser or IDE. A browser tab needs to be specifically closed to drop those connections. A way to mitigate that is a "human" test.

The A.I. helpers do not keep context if a connection is dropped, only a bit client-side so by instinct or knowledge people tend to have the tabs open. (I do). This is easy to test with a simple server, connect, open a connection, ask copilot help with the content of that connection: connections raise from 1 to three, then 2. Close yours, all close.

[Marc]

That makes some sense

[Marc]

By me, this night the forum was still inacessible.

And now I can't use any link on the forum (login too): "Cookies error".

I had to go into my browser history to find the direct link to the login page, and now it seems to work.

  need to check this, based on the amount of traffic cookie-less traffic, I blocked more cases

[simone]

By me, this night the forum was still inacessible.

And now I can't use any link on the forum (login too): "Cookies error".

I had to go into my browser history to find the direct link to the login page, and now it seems to work.

I have the same problem, with the cookies page appearing, as happened in the past. To fix it, I have to manually deleting the PHPSESSIONID in the URL.

[rvk]

Bad requests are handled by fail2ban and other tools, so these IPs and networks get blocked. Fail2ban however has one problem I found out. It keeps all blocks in an internal sqlite database without removing entries. This DB has grown to 17GB, causing netblock detecting to fail.

Yikes. Maybe add a unique constraint on that table 
And find out why Fail2ban could try to add duplicates. If all is correct, that duplicate IP shouldn't even touch the server anymore (because it's already blocked). If it does, then there is something wrong with the blocking itself too. But maybe that's due to the problem with netblock.

How are the IP's blocked in Fail2ban? That's through iptables? (of nftables?)
How many IP's are there in the DB?

[Fibonacci]

I took a look at the server response headers:

Server: Apache/2.4.58 (Ubuntu)

A misconfigured Apache is trivial to knock over, and even a well-configured one can fold quite easily. I propose two options, from least invasive:

Variant 1 (low effort): put nginx in front of Apache as a reverse proxy. Apache still handles .htaccess and everything else - the forum config doesn't change.

Variant 2 (long-term): replace Apache entirely with nginx + php-fpm. Catch: if the forum uses .htaccess, those rules must be rewritten into the nginx config. That's why Variant 1 first - small effort, big payoff.

To put some weight behind the claim - I rented a "server by the hour" (so I wouldn't blacklist my own IP by accident) and ran literally two commands on it. The forum went down. Dead. I verified it wasn't responding, waited ~30 seconds, and ended the test.

And to be clear - not a botnet, nothing sophisticated. I can take this server down from a single PC on a connection slower than the forum's.

@Marc - if you give the green light, I'd demonstrate this publicly at a fixed time (suggest 12:00 CEST (10:00 UTC), so in an hour from now), for 5 minutes. Just to show the problem is the HTTP front end itself. Without that it's just "some guy on the forum claims something" - I'd rather show.

[Fibonacci]

After the demo I'll explain what's going on - and it's very simple.

Then, escalation order from cheapest to biggest effort:

1. Apache tuning - try an add-on module first. Install, tune, see if it holds. I'll give you the mod names, you do the rest.
2. If not enough - nginx in front of Apache (Variant 1).
3. If still not enough - full Apache removal, nginx + php-fpm (Variant 2).

[Fibonacci]

Waiting for the green light from an admin and a confirmed time - then I'll do a short "down" for a few minutes (or maybe I'll fail and the 30-second takedown was just a fluke ).

I'm online until 14:00-15:00 CEST, so the test can happen anytime before that.

[Thaddy]

Fred,

Fail2Ban can be configured in many ways.
Usually an IP is blocked for a number of attempts in a given timespan and it should be unblocked after a fixed, configurable, number of efforts. In my case 5 in 1 minute.
But it seems the IP's are held in the database after that: that makes sense if the same ip or block is a recurring visitor, I use fail2ban myself.
This behavior is in principle correct, although the offending IP or range should be removed after a week or so without offenses. If I remember well, that can be also configured (but my servers currently run low to medium traffic).
Recurring offenders can be blocked permanently, although blocking a sub-range can cause problems to some users in the same block, same hoster, and that is a real world scenario.

Cloudflare has a free plan, but there are concerns about Cloudflare.

But fail2ban alone is not the holy grail, and configuring a webserver(s) as a reverse proxy is also not always a good idea, Nitorami.
Although I also run a reverse proxy to keep my servers sane, mostly a bunch of Raspberry Pi's running nature observation camera's, in my case at the cost of high connection rates..(i.e.: server sane,  clients not happy with higher traffic).

Better to use the "Am I human" solution but without the complex puzzles.

[Fibonacci]

Cloudflare has a free plan, but there are concerns about Cloudflare.

What concerns? Cloudflare would probably solve the problem.

I suggested it before in the thread about the wiki going down - they went with Anubis instead because they want to stay "independent".

Better to use the "Am I human" solution but without the complex puzzles.

Served by what? Apache? Pointless. Apache stops responding earlier than that. Even if there was no forum, just a blank static page, it would still go down.

[Martin_fr]

You need to wait for a reply from Marc. 

He might take a bit of time to respond.

[rvk]

1. Apache tuning - try an add-on module first. Install, tune, see if it holds. I'll give you the mod names, you do the rest.

What mod would block DDoS with setting SYN_RECV without a completed handshake?
As I understand it, creating a connection to the server:port it sets the SYN_RECV sends back a response and waits for ACK.

This is done in the kernel (as I understand) and doesn't reach the program-process. So mods would be useless in this case, aren't they?

This could be tackled by nftables or other low level guard on network layer level.

BTW. I'm also not sure how nginx would help in this case because these connections attempts (without ACK) wouldn't reach nginx.
Or am I missing something?

[Marc]

@Fibonacci
Apache is NOT the bottleneck. Handling PHP is (and yes that is offloaded, so ngnix would suffer too)

[Fibonacci]

@Fibonacci
Apache is NOT the bottleneck. Handling PHP is (and yes that is offloaded, so ngnix would suffer too)

Let me start the test, you watch the logs and server load. Tell me when and for how long.

[Marc]

Let me start the test, you watch the logs and server load. Tell me when and for how long.

I don't need a test, I know where the bottleneck is. And that is hard to solve