504 Gateway Timeout

[Fibonacci]

The forum has been throwing 504 Gateway Timeout errors pretty frequently lately - during page loads, posting, simple browsing. It makes the forum quite difficult to use.

Is the team aware of this? Are there any plans to address it?

[cdbc]

Hi
+1
It has been going for about a week or so...
Regards Benny

[creaothceann]

The best way is to ask on the mail list, then it will be escalated

[Graeme]

+1

It's been driving me nuts!!

ps: I've just reported it in the mailing list.

[440bx]

I've been in the same boat too.

The strange thing is that it's very unpredictable.  The forum works perfectly fine (good response time) then it suddenly goes into a period of gateway timeouts that may last from a couple of minutes to, sometimes, over 15 minutes.

Yes, it really gets in the way of browsing the forum.

[LemonParty]

Why is that happening? DDOS?

[dseligo]

Why is that happening? DDOS?

AI training probably.

[Marc]

I'm fully aware of the timeouts and constantly monitoring it.
Lately we ran into a different kind of "attack", connections to the server are kept alive, without much traffic. To work around this I've shortened the alive time. But this affects the browsing experience, resulting in timeouts happen more often.
I'm still tweaking this

[440bx]

I just wanted to mention that, in the past the timeouts came and went in a matter of some minutes but now, there are periods of 8 to 14 hours (yes, hours!) that any attempt to view the forums gets a timeout.

I just went through a period of about 10 hours getting nothing but timeouts then, the forum was ok for about 5 minutes then another timeout period of about 6 hours.

I've seen many sites using Cloudflare to protect themselves from malicious bots, if the timeouts are in some way connected to this problem, could we use Cloudflare too to solve the problem ?

[JanRoza]

I am glad that the forum at long last is back again after a weekend of gateway timeouts.
Any ideas yet about the cause of these timeouts?

[Thaddy]

@JanRoza

Did you miss that?

Lately we ran into a different kind of "attack", connections to the server are kept alive, without much traffic. To work around this I've shortened the alive time. But this affects the browsing experience, resulting in timeouts happen more often.

Anyway, we are back browsing..

An example of that is that CoPilot keeps the connection open for anyone that is actually using Copilot and have given permission to read along with what they are doing. That generates not much traffic, and helps just a single user. It only drops the connection if you move away and there are no tabs open that access the forum.

[JanRoza]

@Thaddy: Yes I really missed it, a couple of times per day I read all new messages.
Just to learn and get wiser. 

[Marc]

We were "serving" 100+ request per second the past days. That's something the server can't handle.
Bad requests are handled by fail2ban and other tools, so these IPs and networks get blocked. Fail2ban however has one problem I found out. It keeps all blocks in an internal sqlite database without removing entries. This DB has grown to 17GB, causing netblock detecting to fail.
Besides this, there is the past months a different pattern in server load which I don't know yet how to handle.

[LeP]

By me, this night the forum was still inacessible.

And now I can't use any link on the forum (login too): "Cookies error".

I had to go into my browser history to find the direct link to the login page, and now it seems to work.

[ccrause]

We were "serving" 100+ request per second the past days. That's something the server can't handle.
Bad requests are handled by fail2ban and other tools, so these IPs and networks get blocked. Fail2ban however has one problem I found out. It keeps all blocks in an internal sqlite database without removing entries. This DB has grown to 17GB, causing netblock detecting to fail.
Besides this, there is the past months a different pattern in server load which I don't know yet how to handle.

@Marc thank you for maintaining this service, it is much appreciated.  I also had trouble connecting since about Saturday, obviously it is better now  .