Forum slow for others as well ?

[440bx]

Hello,

The forum has been quite slow for me the past 4 hours or so, even getting gateway timeouts.

Anyone else experiencing this ?

[Zvoni]

This morning i had a 504 TimeOut.
But currently it works again....

[440bx]

Yes, it seems to be working fine now.

[cdbc]

Hi
YES!
It's been like that for a couple of days now... 
It started with timeouts ca. 12.00 .. 15.00 GMT+1, then good again
Today it started around 8.00 GMT+1 .. till about now, seems fine again...
 
@Marc: We need your strong Voodoo  ...again 
Regards Benny

[Remy Lebeau]

Right now, it's working fine. But performance has been up and down for the past couple of months at least:

https://forum.lazarus.freepascal.org/index.php/topic,70194.0.html
https://forum.lazarus.freepascal.org/index.php/topic,70391.0.html
https://forum.lazarus.freepascal.org/index.php/topic,70517.0.html

[440bx]

Right now, it's working fine. But performance has been up and down for the past couple of months at least:

Yes, even in the same day, the performance seems to vary significantly. 

Earlier today, performance was really good but, just a minute ago, I got a gateway timeout.  It's quite unpredictable.

[TRon]

Same issues here.

It comes and goes. At my age I'm getting pretty slow with typing these days but I am still able to outpace the login and posting speed (or the speed of posting alone). At least it doesn't time out as it used to.

[Thaddy]

It is a recurring issue but not due to the site manager.
It is more like: we need better hardware, but that costs money and that is not really an option (yet).
Most bots are evil, but not all of them.
There was even a request to index the forum.
The result is that many of the popular AI sites produce better Freepascal code and that is noticable...So it is annoying but also gives us better tooling.
I usually let these engines fight for what code I should use...
Don't worry, 99% I post is by me, myself alone and otherwise I contribute to the source I used.
My observation is that the AI models adapt when you post back your own corrected source code.. e.g. if you now ask for a freepascal example they often do a {$mode objfpc}...  Post back your corrected sourcecode!
Fighting with those AI engines is great fun for the time being!

[PascalDragon]

One could probably deploy a system like Anubis which invokes some proof-of-work and thus makes things harder for AI scrapers. Sites like git.kernel.org employ it because scrapers were repeatedly scraping endpoints like snapshot or blame which are very expensive for the server side.
Disadvantage would be that it requires enabled JavaScript though specific browsers could be excluded based on their user agents. The developers are also trying to convert their proof-of-work code from pure JS to wasm which would prohibit big endian systems cause for those there are no wasm engines available as far as I know... 😕

[dbannon]

But its a Catch-22 situation, we want the forum indexed because that is how end users get good information about FPC/Lazarus. But we need the forum usable for our own purpose.

Would a solution be providing a second, crawler accessible 'copy' of the forum, maybe replicated daily, rsync like to minimize load ?  The crawlers don't seem to mind how slow their target is.

And protect, as best we can, the 'real' forum, maybe redirecting requests identified as crawlers to the relatively static one ?

That seems a better approach that the 'trending' one of feeding bad data to a robot.txt ignoring crawler, we do actually want these crawlers to get good data.

Davo

[440bx]

I'm not sure but I think there is a way of limiting the number of requests per IP.  If an IP is detected to make more than 1 request per second, it could be limited to getting no more than 1 request per second (or every 2 or 3 seconds... crawlers don't have a life to take care of... they can wait.)

That way, any crawler that's hitting the server gets put in its place.
 
A human user would be hard pressed to make more than 1 request per second, therefore there is little chance that could negatively affect forum members and, even if it did, one second isn't that much time to wait to get a page.

Just for the record, Google does that.  When it detects a program using its search engine, it doesn't make the program wait, it simply bans the IP for 24 hours (the first time, I don't know what it does the second time, I didn't want to find out )

[Thaddy]

Marc already uses fail2ban - alongside other tools - as far as I know.

[PascalDragon]

But its a Catch-22 situation, we want the forum indexed because that is how end users get good information about FPC/Lazarus. But we need the forum usable for our own purpose.

Proper behaving search engine crawlers are not a problem (and for example in Anubis they are excluded by default). The problem is AI scrapers that burden the servers of open source software and drive up both their server usage and their traffic usage for improving their AI models without even remotely respecting things like robots.txt.

A human user would be hard pressed to make more than 1 request per second, therefore there is little chance that could negatively affect forum members and, even if it did, one second isn't that much time to wait to get a page.

Well, I tend to open all the "most recently" posts in new tabs and depending on my browser's usage it might indeed do more than one request a second then 😅