Forum slow

[Zvoni]

OTOH ignoring robots.txt is not a nice way to operate.

Redirect the IP of that search engine to a page showing a "middlefinger"?

[Marc]

OTOH ignoring robots.txt is not a nice way to operate.

Redirect the IP of that search engine to a page showing a "middlefinger"?

On Apache level: return 403 for ClaudeBot
On fail2ban level: block IP for bad-bots

even on a 403 they kept on going

[Thaddy]

I am fine with the fail2ban solution, merely remarked the bot its purpose.
Thanks Marc!

[440bx]

even on a 403 they kept on going

Just a thought... for bots that keep going, is it possible to cause the server to reply only every five (5) seconds to requests from their IP ?

That would take a load off the server

[Marc]

even on a 403 they kept on going

Just a thought... for bots that keep going, is it possible to cause the server to reply only every five (5) seconds to requests from their IP ?

That would take a load off the server

Not if they do it from 467 different IP addresses

[Thaddy]

That would take a load off the server

Marc knows what he is doing.
fail2ban (or similar) works.
And he also knows what to do with my second suggestion, which limits the bot on the server software directly at least on the server software the forum uses.

Your suggestion comes close to 2) but is a bit, do not misunderstand me, a bit naive.
The server configuration already does something like that with the same effect.

The reason I made my remarks is that fail2ban ca be a bit crude and it may be beneficial to the community if a crawler/bot is crawling with the intention to improve their AI brand. But in this case it interfered with the workings of the forum.
In this case there is also an ethical question at play, since the bot ignores well established standards on how to behave.

Better leave it to Marc: he is a pro.
What you suggested is - part of - an option that is already available to him and he actually uses. So it is not a bad idea, but usually already implemented.
It takes time though to mitigate such issues. fail2ban can be configured like a sledge hammer. or a fly killer. But note my remarks: if the ethical question is considered and the opinion is, that the indexing by an AI company bot is not harmful and Marc has the time, he can configure the server to let the bot do its work but under his control.

[440bx]

Better leave it to Marc: he is a pro.

I have no doubt he knows what he's doing.

I just made a suggestion, it is entirely up to him to decide if it is possible, practical and/or worthwhile and by his response it doesn't look like it's practical. 

[Joanna from IRC]

Is there a way to ban bots altogether? It sounds like the entire forum is being harvested for the benefit of companies that want to profit from other people’s work..

[Thaddy]

Yes there is a way and Marc used it this morning: server tooling like fail2ban and with some - a lot - more work the server software itself, like ngynx and apache.
In this case, you must understand that a bot is not always harmful and can even benefit the community. But there is indeed an ethical question, that is correct.
It was not my intention to open a hornet's nest. I should have made the conversation with Marc private.
BTW my suggestions are used by myself on some of my suppositely vulnerable servers.
And it is clear our sysop uses them too.

People that experience such issues with their own servers can send a private request.
The topic is too complicated to do it right if you lack knowledge on how to manage servers. This is beyond the scope of this forum, so I hope the reactions stay at a minimum.

[Joanna from IRC]

How are bots identified besides being able to do things at a speed that a human can’t?

[Thaddy]

In this case just by identifying the traffic. Reading logs. The logs are round-robin and can partially be mitigated by automation.
But as I intended to make clear: this is not the place for a server management course.
Do not spoil that Joanna, I am in a friendly mood today...

[Kays]

I don’t understand. What has fail2ban to do with web crawlers? Fail2ban already has in its name that there was/were failed attempt(s) regarding a network resource, e. g. you entered five times a wrong password thus for the next fifteen minutes any request from your IP address are immediately rejected. What kind of failed requests does a web crawler do though that fail2ban could be useful? A web crawler follows all kinds of HTML <a href="…">…</a> links, it does not invent random URLs thus causing loads of 404 resource not found errors (= failures), does it?

[TRon]

When looking at some of the IP blocks, I noticed they all were owned by amazon. Some googling resolved in ClaudeBot made by anthropic.com.
I've blocked this bot from accessing this forum

That certainly helped a lot. Thank you for having looked into it and solving it MW. Much appreciated.

I don’t understand. What has fail2ban to do with web crawlers?
...

fail2ban can do more than just 'watching' login attempts. For a quick example see here.

[MarkMLl]

Is there a way to ban bots altogether? It sounds like the entire forum is being harvested for the benefit of companies that want to profit from other people’s work..

That raises the important question of whether this forum- and the overall community of Pascal users- loses more from its content being abused or from cutting itself off from a potential source of publicity.

Pascal- or Delphi, or whatever you want to call it- needs all the help it can get.

MarkMLl

[MarkMLl]

How are bots identified besides being able to do things at a speed that a human can’t?

If something tries to connect via HTTP (i.e. the case we're looking at), there's lots of ways based on headers and so on. See e.g. https://code-maze.com/http-series-part-3/ which I stumbled upon while trying to answer the next para adequately.

For non-HTTP, there's still a surprising number of ways based on the use of known IP sender addresses, spoofed sender addresses, peculiarities in higher-level protocol (e.g. ssh) use, peculiarities in TCP (e.g. how it reacts to a forced error), or IP. See https://lcamtuf.coredump.cx/p0f3/ which I've used when doing some router work: you've no idea how much utter crap hits an exposed router connected to the raw Internet.

If you have a couple of minutes also see https://lcamtuf.coredump.cx/oldtcp/tcpseq.html and https://lcamtuf.coredump.cx/newtcp/ which, by coincidence, are from the same researcher. I don't know the extent to which they're still valid, but they're a very good example of how "rolling your own" network implementation can be unwise.

MarkMLl