Anubis's website security breaks the RSS feed

[ALLIGATOR]

Is it possible to bypass Anubis's protection for an RSS URL?

https://forum.lazarus.freepascal.org/index.php?action=.xml;type=rss

Because it's breaking my RSS reader.

All it takes is switching from one provider to another—for example, between different offices or different Wi-Fi or mobile providers—and my RSS feed stops working!

To get it working again, I first have to open the forum site in my RSS reader so that Anubis can calculate its hash; only then does everything work again—until I switch to a different provider (moving between home and work, other locations, different Wi-Fi networks, different providers).

[Thaddy]

RSS behaves much like a bot, because it technically is a bot. Been a while since I used one and seldom see them anymore, but I guess there must be a way to configure that on the server. I also think that it will probably blow unwanted holes in the purpose of Anubis. If you allow RSS that opens up a hole that can be misused by pretending to be an rss client.
For example, you can use it to clone the content in real time (don't say it isn't a bot) and run a forum.fr33p4scal.io with advertisement or malware. (This is also the scenario why RSS feeds are increasingly unpopular with web admins)

[edit] A quick search and yes, that is likely configurable, but found without a concrete config example.

The responsiveness of the forum is more important than supporting RSS, in my opinion.
Everybody seems very happy since Anubis...

Maybe you can run over a reverse proxy: that will circumvent the location issue for your devices but requires that you run a server yourself, of course.

Also note it is not about the information, it is about he real-time part.

[cdbc]

Hi
@Thaddy: +1
I, for one, am happy to have a __working__ forum again \o/ ...and I think a lot of other forum-users feel alike.
I dunno, maybe we can convince @ALLIGATOR to "take one" for the greater good?!? Just this time round 
Regards Benny

[Marc]

We didn't have much rss queries in the past, so i've disabled anubis for rss queries

[ALLIGATOR]

...so i've disabled anubis for rss queries

Thank you!

[ALLIGATOR]

@Marc

It seems... Anubis isn't letting my RSS client access the news feed again... My RSS client is a Firefox add-on called “Brief”

1. I refresh the RSS feed several times -nothing new appears
2. Then I open the forum page via the Firefox browser; Anubis does its calculations and opens the forum for me
3. After that, I refresh the RSS feed again - and this time the news comes through successfully
4. I conclude that Anubis is blocking them. Since everything used to work perfectly, and around the time Anubis appeared, RSS stopped working properly for me

Please disable it for RSS URLs

[Marc]

@Marc

It seems... Anubis isn't letting my RSS client access the news feed again... My RSS client is a Firefox add-on called “Brief”

1. I refresh the RSS feed several times -nothing new appears
2. Then I open the forum page via the Firefox browser; Anubis does its calculations and opens the forum for me
3. After that, I refresh the RSS feed again - and this time the news comes through successfully
4. I conclude that Anubis is blocking them. Since everything used to work perfectly, and around the time Anubis appeared, RSS stopped working properly for me

Please disable it for RSS URLs

The one at the bottom of the page is not challenged, if there are other rss urls in use let me know.

[rvk]

The one at the bottom of the page is not challenged, if there are other rss urls in use let me know.

Haha, and you can remove that WAP2 link (which doesn't work) 

(but I rather have a working reactive view for mobile  )

[ALLIGATOR]

...if there are other rss urls in use let me know.

Code: [Select]

https://forum.lazarus.freepascal.org/index.php?type=rss;action=.xml;limit=255

[Marc]

...if there are other rss urls in use let me know.

Code: [Select]

https://forum.lazarus.freepascal.org/index.php?type=rss;action=.xml;limit=255

I't hard to find in the documentation how query parameters are parsed, please retry

[ALLIGATOR]

...please retry

Anubis is still protecting this URL

It’s easy to check for yourself: just open a private window (Firefox) or switch to Incognito mode (Chrome), etc., and paste this URL into it

The first time you do this, you’ll see the Anubis interface

If you need to repeat the process, close the private window and open

[rvk]

Anubis is still protecting this URL

That's strange. Yes, when using a browser Anubis pops up and needs cookies to be redirected. When using curl on Linux it does not.
So it seems to be that browsers are handled differently.

With what kind of program are you retrieving the XML ??

Code: [Select]

$ curl -v "https://forum.lazarus.freepascal.org/index.php?type=rss;action=.xml;limit=1"
* Host forum.lazarus.freepascal.org:443 was resolved.
* IPv6: 2a01:7c8:aac1:2f0::1
* IPv4: 37.97.187.115
*   Trying [2a01:7c8:aac1:2f0::1]:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=lazarus-ide.org
*  start date: Apr  7 03:26:45 2026 GMT
*  expire date: Jul  6 03:26:44 2026 GMT
*  subjectAltName: host "forum.lazarus.freepascal.org" matched cert's "forum.lazarus.freepascal.org"
*  issuer: C=US; O=Let's Encrypt; CN=R12
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to forum.lazarus.freepascal.org (2a01:7c8:aac1:2f0::1) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://forum.lazarus.freepascal.org/index.php?type=rss;action=.xml;limit=1
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: forum.lazarus.freepascal.org]
* [HTTP/2] [1] [:path: /index.php?type=rss;action=.xml;limit=1]
* [HTTP/2] [1] [user-agent: curl/8.14.1]
* [HTTP/2] [1] [accept: */*]
> GET /index.php?type=rss;action=.xml;limit=1 HTTP/2
> Host: forum.lazarus.freepascal.org
> User-Agent: curl/8.14.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
< cache-control: private
< content-disposition: inline; filename="Lazarus-recent-rss.xml"
< content-type: application/rss+xml; charset=UTF-8
< date: Thu, 21 May 2026 15:08:22 GMT
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< pragma: no-cache
< server: Apache/2.4.58 (Ubuntu)
< vary: Accept-Encoding
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1
< set-cookie: PHPSESSID=aearqdc10n77lqjnqrcgpbtcb3; path=/
<
<?xml version="1.0" encoding="UTF-8"?>
<rss version="0.92" xml:lang="en-US">
        <channel>
                <title>Lazarus</title>
                <link>https://forum.lazarus.freepascal.org/index.php</link>
                <description><![CDATA[Live information from Lazarus]]></description>
                <item>
                        <title>Re: Benchmark regular vs SIMD vs constref</title>
                        <link>https://forum.lazarus.freepascal.org/index.php/topic,74080.msg583215.html#msg583215</link>
                        <description>I also believe that if you manually unroll the loop (by a factor of 2, 3, or 4) for standard calculations, you can achieve a further increase in speed</description>
                        <category>General</category>
                        <comments>https://forum.lazarus.freepascal.org/index.php?action=post;topic=74080.0</comments>
                        <pubDate>Thu, 21 May 2026 15:01:24 GMT</pubDate>
                        <guid>https://forum.lazarus.freepascal.org/index.php/topic,74080.msg583215.html#msg583215</guid>
                </item>
        </channel>
* Connection #0 to host forum.lazarus.freepascal.org left intact

[ALLIGATOR]

With what kind of program are you retrieving the XML ??

My RSS client is a Firefox add-on called “Brief”

[rvk]

With what kind of program are you retrieving the XML ??

My RSS client is a Firefox add-on called “Brief”

Ok. That might be the problem because then Anubis will treat it as a browser. I think that matters.

Just checked... yes, with a user-agent for Firefox you get the problem. Use another user-agent it it might work. Can you set the user-agent in "Brief" ??

Code: [Select]

$ curl -v curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0" "https://forum.lazarus.freepascal.org/index.php?type=rss;action=.xml;limit=1"
* Could not resolve host: curl
* shutting down connection #0
curl: (6) Could not resolve host: curl
* Host forum.lazarus.freepascal.org:443 was resolved.
* IPv6: 2a01:7c8:aac1:2f0::1
* IPv4: 37.97.187.115
*   Trying [2a01:7c8:aac1:2f0::1]:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=lazarus-ide.org
*  start date: Apr  7 03:26:45 2026 GMT
*  expire date: Jul  6 03:26:44 2026 GMT
*  subjectAltName: host "forum.lazarus.freepascal.org" matched cert's "forum.lazarus.freepascal.org"
*  issuer: C=US; O=Let's Encrypt; CN=R12
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to forum.lazarus.freepascal.org (2a01:7c8:aac1:2f0::1) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://forum.lazarus.freepascal.org/index.php?type=rss;action=.xml;limit=1
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: forum.lazarus.freepascal.org]
* [HTTP/2] [1] [:path: /index.php?type=rss;action=.xml;limit=1]
* [HTTP/2] [1] [user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0]
* [HTTP/2] [1] [accept: */*]
> GET /index.php?type=rss;action=.xml;limit=1 HTTP/2
> Host: forum.lazarus.freepascal.org
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
< cache-control: no-store
< content-type: text/html; charset=utf-8
< date: Thu, 21 May 2026 15:27:34 GMT
< set-cookie: techaro.lol-anubis-auth=; Path=/; Expires=Thu, 21 May 2026 15:26:34 GMT; Max-Age=0; Secure; SameSite=None
< set-cookie: techaro.lol-anubis-cookie-verification=019e4b26-3078-7779-b5af-0a14cf0013f7; Path=/; Expires=Thu, 21 May 2026 15:57:34 GMT; Secure; SameSite=None
< vary: Accept-Encoding
< server: Apache/2.4.58 (Ubuntu)
<
<!doctype html><html lang="en"><head><title>Making sure you&#39;re not a bot!</title><link rel="stylesheet" href="/.within.website/x/xess/xess.min.css?cachebuster=1.25.0"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="robots" content="noindex,nofollow"><style>
        body,
        html {
            height: 100%;
            display: flex;
            justify-content: center;
            align-items: center;
            margin-left: auto;
            margin-right: auto;
        }

        .centered-div {
            text-align: center;
        }

        #status {
            font-variant-numeric: tabular-nums;
        }

        #progress {
            display: none;
            width: 90%;
            width: min(20rem, 90%);
            height: 2rem;
            border-radius: 1rem;
            overflow: hidden;
            margin: 1rem 0 2rem;
            outline-offset: 2px;
            outline: #b16286 solid 4px;
        }

        .bar-inner {
            background-color: #b16286;
            height: 100%;
            width: 0;
            transition: width 0.25s ease-in;
        }
        </style><script id="anubis_version" type="application/json">"1.25.0"
</script><script id="anubis_challenge" type="application/json">{"rules":{"algorithm":"fast","difficulty":2},"challenge":{"issuedAt":"2026-05-21T17:27:34.520494533+02:00","metadata":{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0","X-Real-Ip":"2a02:a44e:edd1:0:2df3:6dde:2710:4436"},"id":"019e4b26-3078-7779-b5af-0a14cf0013f7","method":"fast","randomData":"ff25606cdb898317425540b65a4f50186faab995e2eb29d959f30765d3b9b727a021144ae4631f23a75d0b732dbaf091fddf1f38177aa254d4a22d39328cdca4","policyRuleHash":"ac980f49c4d35fab","difficulty":2,"spent":false}}
</script><script id="anubis_base_prefix" type="application/json">""
</script><script id="anubis_public_url" type="application/json">""
</script></head><body id="top"><script type="ignore"><a href="/.within.website/x/cmd/anubis/api/honeypot/37b852c4-405c-43c8-8f90-5881517f1331/init">Don't click me</a></script><main><h1 id="title" class="centered-div">Making sure you&#39;re not a bot!</h1><div class="centered-div"><img id="image" style="width:100%;max-width:256px;" src="/.within.website/x/cmd/anubis/static/img/pensive.webp?cacheBuster=1.25.0"> <img style="display:none;" style="width:100%;max-width:256px;" src="/.within.website/x/cmd/anubis/static/img/happy.webp?cacheBuster=1.25.0"><p id="status">Loading...</p><script async type="module" src="/.within.website/x/cmd/anubis/static/js/main.mjs?cacheBuster=1.25.0"></script><div id="progress" role="progressbar" aria-labelledby="status"><div class="bar-inner"></div></div><details><p>You are seeing this because the administrator of this website has set up Anubis to protect the server against the scourge of AI companies aggressively scraping websites. This can and does cause downtime for the websites, which makes their resources inaccessible for everyone.</p><p>Anubis is a compromise. Anubis uses a Proof-of-Work scheme in the vein of Hashcash, a proposed proof-of-work scheme for reducing email spam. The idea is that at individual scales the additional load is ignorable, but at mass scraper levels it adds up and makes scraping much more expensive.</p><p>Ultimately, this is a placeholder solution so that more time can be spent on fingerprinting and identifying headless browsers (EG: via how they do font rendering) so that the challenge proof of work page doesn&#39;t need to be presented to users that are much more likely to be legitimate.</p><p>Please note that Anubis requires the use of modern JavaScript features that plugins like JShelter will disable. Please disable JShelter or other such plugins for this domain.</p></details><noscript><p>Sadly, you must enable JavaScript to get past this challenge. This is required because AI companies have changed the social contract around how website hosting works. A no-JS solution is a work-in-progress.</p></noscript><div id="testarea"></div></div><footer><div * Connection #1 to host forum.lazarus.freepascal.org left intact
class="centered-div"><p>Protected by <a href="https://github.com/TecharoHQ/anubis">Anubis</a> From <a href="https://techaro.lol">Techaro</a>. Made with ❤️ in 🇨🇦.</p><p>Mascot design by <a href="https://bsky.app/profile/celphase.bsky.social">CELPHASE</a>.</p><p>This website is running Anubis version <code>1.25.0</code>.</p></div></footer></main></body></html>

[ALLIGATOR]

Can you set the user-agent in "Brief" ??

That option is not available