Lazarus

Programming => General => Topic started by: What I can do on January 10, 2025, 11:46:42 pm

Title: [EXAMPLE] HTTPClient test web site valid- not 100% depends on Browser
Post by: What I can do on January 10, 2025, 11:46:42 pm
OS:Windows 10@64
Lazarus 3.4
Why do all these links report as not valid
yahoo.com
http://yahoo.com
httP://www.yahoo.com
NOTE: the second set of "//" was add automatically
I'm doing something wrong...

Code: Pascal  [Select][+][-]
  1. uses
  2.   fphttpclient  ,Opensslsockets
  3. //...
  4.  
  5. function IsURLValid(const URL: string): Boolean;
  6. var
  7.   HTTPClient: TFPHTTPClient;
  8. begin
  9.   Result := False;
  10.   HTTPClient := TFPHTTPClient.Create(nil);
  11.   try
  12.     try
  13.       HTTPClient.Get(URL);
  14.       // If we get here without an exception, the URL is likely valid
  15.       Result := True;
  16.     except
  17.       on E: EHTTPClient do
  18.         // URL not valid or other HTTP errors
  19.         Result := False;
  20.     end;
  21.   finally
  22.     HTTPClient.Free;
  23.   end;
  24. end;
  25.  
==============================================
Updated 1/28/2025
sometimes I am so close to the trees that I can't see the forest
from what I needed
could be sumed up with the Open command because it test the web open location the if successful then TRUE
so a lot of that code i rewrote to
//---------------
if not OpenURL(s)
 then OpenURL('https://forum.lazarus.freepascal.org/');
//---------------
Title: Re: test if web site is valid
Post by: TRon on January 10, 2025, 11:54:36 pm
http isn't valid for ages. The second url, is invalid by deifnition

Did you have redirection turned on ? because that also results in an error as per your example.
Title: Re: test if web site is valid
Post by: What I can do on January 11, 2025, 02:03:08 am
Hi TRon
yes the line
http://httP://www.yahoo.com
changes itself when I hit the post button
the original code is
http://www.yahoo.com
but this post utility adds the additional "//http:" for some reason

How do I turn redirection on?
is that a lazarus or browser thing?
Title: Re: test if web site is valid
Post by: TRon on January 11, 2025, 03:17:43 am
Quote from: What I can do on January 11, 2025, 02:03:08 am
but this post utility adds the additional "//http:" for some reason
Ah, sorry. I missed your edit. Sorry about that. Next time just paste the url as text (not a link) or use code tags.

Quote
How do I turn redirection on?
Code: Pascal  [Select][+][-]
  1. HTTPClinet.AllowRedirect:= true
  2.  

See also https://www.freepascal.org/daily/packages/fcl-web/fphttpclient/tfphttpclient.html or in particular https://www.freepascal.org/daily/packages/fcl-web/fphttpclient/tfphttpclient.allowredirect.html

The wiki on fphttpclient (https://wiki.freepascal.org/fphttpclient) can also be helpful

Quote
is that a lazarus or browser thing?
It is a TFPHTTPClient class thing  :)

Because you used the on Except handler in the example TFPHTTPClient will silently fail exceptions (as instructioned) so that you never see the 'error'. So as a hint when running into issues always display the exception message in order to get feedback on what is actually happening.
Title: Re: test if web site is valid
Post by: Remy Lebeau on January 11, 2025, 04:19:26 am
Quote from: TRon on January 10, 2025, 11:54:36 pm
http isn't valid for ages.

That is a bit misleading. There are plenty of sites that are still using HTTP and not HTTPS. Just because HTTPS is popular doesn't necessarily mean that every site needs/uses it.
Title: Re: test if web site is valid
Post by: TRon on January 11, 2025, 04:24:44 am
Quote from: Remy Lebeau on January 11, 2025, 04:19:26 am
That is a bit misleading. There are plenty of sites that are still using HTTP and not HTTPS. Just because HTTPS is popular doesn't necessarily mean that every site needs/uses it.
You are correct Remy. It is misleading because you are right: there still exists websites that accept http. My apologies.

However, major browser builders have mandated that we are not allowed to access those anymore. As a result the number of websites that are accessible by http seem to decline as a result.

Title: Re: test if web site is valid
Post by: Thaddy on January 11, 2025, 10:42:15 am
Quote from: TRon on January 10, 2025, 11:54:36 pm
http isn't valid for ages. The second url, is invalid by deifnition
Wrong, still valid.
three of my static websites:
http://thaddy.com
http://thaddy.org
http://thaddy.nl

If you happen to have a certificate installed issued by me, you will be redirected to a https:// website. The certificate acts as a lock-keypass, it is not ssl, that comes after. Logging in with password is not possible, only with my certificate.
If you try https on these sites *without* my certificate, you get a self-signed dummy.

Since the http:// sites are 100% static, there is no need for https.
Title: Re: test if web site is valid
Post by: TRon on January 11, 2025, 06:34:32 pm
Quote from: Thaddy on January 11, 2025, 10:42:15 am
Wrong, still valid.
No idea in what universe the url ...
Code: [Select]
http://httP://www.yahoo.com
.. is valid, but hey if it does in yours then I am happy for you (forum software seems to agree with you) ;D

But it is dead horse as TS already mentioned the reason why (as already stated I simply and initially missed it). As well as that modern established browser (by default some not even allowing to change that behaviour) forces https on the url no matter what (and as a result you are unable to access the website using http).
Title: Re: test if web site is valid
Post by: jamie on January 11, 2025, 06:51:06 pm
Quote from: TRon on January 11, 2025, 04:24:44 am
Quote from: Remy Lebeau on January 11, 2025, 04:19:26 am
That is a bit misleading. There are plenty of sites that are still using HTTP and not HTTPS. Just because HTTPS is popular doesn't necessarily mean that every site needs/uses it.
You are correct Remy. It is misleading because you are right: there still exists websites that accept http. My apologies.

However, major browser builders have mandated that we are not allowed to access those anymore. As a result the number of websites that are accessible by http seem to decline as a result.

I hope that is not the case, I use http//.. to connection to hardware using the default browser in the pc. That would be a tragic event if that stopped.
Title: Re: test if web site is valid
Post by: TRon on January 11, 2025, 06:56:32 pm
Quote from: jamie on January 11, 2025, 06:51:06 pm
I hope that is not the case, I use http//.. to connection to hardware using the default browser in the pc. That would be a tragic event if that stopped.
Unfortunately it is. And indeed it is a PITA as I am also unable to access some devices on my local network in that way. On some of my systems I have to resort to either a custom solution or use an older browser that still supports http. Not every (recent) browser allows to override the behaviour of forcing https. It the same annoying behaviour as that some browsers removed ftp support.
Title: Re: test if web site is valid
Post by: dseligo on January 11, 2025, 07:08:05 pm
Quote from: TRon on January 11, 2025, 06:56:32 pm
Unfortunately it is. And indeed it is a PITA as I am also unable to access some devices on my local network in that way. On some of my systems I have to resort to either a custom solution or use an older browser that still supports http. Not every (recent) browser allows to override the behaviour of forcing https.

I just tried to connect to my router using http://10.1.1.1 with Firefox, Edge and Chrome (all latest versions) and all of them works. Am I missing something?
Title: Re: test if web site is valid
Post by: Remy Lebeau on January 11, 2025, 10:40:14 pm
Quote from: TRon on January 11, 2025, 04:24:44 am
However, major browser builders have mandated that we are not allowed to access those anymore. As a result the number of websites that are accessible by http seem to decline as a result.

That is absolutely not true. All  browsers handle HTTP and HTTPS equally just fine. The browser vendors are not mandating anything, and they are not the ones driving sites away from HTTP towards HTTPS. That is wholly up to the site owners to decide.
Title: Re: test if web site is valid
Post by: Thaddy on January 11, 2025, 11:32:00 pm
Indeed, all browsers that i know of, al the major ones.
Title: Re: test if web site is valid
Post by: Thaddy on January 11, 2025, 11:33:49 pm
Quote from: TRon on January 11, 2025, 06:56:32 pm
Unfortunately it is.
Fortunately it is not. Come on Ron, admit you are wrong.
Title: Re: test if web site is valid
Post by: What I can do on January 13, 2025, 07:24:27 pm
OS: Windows 10@64
Compiler: Lazarus 3.4
Project: Personal Help notes, data base
I keep a massive data base of links, tip, examples, and anything I deem worthy of noting.
So I have collected some web site links and social post links.
I know that links are not permanent so here is the logic
Test if link is still valid and if so load it into the default browser (Brave)
else link not valid then load the default link "https://forum.lazarus.freepascal.org/"
So I don't care what or if any errors occur during the testing only does it return as valid.
if any error of any sort then load up default Lazarus which is like 90% of the time anyway.

 TRon
Remy Lebeau
Thaddy
jamie
dseligo

I thank you all very sincerely
Here is my code: for testing the links valid
Code: Pascal  [Select][+][-]
  1. function IsURLValid(const URL: string): Boolean;
  2. var
  3.   HTTPClient: TFPHTTPClient;
  4.  
  5. begin
  6.   HTTPClient := TFPHTTPClient.Create(nil);
  7.   HTTPClinet.AllowRedirect:= TRUE;
  8.   try
  9.     try
  10.       HTTPClient.SimpleGet(URL); // Try to fetch the page
  11.       Result:=TRUE;
  12.     except
  13.       on E: Exception do
  14.         Result := False; // If any exception occurs, consider the site as invalid
  15.     end;
  16.   finally
  17.     HTTPClient.Free;
  18.   end;
  19. end;
  20.  

I do not know why some links work while other don't. However all links work if just copy and paste into browser address bar.
I grabbed some random links from X and hacked around with the meta tags of privet, business, and government

links that failed
-----------------
yahoo.com
www.yahoo.com
http://yahoo.com
http://www.yahoo.com
https://infowars.com/
https://www.infowars.com/
http://stewpeters.com/
http://zeeemedia.com
x.com
www.x.com
http://x.com
http://www.x.com

Link with successful validation
+++++++++++++++++++
https://www.yahoo.com
https://x.com/i/status/1869969518908584302
https://x.com
https://www.yahoo.com
https://okdhslive.org/Default.aspx
 
Is the problem with Lazarus, the Brave browser, or me?
Is there any other web type component that might work better just for testing?
Title: Re: test if web site is valid
Post by: Warfley on January 13, 2025, 10:16:24 pm
Quote from: Thaddy on January 11, 2025, 10:42:15 am
Since the http:// sites are 100% static, there is no need for https.
Except that an attacker can impersonate your website. There are a whole host of attacks that are possible. E.g. an attacker can inject javascript into the website to mine cryptocurrencies with the users browser while they visit your website or more overtly could embedd advertisements to gain money from the people visiting your website. Or an attacker could use the trust the user has into your website and trick people with scams, e.g. advocating for some dubious services or something. A bit more malicious they could provide downloads which contain malware or link to malicious websites etc.

HTTPS is not just encryption, it's also authorization. While DNS certificates are just domain verified, certificate transparency (which is a requirement for CAs to be accepted into the browsers certificate stores) ensures that no two certificates should be issued to the same domain. Additionally you can register a DNSSec entry, with the CA you get your certificate from, to ensure that no corrupted other CA impersonates your website.

Even static websites can be dangerous. I mean your website is quite small so the risk is negligible, but image a news website from a big news organization. News articles are 100% static, yet someone who could just put anything up on such a website, any fake stories, any scams, etc. could seriously do a lot of harm.

Static or not, HTTPS should always be used unless in very specific circumstances (e.g. on calls to localhost, or extremely small local networks), especially as Let's Encrypt makes it completely free
Title: Re: test if web site is valid
Post by: dbannon on January 14, 2025, 12:36:15 am
Quote from: Warfley on January 13, 2025, 10:16:24 pm
.....
Static or not, HTTPS should always be used unless in very specific circumstances (e.g. on calls to localhost, or extremely small local networks), especially as Let's Encrypt makes it completely free

While I agree with the general position, there is a real need for people to be able to connect to a 'local' website without security. For example, I have a RasPi managing my solar hotwater service, its on my private subnet and shows me an number of relevant temperatures. Its on my "small local domain" (of course) so, not possible to get a Lets Encrypt cert, they, sensibly want to verify I own the domain. I have two other, similar examples but won't bore you with the details.

If I could get a Lets Encrypt cert, maybe I would but probably not.

The key may be whether its a publicly visible website or not.

Davo



Title: Re: test if web site is valid
Post by: Warfley on January 14, 2025, 12:42:12 am
Yeah as I said like small local networks, where you know each of the devices in it by heart it's not an issue. If it gets bigger than that, especially if you don't fully trust all the devices (e.g. I don't trust any smart home devices at all) you can use self signed certificates. Just create them locally and add them to your browser and you're fine without any public CA like let's encrypt
Title: Re: test if web site is valid
Post by: dbannon on January 14, 2025, 01:08:05 am
Indeed. I used to run a Certificate Authority for scientific use, because we were not recognized by the browsers, our users, maybe 1000 in AU, had to do just that and they hated it !  I expect (and hope) it has not got any easier.

At present, my Firefox on both laptop and phone are happy to work with http, 18 months ago, for a brief time, they were not. I expect thats what is confusing TRon, there was sufficient complaints that eg Firefox decided to reverse that extra security decision. Maybe only for local subnets ?

Davo
Title: Re: test if web site is valid
Post by: Warfley on January 14, 2025, 01:22:48 am
Both Mozilla and Google are pretty open about wanting to get rid of the http and https distinction. The reason for this is that people confuse the lock symbol as some form of seal of quality or authenticity, so while the proliferation of free certificates is overall a net good for security online, it had the side effect that it lend legitimacy to scamming websites.

This is why the https lock has been changing over the versions, being like very highlighted in green, even with a text saying "secure" in earlier versions of FF and Chrome, to now being just a very small pictogram, and much more highlight (i.e. red and with text) when it's not HTTPS.

The final step they want to see is to basically don't have any indication for https anymore and just have a warning when there is bare http, and as part of that they recently tried to go into that direction by having the user to explicitly accept wanting to open an http website.

There were some exceptions tho. Localhost always worked (and is also except from CORS enforcement), and I think anything you call without a domain but directly via IP was also excluded. They seemed to have rolled back this change, but from what I know they still intend to go further down this direction.
Title: Re: test if web site is valid
Post by: Thaddy on January 14, 2025, 06:51:18 pm
fwiw, that's why I need to change some of my hosting after 25 yyears, because somebody wants to make money selling certificates, and it did not dawn on them that let's encrypt is free...
TinyPortal © 2005-2018