test if web site is valid

[What I can do]

OS:Windows 10@64
Lazarus 3.4
Why do all these links report as not valid
yahoo.com
http://yahoo.com
http://httP://www.yahoo.com
NOTE: the second set of "//" was add automatically
I'm doing something wrong...

Code: Pascal  [Select][+][-]

1. uses
2.   fphttpclient  ,Opensslsockets
3. //...
4.  
5. function IsURLValid(const URL: string): Boolean;
6. var
7.   HTTPClient: TFPHTTPClient;
8. begin
9.   Result := False;
10.   HTTPClient := TFPHTTPClient.Create(nil);
11.   try
12.     try
13.       HTTPClient.Get(URL);
14.       // If we get here without an exception, the URL is likely valid
15.       Result := True;
16.     except
17.       on E: EHTTPClient do
18.         // URL not valid or other HTTP errors
19.         Result := False;
20.     end;
21.   finally
22.     HTTPClient.Free;
23.   end;
24. end;
25.  

[TRon]

http isn't valid for ages. The second url, is invalid by deifnition

Did you have redirection turned on ? because that also results in an error as per your example.

[What I can do]

Hi TRon
yes the line
http://httP://www.yahoo.com
changes itself when I hit the post button
the original code is
http://www.yahoo.com
but this post utility adds the additional "//http:" for some reason

How do I turn redirection on?
is that a lazarus or browser thing?

[TRon]

but this post utility adds the additional "//http:" for some reason

Ah, sorry. I missed your edit. Sorry about that. Next time just paste the url as text (not a link) or use code tags.

How do I turn redirection on?

Code: Pascal  [Select][+][-]

1. HTTPClinet.AllowRedirect:= true
2.  

See also https://www.freepascal.org/daily/packages/fcl-web/fphttpclient/tfphttpclient.html or in particular https://www.freepascal.org/daily/packages/fcl-web/fphttpclient/tfphttpclient.allowredirect.html

The wiki on fphttpclient can also be helpful

is that a lazarus or browser thing?

It is a TFPHTTPClient class thing 

Because you used the on Except handler in the example TFPHTTPClient will silently fail exceptions (as instructioned) so that you never see the 'error'. So as a hint when running into issues always display the exception message in order to get feedback on what is actually happening.

[Remy Lebeau]

http isn't valid for ages.

That is a bit misleading. There are plenty of sites that are still using HTTP and not HTTPS. Just because HTTPS is popular doesn't necessarily mean that every site needs/uses it.

[TRon]

That is a bit misleading. There are plenty of sites that are still using HTTP and not HTTPS. Just because HTTPS is popular doesn't necessarily mean that every site needs/uses it.

You are correct Remy. It is misleading because you are right: there still exists websites that accept http. My apologies.

However, major browser builders have mandated that we are not allowed to access those anymore. As a result the number of websites that are accessible by http seem to decline as a result.

[Thaddy]

http isn't valid for ages. The second url, is invalid by deifnition

Wrong, still valid.
three of my static websites:
http://thaddy.com
http://thaddy.org
http://thaddy.nl

If you happen to have a certificate installed issued by me, you will be redirected to a https:// website. The certificate acts as a lock-keypass, it is not ssl, that comes after. Logging in with password is not possible, only with my certificate.
If you try https on these sites *without* my certificate, you get a self-signed dummy.

Since the http:// sites are 100% static, there is no need for https.

[TRon]

Wrong, still valid.

No idea in what universe the url ...

Code: [Select]

http://httP://www.yahoo.com
.. is valid, but hey if it does in yours then I am happy for you (forum software seems to agree with you)

But it is dead horse as TS already mentioned the reason why (as already stated I simply and initially missed it). As well as that modern established browser (by default some not even allowing to change that behaviour) forces https on the url no matter what (and as a result you are unable to access the website using http).

[jamie]

That is a bit misleading. There are plenty of sites that are still using HTTP and not HTTPS. Just because HTTPS is popular doesn't necessarily mean that every site needs/uses it.

You are correct Remy. It is misleading because you are right: there still exists websites that accept http. My apologies.

However, major browser builders have mandated that we are not allowed to access those anymore. As a result the number of websites that are accessible by http seem to decline as a result.

I hope that is not the case, I use http//.. to connection to hardware using the default browser in the pc. That would be a tragic event if that stopped.

[TRon]

I hope that is not the case, I use http//.. to connection to hardware using the default browser in the pc. That would be a tragic event if that stopped.

Unfortunately it is. And indeed it is a PITA as I am also unable to access some devices on my local network in that way. On some of my systems I have to resort to either a custom solution or use an older browser that still supports http. Not every (recent) browser allows to override the behaviour of forcing https. It the same annoying behaviour as that some browsers removed ftp support.

[dseligo]

Unfortunately it is. And indeed it is a PITA as I am also unable to access some devices on my local network in that way. On some of my systems I have to resort to either a custom solution or use an older browser that still supports http. Not every (recent) browser allows to override the behaviour of forcing https.

I just tried to connect to my router using http://10.1.1.1 with Firefox, Edge and Chrome (all latest versions) and all of them works. Am I missing something?

[Remy Lebeau]

However, major browser builders have mandated that we are not allowed to access those anymore. As a result the number of websites that are accessible by http seem to decline as a result.

That is absolutely not true. All  browsers handle HTTP and HTTPS equally just fine. The browser vendors are not mandating anything, and they are not the ones driving sites away from HTTP towards HTTPS. That is wholly up to the site owners to decide.

[Thaddy]

Indeed, all browsers that i know of, al the major ones.

[Thaddy]

Unfortunately it is.

Fortunately it is not. Come on Ron, admit you are wrong.

[What I can do]

OS: Windows 10@64
Compiler: Lazarus 3.4
Project: Personal Help notes, data base
I keep a massive data base of links, tip, examples, and anything I deem worthy of noting.
So I have collected some web site links and social post links.
I know that links are not permanent so here is the logic
Test if link is still valid and if so load it into the default browser (Brave)
else link not valid then load the default link "https://forum.lazarus.freepascal.org/"
So I don't care what or if any errors occur during the testing only does it return as valid.
if any error of any sort then load up default Lazarus which is like 90% of the time anyway.

 TRon
Remy Lebeau
Thaddy
jamie
dseligo

I thank you all very sincerely
Here is my code: for testing the links valid

Code: Pascal  [Select][+][-]

1. function IsURLValid(const URL: string): Boolean;
2. var
3.   HTTPClient: TFPHTTPClient;
4.  
5. begin
6.   HTTPClient := TFPHTTPClient.Create(nil);
7.   HTTPClinet.AllowRedirect:= TRUE;
8.   try
9.     try
10.       HTTPClient.SimpleGet(URL); // Try to fetch the page
11.       Result:=TRUE;
12.     except
13.       on E: Exception do
14.         Result := False; // If any exception occurs, consider the site as invalid
15.     end;
16.   finally
17.     HTTPClient.Free;
18.   end;
19. end;
20.  

I do not know why some links work while other don't. However all links work if just copy and paste into browser address bar.
I grabbed some random links from X and hacked around with the meta tags of privet, business, and government

links that failed
-----------------
yahoo.com
www.yahoo.com
http://yahoo.com
http://www.yahoo.com
https://infowars.com/
https://www.infowars.com/
http://stewpeters.com/
http://zeeemedia.com
x.com
www.x.com
http://x.com
http://www.x.com

Link with successful validation
+++++++++++++++++++
https://www.yahoo.com
https://x.com/i/status/1869969518908584302
https://x.com
https://www.yahoo.com
https://okdhslive.org/Default.aspx
 
Is the problem with Lazarus, the Brave browser, or me?
Is there any other web type component that might work better just for testing?