How determine slash/no-slash end of URL

[Michael Collier]

I'm looking for way to determine if a URL has a trailing slash or not.

An example of URL without trailing slash..
https://www.gov.uk/browse/driving

An example of URL with trailing slash..
https://www.gov.uk/
But is displayed in browser only as..
https://www.gov.uk

Trailing slashes seem to be hidden from chrome browser address bar.
So I can't rely on how the URL is displayed as a means to determine its actual value.
So need a way to confirm URL before storing in database as the application that uses these URLs needs them to `correct`.

I've tried checking for redirect but didn't work, any ideas what else to try?

Thanks,
Mike

[anse]

> But is displayed in browser only as...

You cannot rely the displayed url is valid, it's just for the user to see where he is. Firefox even hides the "https://" from it once the page is loaded.

On the server side, you might be able to access the REQUEST_URI environment variable, which should be always valid.

[Michael Collier]

On the server side, you might be able to access the REQUEST_URI environment variable, which should be always valid.

Thanks but I only have access to the client side of things though... I'm logging the URL of multiple websites that I have no control over..

[dsiders]

On the server side, you might be able to access the REQUEST_URI environment variable, which should be always valid.

Thanks but I only have access to the client side of things though... I'm logging the URL of multiple websites that I have no control over..

Most responses include link metadata with the canonical or alternate URIs. Beyond that... you should not care.

[Michael Collier]

Beyond that... you should not care.

The problem though is that the application I feed the URLs into does care (it seems to be able to determine the actual URL) and throws error if they don't match.

[anse]

You could check with a regular expression if the url has no path, and then ensure a slash exists at the end:

Code: Pascal  [Select][+][-]

1. uses RegExpr;
2.  
3. begin
4.   r := TRegExpr.Create('^https?\://[^/]+$');
5.   url := 'http://test.com';
6.   if r.Exec(url) then
7.     url := url + '/';
8.   r.Free;
9. end;
10.  

[Remy Lebeau]

The problem though is that the application I feed the URLs into does care (it seems to be able to determine the actual URL) and throws error if they don't match.

Sounds like the app is just doing plain string comparisons instead of using URL normalization & comparison rules (see RFC 3986 Section 6).  For example, when comparing HTTP/S URLs, an empty path is supposed to compare equivalent to "/", so it shouldn't matter if a trailing "/" is actually present or not.

[dbannon]

Michael, is this what you are looking for ?

Code: Pascal  [Select][+][-]

1. if not URL.EndsWith('/') then  URL := URL + '/';

Davo

[Thaddy]

I believe includetrailingpathdelimiter will also work on URL's.

Correction: Linux only, On windows it will add '\'.

[cdbc]

Hi
Yup, me too Thaddy, at least an *nixes...
Regards Benny

[Michael Collier]

The problem I'm having is more to do with how the server/browser handles slashes (adds or removes them without me seeing) and being able to determine whether a change has happened so I'm using the `correct` value in my database.

I've used TFPHttpClient to visit URL and track any redirects.

Here is example of it working ok for me..

Code: Pascal  [Select][+][-]

1. ==========================================
2. Added Un-Needed /
3. https://www.gov.uk/browse/driving/
4. ==========================================
5. --redirects--
6. src =https://www.gov.uk/browse/driving/
7. dest=http://www.gov.uk/browse/driving
8. src =http://www.gov.uk/browse/driving
9. dest=https://www.gov.uk/browse/driving <<==`correct`
10. ==========================================
11.  
12. ==========================================
13. Removed Needed /
14. https://examples.eze2e.com/command
15. ==========================================
16. --redirects--
17. src =https://examples.eze2e.com/command
18. dest=https://examples.eze2e.com/command/ <<==`correct`
19. ==========================================

But this method doesn't work on my tests for the host main page URL..
If I remove slash and visit https://www.gov.uk I don't get redirects to https://www.gov.uk/

The application I feed these URLs into acts like a browser (in fact it is a form of browser) and it waits for internal redirects to happen e.g. (slashes added/removed), so it needs to be using `correct` values for further tests to work e.g. confirming database URL is `correct`.

So I guess I'm looking for component that tracks internal redirects?

Edit: The image shows what happens when dragging URL into mousepad

[Thaddy]

Yes, you will with these links, but you can't automate that in this case because user input is required.
That is a sane - but small - security precaution. Especially for government sites.
Once the cookie is set, you never experience it again.
Bots usually can not set cookies like this.

[Remy Lebeau]

The problem I'm having is more to do with how the server/browser handles slashes (adds or removes them without me seeing) and being able to determine whether a change has happened so I'm using the `correct` value in my database.

The browser doesn't care, it will happily request a URL any way it is given.  It is the server that decides what the URL needs to look like.  So, I agree that you should query the URL live and if the server redirects it then update your database accordingly (only if it is a permanent redirect, not a temporary redirect).

But this method doesn't work on my tests for the host main page URL..
If I remove slash and visit https://www.gov.uk I don't get redirects to https://www.gov.uk/

Because there is no need for such a redirect, as they are logically the same URL.  This goes back to my earlier comment about needing to follow the normalization and comparison rules defined in RFC 3986.  For instance, you should consider normalizing the URLs that you store in your database.

The application I feed these URLs into acts like a browser (in fact it is a form of browser) and it waits for internal redirects to happen e.g. (slashes added/removed), so it needs to be using `correct` values for further tests to work e.g. confirming database URL is `correct`.

Then it is using broken logic, as there is no guarantee that a redirect will occur on any given URL.  That is entirely up to the server.  If you request a URL, you will get either a success, an error, or a redirect.  That is all you have to go on, so act accordingly.  If you request a URL and get a success, then the URL is 'correct' regardless of how the server decides to process it.

So I guess I'm looking for component that tracks internal redirects?

There is no such thing as an "internal redirect".  Either the server redirects or it does not.  That is an explicit response.

When you drag the URL from your browser address bar to your Mousepad, you are being given a normalized URL, not the exact text typed into the address bar.

[Thaddy]

There is no such thing as an "internal redirect".  Either the server redirects or it does not.  That is an explicit response.

But there is URL validation and that can work in a similar way.
In this case that happens.
(I had to delete my cache to replay this)