Reading a text file from a web-site.

[J-G]

You can check in the wiki there's an auto update software already working for Lazarus applications.

I'll look at that option in detail once I've sorted the 'Download2File'.

Or you can make your own to learn a lot more =)

That's a more likely scenario

I'm intrigued on how your application behaves, can you share us at least a screenshot?!

No problem.
The image on the left is the full design form which includes the fall-back components when there is no internet available (our rehearsal room is in a 'blind' spot). The one on the right is the normal screen with the 'Disney' Concert selected.
The .PDFs are printed copies of the music, .CAP files are 'Playable' Music using the Capella Reader. Both of those are 'universal' (all voices use them) but the .MP3s are created by our MD for each of the 4 voices (SATB) so selecting the voice you want to hear changes the top selection line only.

[lainz]

Looking good =)

[J-G]

I've had a VERY productive morning successfully downloading files from the web-site to local directories and even automatically (as far as the end user is concerned) invoking the 'setup' program for Capella Reader.

I'm sure that bulk downloading of the music files will be a breeze (my next job) but I can't yet see a way to create a shortcut on the desktop for one of the support programs (which doesn't need 'installing').

I've found a directory string assignment for the desktop . . . .
    '%systemdrive%\users\%username%\Desktop/'

... but using that returns an error - presumably because it's a protected directory - so I'm looking for a key that will unlock that protection  -  or a hint as to the proper way to go about the job

[molly]

Let me start with saying that i do not have any experience with creating desktop shortcuts in code. What i do know is that i personally despise them, as they clutter my desktop.

So, as a suggestion (and perhaps to make life easier) would it be an idea to create your shortcuts into a specific directory and open that directory for the user on the desktop instead ? That leaves the desktop uncluttered and in case you must you could add a shortcut to that directory on the desktop.

Keep in mind that /users/ is specific for higher windows versions. Older windows (like xp i believe, still use documents and settings folder).

The privileges might be a problem but you should be able to store data at least for your current logged in user (at least the user who started your program).

Try to manually create a shortcut in there or try some code that writes a .txt file in there to see what exact error you receive. Explorer should be able to tell you what right the current user has in those directories.

PS: are you aware there are special functions present in unit SysUtils such as GetAppConfigDir, GetUserDir and others ?

[lainz]

The desktop does not requires privileges to create a shortcut, tested it in my own applications.

http://wiki.freepascal.org/Windows_Programming_Tips#Creating_a_shortcut_.28.lnk.29_file

Configuration files should be stored in %AppData% on modern Windows (GetAppConfigDir). I recommend you to store your stuff there instead of using Documents folder, and as molly said just provide a shortcut in your program or desktop to that folder.

[J-G]

@ lainz
Thanks for that pointer - I've been able to simply copy and paste the code and it works (but you knew that!) - I don't understand it, yet, I've just called the procedure with the name of the program and given it a name for the shortcut. What the second argument achieves I haven't needed to fathom, I've passed an empty string ('') and it was accepted. I presume it would be used to pass 'command line' arguments if they were needed.

@ molly
I do like desktop shortcuts but I can see your objections.
I'm dealing with a very wide mix of computing capabilities as far as the end users are concerned so anything that removes a need for them to make a decision is worthwhile.

I ought to have done a search on creating shortcuts before posting but I was so pleased with my progress and keen to report on it. I may have even found the link posted by lainz

I was only nominally aware of the functions you site in sysutils. There is so much code available I scarcely have time to investigate the procedures/functions I know I want to use.    In this particular case I don't need to discover the UserDir since that is determined during the initial run of the program using a SelectDirectoryDialog.
 

[J-G]

It has long been my practice to trap errors and deal with them before the end user is presented with a program crash. Mostly these are IO errors and easily dealt with IOResult but now I'm dealing with HTTP Response codes and although there are many pages of information about what they are and what they mean I can't find any information about 'testing' for them.

With IOResult - {$I-} & {$I+} around the call, and evaluating the result is easy but I have no idea how to check the status code returned.

Molly's function...

Code: Pascal  [Select][+][-]

1. function  Download2File(URL: String; Filename: String): boolean;
2. begin
3.   Result := false;
4.   with TFPHttpClient.Create(Nil) do
5.     try
6.       AllowRedirect := true;
7.       Get(URL, FileName);
8.       Result := true;
9.     finally
10.       Free;
11.   end;
12. end;
13.  

...uses 'Try.. Finally' but doesn't test the status code returned by 'Get()'.

The line 'AllowRedirect := true' handles the 3xx series status but I've now found that I get 4xx series for some files (don't know why yet - still under investigation) so would prefer to trap these and as a minimum warn the user that there was a problem rather than have the program crash.

I should also trap all 5xx series codes even though I haven't yet been presented with one, but presume that once I can test, I will trap all.

[molly]

you could perhaps take a look at this thread for a bit more advanced details with regards to error handling (getmem sums it up nicely), and this rfc for http status codes.

Get() method does not return a value, instead it uses ResponseStatusCode and ResponseStatusText to return information. 'Catching' the eror can be done with a try ... except block. More about exceptions can be read here (also note the remarks that can be read at the link concerning nesting).

[J-G]

Thanks molly,  I'm starting to think that I'm on a rock-climb and have reached an 'Overhang' 

Running your 'nasa_getmem' program, it compiles without an error regarding 'ResponseStatusCode' but using the same statement in my program tells me 'Identifier not found'. I can't see anywhere in nasa_getmem where it is declared and I've checked that I have Classes, SysUtils & fphttpclient in the [uses] clause so can't work out where you have declared it.

Notwithstanding that, I then tried to force an exception in nasa_getmem by changing the filename to an obviously non-existent one (just added 'A_' to the start) but stepping through, it never got to line 26 'on E:  Exeption ... '    - - -    at line 24 it threw a 404 Status Code.

In all probability I'm being a 'Numpty' but would appreciate guidance.

Going back to my code; I think I know why I'm receiving a 400 Status Code - even though it is inconsistent and imponderable.

I create the arguments for Get() from the URL, local-Directory & filename for each file I want to download using global string variables U and F which ultimately become URL and Filename in the Download2File function. Stepping through and watching U, F, URL and Filename I'm seeing that sometimes URL is an empty string rather than what U is. ?????  How can that be?

Further, as a test, I changed the call from Get(URL,Filename) to Get(U,F) but still got a 400 Status Code.

As you know, I have 6 sub-directories for each 'Concert', and currently have three concerts. One concert (Disney) all files download to all directories perfectly but 'Christmas' fails on one directory after the first 1 or 2 - even that is inconsistent!   The only thing that changes between each call to Download2File is the filename - the directory and URL are consistent, with the filename being added immediately prior to the call. Putting 'Disney' files into 'Christmas' they download correctly so I've even re-created the 'rogue' files but still get a failure.

In case it helps, here is the latest version of the Download2File function :

Code: Pascal  [Select][+][-]

1. function  Download2File(URL: String; Filename: String): boolean;
2. Var
3.   SC : byte;
4. begin
5.   Result := false;
6.   with TFPHttpClient.Create(Nil) do
7.     try
8.       AllowRedirect := true;
9.       Get(U,F);
10. //      Get(URL, FileName);
11.       Result := true;
12.     except
13.     end;
14.     if ResponseStatusCode = 200 then
15.       begin
16.         //success, do other things if needed
17.         Result := True;
18.       end
19.     else
20.       begin
21.         SC := ResponseStatusCode Div 100;
22.         case SC of
23.           1 : begin
24.               end;
25.           2 : begin
26.               end;
27.           3 : begin
28.               end;
29.           4 : begin
30.                 showmessage('Copy of '+F+' Failed with Code'+ResponseStatusCode);
31.               end;
32.           5 : begin
33.               end;
34.         end;
35.       end;
36.     finally
37.       Free;
38.   end;
39. end;
40.  

... preparing for the Status Code test.

I feel that there should be something else between lines 12 & 13 but can't suggest what.

and the procedure that creates U & F is:

Code: Pascal  [Select][+][-]

1. procedure TForm1.DL_CAPClick(Sender: TObject);
2. Var
3.   i   : byte;
4.   Fl  : string;
5. begin
6.   DL_Label.Show;
7.   DL_Progress.Show;
8.   application.ProcessMessages;
9.   For i := 0 to pred(C_Count) do
10.     begin
11.       Fl := C_List[i] + '.CAP';
12.       U  := CAP_URL + Fl;
13.       F  := MusDir + URL_List[6] + Fl;
14.       if Download2File(U,F) = false then
15.         begin
16.           showmessage('Download of '+F+' failed');
17.         end;
18.       DL_Progress.Position := trunc((succ(i)/C_Count)*100);
19.       application.ProcessMessages;
20.     end;
21.   Sleep(5000);
22.   DL_Label.Hide;
23.   DL_Progress.Hide;
24.   DL_CAP.Enabled:=false;
25. end;
26.  

One of the filenames that fails is 'Hark! The Herald.cap' and yes, I have tried without the exclamation mark. (Clutching at straws!)  Spaces are no problem in other filenames either  eg. '12 days AFTER Christmas.cap' doesn't fail. Thinking that file size might be the issue I've checked that, and 'Hark' is 16Kb but others (which don't fail) exceed 7Mb.

[molly]

Thanks molly,  I'm starting to think that I'm on a rock-climb and have reached an 'Overhang' 

Sorry to hear. Sometimes it is a bit difficult to judge correctly what you are able to 'see' and what not. No problem, then we simply take a bit smaller steps 

Running your 'nasa_getmem' program, it compiles without an error regarding 'ResponseStatusCode' but using the same statement in my program tells me 'Identifier not found'. I can't see anywhere in nasa_getmem where it is declared and I've checked that I have Classes, SysUtils & fphttpclient in the [uses] clause so can't work out where you have declared it.

That property belongs to TFpHttpClient.

That means that as long as you are between the creation and free ...

Code: [Select]

  with TFPHttpClient.Create(Nil) do
  begin
    // here the use of ResponseStatusCode is valid
    Free;
  end;
... the use of ResponseStatusCode is valid. To use programmers speak: as long as you are in scope of TFPHttpClient you can use/access this property.

Another method of doing things is:

Code: [Select]

var
  Client : TFPHttpClient;
begin
  Client := TFPHttpClient.Create(nil);
  // do many things with client
  if Client.ResponseStatusCode = 200 then blah
 Client.Free;
  // here you still could compile if Client.StatusCode = 200, but the Client variable was freed so does not contain a valid instance of TFPHttpClient anymore (usually results in access violation)
end;


Stepping through and watching U, F, URL and Filename I'm seeing that sometimes URL is an empty string rather than what U is. ?????  How can that be?

Sorry no idea atm. I would have to create similar circumstances and run code through the debugger (currently i am unable to do that. At soonest i can run that test tomorrow).


Some modifications to your code:

Code: Pascal  [Select][+][-]

1. function  Download2File(URL: String; Filename: String): boolean;
2. Var
3.   SC : byte;
4. begin
5.   Result := false;
6.   with TFPHttpClient.Create(Nil) do
7.     try
8.       AllowRedirect := true;
9.       Get(URL, FileName);
10.     except
11.       // if you reached here then something went amiss, but you had no code in here to do something with it. so let's fix that
12.      // 'catch' the mother of all exceptions -> can also be specific exception names but i do not know them from mind -> see source-code of fphhtpclient for info
13.       on E: Exception do
14.       begin
15.        WriteLn('Exception: ' + E.Message); // write the exception error message
16.        WriteLn('The client returned statuscode        : ', ResponseStatusCode);
17.        WriteLn('The client returned statusmessage : ', ResponseStatusText);
18.        // you could also do a case for ResponseStatusCode for instance and act according to the specific code (or range of codes)
19.      end;
20.     end;
21.     // Only when the code reached here _and_ the status code was 200 then we received the data correctly, if not then something is amiss and we must not act like the data was received at all.
22.     if ResponseStatusCode = 200 then
23.       begin
24.         //success, do other things if needed
25.         Result := True;
26.       end
27.     else  // if there was an exception then we should not reach this code below at all. The on Except should catch that. Nevertheless it would be interesting to know how we even got here, so we want to know the ResponseStatusCode (case below can be omitted, just write the ResponseStatusCode/ResponseStatusText).
28.       begin
29.         SC := ResponseStatusCode Div 100;
30.         case SC of
31.           1 : begin
32.               end;
33.           2 : begin
34.               end;
35.           3 : begin
36.               end;
37.           4 : begin
38.                 showmessage('Copy of '+F+' Failed with Code'+ResponseStatusCode);
39.               end;
40.           5 : begin
41.               end;
42.         end;
43.       end;
44.     finally
45.       Free;
46.   end;
47. end;
48.  

I feel that there should be something else between lines 12 & 13 but can't suggest what.

Yes that is affirmative, see also my modifications

and the procedure that creates U & F is:

I'll have a look at that code a bit later this evening. try to see if you can debug the code again, and reach the on except in case things go wrong. Please report back if that works better or not.

edit: typos

[molly]

The previous code was a quick writeup and i overlooked some errors. Sorry for that 

This should work better:

Code: Pascal  [Select][+][-]

1. program FireBird_Singers;
2.  
3. {$MODE OBJFPC}{$H+}
4. {.$DEFINE NO_LAZARUS}  // uncomment when using freepascal (else assume lazarus
5. uses
6.   Classes, SysUtils, fphttpclient;
7.  
8.  
9. var
10.   // Assuming that F is either a local form variable or a global variable
11.   // that contains the (online?) filename ??
12.   F : String;
13.  
14.  
15. // for your convenience. Just one WriteLn to replace with f.i.
16. procedure Log(S: String);
17. begin
18.   {$IFDEF NO_LAZARUS}
19.   WriteLn(S);
20.   {$ELSE}
21.   // replace this with something usable for your Lazarus form. Assuming
22.   // a memo named memo1 placed on form with name form1.
23.   Form1.Memo1.Lines.Append(S);
24.   {$ENDIF}
25. end;
26.  
27. procedure Log;
28. begin
29.   Log('');
30. end;
31.  
32. procedure Log(S: String; Args: array of const);
33. begin
34.   Log(Format(S, Args));
35. end;
36.  
37.  
38. function  Download2File(URL: String; Filename: String): boolean;
39. Var
40.   SC : byte;
41. begin
42.   Result := false;
43.   with TFPHttpClient.Create(Nil) do
44.   try
45.     AllowRedirect := true;
46.     try
47.       Get(URL, FileName);
48.     except
49.       // if the code reached here, then an exception was raised by one of the
50.       // above lines after the try statement. Unfortunately your original source
51.       // did not had any code here to do something with that fact. So let's
52.       // try to fix that.
53.       //
54.       // 'catch' the mother of all exceptions:
55.       //  You could also address a (or several) specific Exceptions. For
56.       //  TFPHTTPClient specific this is exception EHTTPClient.
57.       //  But do realize that other execute code (direct or indirectly)
58.       //  performed by TFPHTTPClient could cause any other type of exception,
59.       //  such as for example a EDivByZero exception or file IO related
60.       //  Exceptions.
61.       on E: Exception do
62.       begin
63.         Log('Exception: ' + E.Message); // write the exception error message
64.         Log('The client returned statuscode    : %d', [ResponseStatusCode]);
65.         Log('The client returned statusmessage : %s', [ResponseStatusText]);
66.         // you could f.i. also do a case for ResponseStatusCode and let your
67.         // code act according to the specific status-code (or range of codes)
68.       end;
69.     end;  // end try...except block
70.  
71.     // Only when the code reached here _and_ the status code was 200 then we
72.     // received the data correctly. If not then something is amiss and we must
73.     // act like the data was not received at all.
74.     if ResponseStatusCode = 200 then
75.     begin
76.       //success, do other things if needed
77.       Result := True;
78.     end
79.     else  
80.     // if there was an exception, then we should not have reached this code
81.     // at all. The on... Except should have catched that. Nevertheless it
82.     // would be interesting to know how we even got here, so we want to know
83.     // the ResponseStatusCode. We write the ResponseStatusCode/
84.     // ResponseStatusText.
85.     begin
86.       SC := ResponseStatusCode;
87.       Log('Copy of %s Failed with Code %d', [F, SC]);
88.     end;
89.   finally
90.     Free;
91.   end;  // end try... finally block
92.  
93. end;
94.  
95. begin
96. // Good url
97. //  Download2File('http://www.google.com/index.html', 'index');
98. // wrong URL = 404
99.   Download2File('http://www.google.com/make_error.html', 'index');
100. end.
101.  

Replace the begin ... end at the bottom (my main program entry) with your own routine that invokes a download. Start simple like i did with a static url on a known site. Then try to debug your DL_CAPClick code again. First we have to get the download to work and exceptions should be catched/handled as intended before continuing.

[J-G]

Yet again - thanks molly.

Just a quick post as I now have a rehearsal to go to for Sunday's concerts  - yes a matinee & evening performance - so have only briefly looked at your excellent and detailed response but the first thing I see is that with 'Create' on line 6 and 'Free' on line 37 I would surmise that 'ResponseStatusCode' on line 14 would be within scope.

[molly]

Just a quick post as I now have a rehearsal to go to for Sunday's concerts

I wish you fun and a good rehearsal !

...but the first thing I see is that with 'Create' on line 6 and 'Free' on line 37 I would surmise that 'ResponseStatusCode' on line 14 would be within scope.

I disagree 

(and yes, i overlooked that initially in my first response as well)

i've commented your original code

Code: Pascal  [Select][+][-]

1. function  Download2File(URL: String; Filename: String): boolean;
2. Var
3.   SC : byte;
4. begin
5.   Result := false;
6.   // scope starts
7.   with TFPHttpClient.Create(Nil) do
8.     try  // here starts the exception block
9.       AllowRedirect := true;
10.       Get(U,F);
11. //      Get(URL, FileName);
12.       Result := true;
13.     except  // This is the exception which is part of the exception block
14.     end;  // This ends the exception block
15.    // That also means that here on this line with comment, the with has ended. Ergo the TFPHttpClient is "out of scope"
16.    // rest of code omitted because it has no influence for the issue at hand
17.  

Now, let's make that a bit more clear by using explicit begin/end statements

Code: Pascal  [Select][+][-]

1. function  Download2File(URL: String; Filename: String): boolean;
2. Var
3.   SC : byte;
4. begin
5.   Result := false;
6.   // scope starts
7.   with TFPHttpClient.Create(Nil) do
8.   begin
9.      try  // here starts the exception block
10.        AllowRedirect := true;
11.        Get(URL, FileName);
12.        Result := true;
13.      except  // This is the exception which is part of the exception block
14.         // here should be statements that handle the exception(s)
15.      end;  // This ends the exception block
16.   end; // here on this line, the begin statement has been "ended"
17.   // rest of code omitted because it has no influence for the issue at hand
18.  

You can interpret try .. finally .. end and/or try .. except ... end as  begin ... finally ... end and begin ... except ... end. Try starts a "block" just like begin does.

... and that is why i disagreed with your conclusion 


As a heads up, my corrected code uses a nested exception block, outer being try ... finally and inner being try ... except. Remember that i made the comment to note the link that leads to explanation of nested exceptions ?

That is where the flow of my (corrected) code can get a bit complicated.

Whenever an exception occurs (no matter on which line it occurs) the flow of the code will 'jump' to except/finally (depending on whether the code is in a try ... finally -block or a try... except -block. Whenever that code has finished executing (because it reaches the end statement of that block) a non nested exception will exit the current block. In case of nested exceptions it will turn over control to the outer exception (unless executed lines are already part of the outer exception block, in which case the flow is returned to the last block). The outer exception will act in a similar manner and 'jump' directly to the except or finally statement.

And in theory you can nest exceptions as deep as you would like to, so that the flow can get so complicated that it is able to make your head spin  (and i do hope i wrote it correctly because i got lost myself when trying to write it down)

[lainz]

All the stuff can be played in the browser (PDF, MP3, Photos, Videos, Capella).. maybe is a lot easier to make a website with some interaction with JavaScript. Just my opinion.

Capella in the browser:
https://www.capella-software.com/us/index.cfm/support/hotline-issues/exchange-scores-with-other-musicians-part-3/

[molly]

One of the filenames that fails is 'Hark! The Herald.cap' and yes, I have tried without the exclamation mark. (Clutching at straws!)  Spaces are no problem in other filenames either  eg. '12 days AFTER Christmas.cap' doesn't fail. Thinking that file size might be the issue I've checked that, and 'Hark' is 16Kb but others (which don't fail) exceed 7Mb.

Are you sure that other files (with spaces in their name) succeed with downloading ?

Because i've tried several local http daemons and online http debuggers, and whenever i put a space into a request it fails (the returned status codes varies a lot though, from 400 to 505 to others).

I think i've also found out why. I was expecting fphttpclient to encode the url that is passed to method Get(), but that does not seem to be the case (oversight ?, left for user ?). So that should be done manually.

Note that spaces are not allowed in a URI/http request, see rfc 3986, or in case you are like me and get bored with reading rfc's, see stackoverflow for a quick list of accepted characters.

That might be the reason for your other experienced errors as well. As said, try the new download2file first with using simple tests, then make the same test with directly downloading the 'problem' files from your server. In case that does not provide enough usable information (from download2file) then please report. Make sure to mention the correct status code.

In order to solve the URI encoding, take a look at unit URIParser with its functions ParseURI and EncodeURI.

In Download2file the first line in that function now reads for me:

Code: [Select]

URL := VerifiedURL(URL); 
and the rVerified routine:

Code: Pascal  [Select][+][-]

1. function VerifiedURL(UnverifiedURL:String): String;
2. var
3.   URI: TURI;
4. begin
5.   URI := ParseURI(UnverifiedURL, false);
6.   VerifiedURL := EncodeURI(URI);
7. end;
8.  

Which solves the errors i've been receiving with regards to "spaced" filenames