Recent

Author Topic: A Simple CGI Program - Why 'File Not Found'?  (Read 16086 times)

teamtempest

  • New Member
  • *
  • Posts: 10
A Simple CGI Program - Why 'File Not Found'?
« on: April 07, 2014, 07:48:23 pm »
Using Lubuntu 13.10, Lazarus 1.2, FPC 2.6.2.

This is not a question about where to put a CGI executable or how to execute it. The real question follows after a long preamble which should (with any luck) outline what the answer isn't.

In my baby steps way I learned about Python's simple built-in web servers. Ah! Python is already installed on Lubuntu! So I copied the website I'm working on from WinVista to Lubuntu and put it in a directory called 'www'.

The pages of the website are all static (that's what I'm trying to change, but baby steps...). So I executed this command from a terminal window in the 'www' directory.

Code: [Select]
python -m SimpleHTTPServer

Now Lubuntu's default Firefox browser can access the local website via either '127.0.0.1:8000/index.htm' (which is what the tutorial I found said) or 'localhost:8000/index.htm' (which I suspected would also work, and it does). All the pages work just like the actual website on my ISP (so: one live website and two local copies, one for Windows and one for Linux).

Incidentally this where I noticed in the terminal window log file that Firefox spontaneously asked the server for 'favicon.ico'. Twice. The server denied this file existed (since it didn't) - '404' both times. Hmm. So I read up on how non-standardised all this is, sighed, created a simple one, and put it in the root directory of all three versions of the website. Firefox likes it. IE9 doesn't. Oh well.

Next is to try to execute CGI program from a shiny new page with its form:

Code: [Select]
<form method="post" action="cgi-bin/pricelookupcgi.exe">

which I did not expect to work - 'SimpleHTTPServer' does not have CGI capabilities. And it didn't. The server returned a 'POST not supported' message. Baby steps...

Okay, this time I'll try

Code: [Select]
python -m CGIHTTPServer

instead. Oh, and maybe I should change that 250K Lazarus-generated binary for Windows to the 600K Lazarus-generated binary for Linux as well (oops!).

I still didn't expect this to work. The CGI library module in Python automatically sends a '200 OK' response before it actually tries to execute the script. Makes redirects within a script rather - messy - for one thing. And as my program uses redirects, I figured I'd have to edit that CGI module before things would behave properly. Still, I wanted to see what would happen. Baby steps...

What I got was a message in the browser

Code: [Select]
The CGI app reports an error:

File not found: '..\template\rateshow.tpl'

The thing about this particular message is that it is not coming from the server. It is coming from my CGI program. Specifically this part:

Code: [Select]

{ show rudimentary HTML error page }
{ - our own, not the server's }

procedure UIshowerror(const mesg: String);
begin

    writeok;

    writeln( Output, '<HTML><HEAD><TITLE>CGI APP ERROR</TITLE></HEAD><BODY>' );
    writeln( Output, '<P>The CGI application reports an error:</P>' );
    writeln( Output, '<H3>' + mesg + '</H3>' );
    writeln( Output, '</BODY></HTML>' );

end;

{ check if a file can be accessed }

function canaccess(const fname: String): Boolean;
begin

    canaccess := False;
    if ( FileExists(fname) ) then
        canaccess := True
    else
        UIshowerror( 'File not found: ' + fname );

end;

which I put in after having trouble with a local IIS7 server. Never did figure out why it happened; never needed to after discovering the 'tinyweb' server (which runs my CGI program just fine under Windows).

But now I see the same or a similar problem is happening with a much less sophisticated web server than IIS. At this point in the program the user input has been accepted and all the calculations done. What it's trying to do now is open an HTML template file and output it after replacing various markers in the template with the results of its calculations (sort of a primitive version of ASP).

Sez I to self, 'Self, this may be a directory separator thingy'. So now I know about 'PathDelim' and how to use it for portability. Nope, not the problem (all the '\'s are changed to '/'s in the error message, though).

Nor is it a case problem - all website filenames are lower case now (one of the first problems I discovered while playing with 'SimpleHTTPServer').

A permissions thingy? Lubuntu's file manager reports 'Anyone' can 'View', 'Only Owner' can 'Change Content' and 'Nobody' can 'Execute'. Which I assume mean 'read, write, execute', respectively. I could even change them using the file manger, but I would have thought universal read access was already sufficient.

So...what next?

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #1 on: April 08, 2014, 01:54:42 am »
So...what next?
I tend to believe the results you received from FileExists. Use full absolute path and see if it works.

Leledumbo

  • Hero Member
  • *****
  • Posts: 8746
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #2 on: April 08, 2014, 02:06:13 am »
You can't assume where your web server executes the CGI, always use absolute path to refer to resources.

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #3 on: April 08, 2014, 04:05:13 pm »
Of course, you can write to a log file as well to help debug what's going on e.g. (air code, untested, will probably saturate the ozone layer):
Code: [Select]
function canaccess(const fname: String): Boolean;
{$define cgidebug}
{$ifdef cgidebug}
var
  debuginfo: tstringlist;
{$endif}
begin

    canaccess := False;
   {$ifdef cgidebug}
    debuginfo:=tstringlist.create;
    try
      debuginfo.add('canaccess: looking for');
      debuginfo.add(fname);
      debuginfo.add('normalized path is:');
      debuginfo.add(sysutils.expandfilename(fname));
      debuginfo.savetofile('cgi.log'); // in a directory that cgi process can write to preferably
    finally
      debuginfo.free;
    end;
  {$endif}
    if ( FileExists(fname) ) then
        canaccess := True
    else
        UIshowerror( 'File not found: ' + fname );

end;

Further debugging tips:
http://wiki.lazarus.freepascal.org/CGI_Web_Programming#Debugging_CGI
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

teamtempest

  • New Member
  • *
  • Posts: 10
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #4 on: April 08, 2014, 05:44:11 pm »
Thank you for all the hints. Changing the path specification does turn out to alter the results. Not always in the way I expect, however. Perhaps there is something I still don't understand about the difference between absolute and relative paths.

I changed my 'canaccess()' function to:

Code: [Select]
{ check if a file can be accessed }

function canaccess(const fname: String): Boolean;
begin

    canaccess := False;
    if ( FileExists(fname) ) then
        canaccess := True
    else
        UIshowerror( 'File not found: ' + fname
                     + ' (' + ExpandFileName(fname) + ')' );

end;         

With no other changes, the reported error message became:

Code: [Select]
File not found: ../template/rateshow.tpl (/home/anton/template/rateshow.tpl)

The Python CGI webserver was started from the 'www' directory. Which is missing from the absolute path. I decided that this meant the relative path was interpreted in relation to the 'www' directory (where the server started from) rather than the 'cgi-bin' directory (where the CGI executable is).

Using my newfound knowledge of Linux paths and where my CGI app was looking, I changed the path header of the template files from

Code: [Select]
tmplPath = '..' + PathDelim + 'template' + PathDelim;

to

Code: [Select]
tmplPath = 'template' + PathDelim;

reasoning that while this is still a relative path, it should now be relative to the 'www' directory instead of its parent.

And so it is. My CGI app works as I expect. No 'File not found' problems.

So what about absolute paths? I'm not terribly keen on the idea of hard coding such a thing because it means potentially changing that hard code for every place I might want to put the result. But maybe I should try it just to check my understanding. So I changed the path header again:

Code: [Select]
tmplPath = '~' + PathDelim + 'www' + PathDelim + 'template' + PathDelim;

If I'm correct that should be a complete path from the root to my template file. BUT...

Code: [Select]
File not found: ~/www/template/rateshow.tpl (/home/anton/www/template/rateshow.tpl)

The expanded path is absolutely correct (no pun intended). That IS where that file is located.

Is there something I'm not understanding?

Leledumbo

  • Hero Member
  • *****
  • Posts: 8746
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #5 on: April 08, 2014, 07:15:15 pm »
Quote
I'm not terribly keen on the idea of hard coding such a thing because it means potentially changing that hard code for every place I might want to put the result
Use configuration file such as TINIFile or something else. That's what I do to all my web apps.
Quote
The expanded path is absolutely correct (no pun intended). That IS where that file is located.

Is there something I'm not understanding?
Only the shell understands that ~ must be expanded to /home/<username>, so you must expand it yourself first before passing it to any API looking for files.

teamtempest

  • New Member
  • *
  • Posts: 10
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #6 on: April 09, 2014, 06:17:10 am »
Huh. So I played around with a few more variations of file paths:

Code: [Select]
{        tmplPath = '..' + PathDelim + 'template' + PathDelim; }
{        tmplPath = 'template' + PathDelim; }
{        tmplPath = '~' + PathDelim + 'www' + PathDelim + 'template' + PathDelim; }
{        tmplPath = PathDelim + 'www' + PathDelim + 'template' + PathDelim; }
{        tmplPath = 'www' + PathDelim + 'template' + PathDelim; }

Only the second variation actually worked.

The expanded version of the fourth was the same as the unexpanded version.

The expanded version of the fifth had two consecutive 'www' directories listed. Hmm. So it was  pre-pended with the remainder of a complete path to the root from the directory the web server started in. Because it's a relative path? Is that what happened in the case of the second? That would explain why the one worked and the other didn't.

The first gets '..' replaced by a complete path to the root minus one directory.

The third gets '~' replaced by the home directory.

Hmm.

Dunno what shell expansion of filenames has to do with anything. I admit I know little about them, but my limited understanding suggests that any such expansion generally happens before a program starts running - unless a running program is not communicating with the OS directly but a shell intermediary?

Anyhow, clearly 'ExpandFileName()' knows how to put together a path. That's the whole point of using it to show what's happening in the first place, isn't it? However it manages to do so. Well then:

Code: [Select]
function canfind(const basename: String): String;
var

    expname : string;

begin

    canfind := '';
    expname := ExpandFileName( basename );
    if ( FileExists(expname) ) then
         canfind := expname
    else
        UIshowerror( 'File not found: ' + basename
                     + ' (' + expname + ')' );

end;

Using this function instead of the prior 'canaccess()', both the second and third variations work in the sense that when used my CGI application can successfully open the template file. The first, fourth and fifth fail with a wrong path.

My feeling at this point is that the third variation (with 'canfind()') is safer, as it should yield the same result whatever directory a server chooses to regard relative paths as originating from.

It still may have to be tailored to the actual deployment location (*sigh*), but perhaps nothing a few {$IFDEF}s can't handle.
« Last Edit: April 09, 2014, 06:19:48 am by teamtempest »

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #7 on: April 09, 2014, 12:06:17 pm »
The third gets '~' replaced by the home directory.
..
Dunno what shell expansion of filenames has to do with anything.
..
Anyhow, clearly 'ExpandFileName()' knows how to put together a path.
Linux kernel functions do not replace/expand/understand the tilde mark '~'. On the other hand, ExpandFileName() is an FPC/RTL function (not OS function) that imitates the shell and does that replacement. Notice that it will not replace ~<username> nor a directory that starts with ~<username>/ with the home directory of <username>.

Leledumbo

  • Hero Member
  • *****
  • Posts: 8746
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #8 on: April 09, 2014, 05:31:30 pm »
Quote
Because it's a relative path? Is that what happened in the case of the second?
Yes and yes. Basically every path that doesn't start with / in *nix is relative path.

teamtempest

  • New Member
  • *
  • Posts: 10
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #9 on: April 10, 2014, 04:22:28 pm »
Quote
My feeling at this point is that the third variation (with 'canfind()') is safer, as it should yield the same result whatever directory a server chooses to regard relative paths as originating from.

Aw, you all know that won't work with non-*nix systems that don't understand the '~' character. It might not even work on *nix systems if the website root directory doesn't happen to be stored (mounted?) somewhere in the '/user' tree (dunno; haven't tried it).

But I'm exceptionally lazy and don't want to mess around with OS conventions any more than I have to. So I tried something that appears to depend only on relations within the website itself.

Given this website layout:

www
  form1.htm
cgi-bin
  processform.cgi
template
  form1.tpl
  form2.tpl
  form3.tpl
  form4.tpl

'form1.htm' is what the user sees the first time the request is made. ' form1.tpl' is the same form after it's been filled in by 'processform.cgi'. It also handles the next user request (if any). The other forms are for clarifying requests and error reporting.

Initially the forms themselves specified absolute paths to the CGI application. The CGI application specified relative paths to the templates. And on Windows that did not work using IIS7 as the server because it could not find the template files. But it did work fine using 'tinyweb' as the server.

And it didn't work on Linux using python's CGIHTTPServer. Hence this thread.

I wrote a new function to dynamically create an absolute path to a template file based on whether or not the current directory is the one holding the CGI application. It assumes that the CGI application is being called from either 'form1.htm' or a page generated from a '*.tpl' file:

Code: [Select]
{ create full path to template file }

function makeabspath(const basename: String): String;
const

     cgidir = 'cgi-bin';

     fromsiteroot = 'template' + PathDelim;
     fromcgidir = '..' + PathDelim + 'template' + PathDelim;

var

    dirname, sitepath, abspath : string;

begin

    makeabspath := '';

    dirname := AnsiLowerCase( GetCurrentDir );
    if ( Pos(cgidir, dirname) > 0 ) then
         sitepath := fromcgidir + basename
    else
         sitepath := fromsiteroot + basename;
    abspath := ExpandFileName( sitepath );
    if ( FileExists(abspath) ) then
         makeabspath := abspath
    else
        UIshowerror( 'File not found: ' + sitepath + ' (' + abspath + ')' );

end;                             

Using the python server, all the template files work correctly. This even after I changed the "action" paths in the forms to be relative. 'form1.htm' uses 'cgi-bin/processform.cgi' but the templates use just 'processform.cgi'. It took a while to understand where the browser considered the generated pages were in relation to the application that produced them (roughly the time necessary to realize the address bar had been displaying it all along). The two different "action" paths suggest to me that the absolute template paths are in fact being produced using both prefixes, although I haven't verified that.

The only thing not working at this point is redirection (for when the user wants to cancel out of everything and go do something else). But I expected that because the default python CGI server is, what shall I say, overly optimistic about the results of calling any CGI application.

teamtempest

  • New Member
  • *
  • Posts: 10
Re: A Simple CGI Program - Why 'File Not Found'?
« Reply #10 on: April 11, 2014, 06:30:22 pm »
FWIW, commenting out this single line in the Python 2.7 module 'CGIHTTPServer.py'

Code: [Select]
       self.send_response(200, "Script output follows")

seems to be enough to permit my script's re-directs to work as I intend.

With that line in place, Firefox didn't seem to mind two consecutive '200' responses (one from Python and one from my CGI script), but didn't much like '200' followed by '303'. Commenting that line out and saving the result with a new name in the root directory of the local version of the website (because Python searches for names in its own module directory first) enabled successful testing.

BTW, 'CGHTTPServer' sets a number of web-related environment variables before executing any script, but not 'HTTP_HOST'. Just FYI...


 

TinyPortal © 2005-2018