Recent

Author Topic: FPC cgi-bin handler, how to read content of posted files?  (Read 698 times)

BosseB

  • Full Member
  • ***
  • Posts: 141
FPC cgi-bin handler, how to read content of posted files?
« on: January 24, 2020, 05:43:54 pm »
I am working on a web application in Lazarus. It is a command line program, which will be used in the web interface as the cgi-bin handler for configuring a data collection system running on Raspberry Pi.

The app will output html pages formatted such that Apache can handle the serving of them to the caller. This works great so far.
It also reads the posted data when the user submits a request and adjusts the returned webpage accordingly.

I have this working fine now for regular form-oriented stuff like sending in config data and extracting them from the call and acting on them.

But now I have come to a showstopper when I want the user to select a definition file on his PC to send it into the system.
I use an input control of type "file" like so:
Code: Text  [Select]
  1. <form method=POST action="/cgi-bin/mywebhandler" enctype="multipart/form-data">
  2. Select file:
  3. <input type="file" name="myFile"><br><br>
  4.   <input type="submit">
  5. </form>
Then in the handler "mywebhandler" I receive in the post data the item
myFile=selectedfilename

I can extract the name this way, but how can I access the file content?
Is it supplied in the posted data, and if so in what format?
How should I read the data in my application?

What I do for post data is that I read each item in a loop with readln() until it returns an empty string. This is what I am required to do according to the specs I have read and it does work...
Every read should produce a name=value pair where data are web encoded such that spaces become + and  other characters are hex encoded and sent for example as %3B for the character ; etc.

But what happens with the contents of the file selected as shown above?

EDIT:
I modified the form tag in my example above according to PascalDragon's suggestion.
« Last Edit: February 12, 2020, 02:37:28 pm by BosseB »
--
Bo Berglund
Sweden

PascalDragon

  • Hero Member
  • *****
  • Posts: 958
  • Compiler Developer
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #1 on: January 25, 2020, 11:49:46 am »
You need to set the request encoding of the form to multipart/form-data like this:

Code: Text  [Select]
  1. <form method="post" action="/cgi-bin/mywebhandler" method="post" enctype="multipart/form-data">

Then the response will include the uploaded file in the StdIn stream as is mentioned here.

You might also want to look at TCgiApplication.ProcessMultiPart in $fpcdir/packages/fcl-web/src/base/cgiapp.pp or any other Pascal web framework.

Maybe you might be better off to switch to such a framework instead of doing it yourself...

BosseB

  • Full Member
  • ***
  • Posts: 141
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #2 on: February 12, 2020, 02:44:16 pm »
Regarding frameworks I have looked at some and they are overwhelmingly big and complex for my simple use. I am just doing a very limited configuration app interacted with via Apache on the RPi4...

So I am still wondering about how to actually retrieve the file body in my FreePascal webapp...
When the webapp is called it receives POST data in order to select what to do and perform the actions.
In this case there would be a file name submitted with the POST action. But where do I get the file body???
I am getting the posted data items from STD_IN line by line. I stuff that into an associative array object I have created myself using two stringlists to be later used in my app for retrieving what was sent. If the file body is in this list, then what format does it come in?

I have the webapp pretty well on its way with user input to set individual config parameters, but among these is also a control file that will be used during processing. This file (a config file in plain text) needs to be uploaded from the user onto the RPi4 file system through the web interface.
Then it will be listed in a selection control when configuring the process.
All files residing in the conf dir are listed as a selectable item in my config form, but I need a way to upload the files to the system...
--
Bo Berglund
Sweden

lucamar

  • Hero Member
  • *****
  • Posts: 2380
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #3 on: February 12, 2020, 03:36:33 pm »
So I am still wondering about how to actually retrieve the file body in my FreePascal webapp...
When the webapp is called it receives POST data in order to select what to do and perform the actions.
In this case there would be a file name submitted with the POST action. But where do I get the file body???

If you use something like this in your page (HTML4):

Code: Text  [Select]
  1. <form method="post" enctype="multipart/form-data">
  2.   <input type=file name="upFile" accept="*/*"></p>
  3.   <input type=submit value="Upload">
  4. </form>

then the data sent by the client to the server should already contain the (ASCII encoded) file contents; that is, it's responsability of the browser to send it in the post request. All you need do in the server is to read, decode and save it to a file (or do whatever).

Note that most web libs (Synapse, Indy, etc.) already contain code to do this mostly "automagically", more so when they allow you to build [Fast-]CGI or server modules, since this is a basic feature.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.4/2.0.6  - FPC 3.0.4 on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

devEric69

  • Full Member
  • ***
  • Posts: 247
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #4 on: February 12, 2020, 03:39:43 pm »
AFAIK, the principle of an HTTP POST method, is that only the fields (HTML controls) between <form> and </form> tags, are POSTed, and thus recoverable on the server side.
use: Linux 64 bits (Ubuntu 18.04 LTS).
Lazarus version: 2.0.4 (svn revision: 62502M) compiled with fpc 3.0.4 - fpDebug \ Dwarf3.

lucamar

  • Hero Member
  • *****
  • Posts: 2380
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #5 on: February 12, 2020, 03:49:52 pm »
AFAIK, the principle of an HTTP POST method, is that only the fields (HTML controls) between <form> and </form> tags, are POSTed, and thus recoverable on the server side.

Of course, but the value of an "input type=file" field is the file content (and optionally, though most clients send it, the filename),

Try it with a simple CGI logging the full request and you'll see.  ;D
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.4/2.0.6  - FPC 3.0.4 on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

BosseB

  • Full Member
  • ***
  • Posts: 141
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #6 on: February 12, 2020, 06:52:55 pm »
Just to clarify when I try to wrap my head around this:
By what I read here I think this is what happens:

The data posted to the server from a form submit arrives at the webapp as a series of name=value items that can be read one by one from the STD_IN of the webapp.
I am using this function up front of the webapp to retrieve all of the parameters:

Code: Pascal  [Select]
  1. function GetCallData: string;
  2. var
  3.   sParams, sTmp: string;
  4. begin
  5.   sParams := GetEnv('QUERY_STRING'); //Input from a GET call
  6.  
  7.   if sParams = '' then //Input from POST call
  8.   begin
  9.     repeat
  10.       Readln(sTmp);  //Get the data from STDIN (POST)
  11.       if sTmp <> '' then sParams := sParams + '&' + sTmp;
  12.     until sTmp = '';
  13.  
  14.     if Length(sParams) > 0 then
  15.       Delete(sParams,1,1); //Remove starting & created above
  16.   end;
  17.   { Replace the html escape sequences with the corresponding characters and
  18.   split string into lines on the & char so there is one line for each item in the resulting string.}
  19.  Result := CleanResponse(sParams); //
  20. end;
  21.  

So then I load the result from this call into an associative array object I have created using two TStringList containers, one for the name and one for the value.

This is then used to check for the expected values in the post.

Now, are you saying that there will be an item in the post data that is named as the control used on the form, which contains the content of the file? In that case what is the name of the file sent as?
Code: Text  [Select]
  1. <input type=file name="upFile">
will produce data as upFile=<content of the file here>?

My target files are text files so they contain a number of lineendings too. But they must be encoded, right? is this using %13%10 or maybe just %10 or what?

I don't know of a way to view what Apache sends to the cgi app, so it is hard to check this.

EDIT:
I will add logging and write out the call data to a logfile and have a look there...
« Last Edit: February 12, 2020, 07:01:21 pm by BosseB »
--
Bo Berglund
Sweden

lucamar

  • Hero Member
  • *****
  • Posts: 2380
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #7 on: February 12, 2020, 07:04:03 pm »
If I may be so bold, I's suggest you read RFC 7578 - Returning Values from Forms: multipart/form-data, where the whole mechanism is explained along with requests examples.

Also, if just for reference, you should do what I suggested above: write a simple CGI that just saves *all* the request, line by line (or byte by byte!), to a file, which you can then examine to your hearts content to see what the client sent *exactly*.

Once you know that, parsing that request in your real program should be easy as pie  ;D

ETA:

My target files are text files so they contain a number of lineendings too. But they must be encoded, right? is this using %13%10 or maybe just %10 or what?

It doesn't matter what they contain: a multipart/form-data request separates the fields with a "boundary" marker which is specified in one of the "normal" fields at the beggining, so the file field is whatever is enclosed between two of those markers, whether verbatim (i.e. as encode-type "plain/text") or encoded as specified in the field header.
« Last Edit: February 12, 2020, 07:14:30 pm by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.4/2.0.6  - FPC 3.0.4 on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

BosseB

  • Full Member
  • ***
  • Posts: 141
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #8 on: February 12, 2020, 07:52:45 pm »
OK will do the read...
Meanwhile I added this to one of my pages:

Code: HTML5  [Select]
  1. <form action="/cgi-bin/getwebpage" method="post" enctype="multipart/form-data">
  2. <input type="hidden" name="command" value="kallekula">
  3. <label for="cmdfile">Select a file:</label>
  4. <input type="file" name="cmdfile">
  5. <input type=submit value="Upload"></form>


This shows a file selector and a submit button, so I used it to select one of my files and then hit submit.

I had also modified the webapp "getwebpage" such that immediately after reading the incoming post data it would log it to file before continuing.

This is what I see in the log after the submit is executed:

Code: Text  [Select]
  1. 2020-02-12 19:42:04 DEBUG:
  2. -----------------------------25708448319199
  3. Content-Disposition: form-data; name="command"
  4.  
I had expected to see
Code: Text  [Select]
  1. command=kallekula
  2. cmdfile=007-12.cmd
  3. or else
  4. cmdfile=<content of 007-12.cmd>
  5.  
Oh, well, I guess it is not as simple as that. Will have to dig into the RFC...

--
Bo Berglund
Sweden

BosseB

  • Full Member
  • ***
  • Posts: 141
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #9 on: February 12, 2020, 09:06:34 pm »
OK will do the read...
....
Oh, well, I guess it is not as simple as that. Will have to dig into the RFC...
I have looked through the RFC document and have to admit I don't understand it much...

Quite possibly I have to look at using a php handler rather than my fpc webapp for uploading files.
I think there is more help to find regarding a php handler for file uploads.
I am pretty sure there are working examples out there showing php handling for this purpose.

Too bad I could not use fpc.
OTOH it will be just one php file to use for handling the file upload data, there is only one type of file to handle and it will be a pretty infrequent task too.
--
Bo Berglund
Sweden

lucamar

  • Hero Member
  • *****
  • Posts: 2380
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #10 on: February 13, 2020, 01:33:16 am »
A drawback of RFCs is that to understand them well you have to read not only those they supersede but also the related ones, in this case, those dealing with the base HTTP request/response mechanism as well as the CGI ones.

On the other hand that is relatively easy, since they are rather well  organized and link one to the others.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.4/2.0.6  - FPC 3.0.4 on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

zamronypj

  • Jr. Member
  • **
  • Posts: 56
    • Fano Framework, Free Pascal web application framework
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #11 on: February 13, 2020, 07:14:11 am »
Too bad I could not use fpc.

You definitely can use FPC for reading uploaded file. It is only matter of string parsing. Format for multipart/form-data explained quite clear in

https://stackoverflow.com/questions/4238809/example-of-multipart-form-data

especially Ciro Santili's answer.

For CGI application, web server sends POST request body via STDIN to CGI application. Application needs to read() STDIN until total string length being read is equal to Content-Length header value.

You can also inspect how browser (thorough Developer tools->Network, F12) sends HTTP request or using Wireshark.

This is example how to parse multpart/form-data

https://github.com/fanoframework/fano/blob/master/src/Http/Request/Helpers/MultipartFormDataParserImpl.pas
« Last Edit: February 13, 2020, 07:34:34 am by zamronypj »
Fano Framework, Free Pascal web application framework https://fanoframework.github.io
Personal Projects https://v3.juhara.com
Github https://github.com/zamronypj

BosseB

  • Full Member
  • ***
  • Posts: 141
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #12 on: February 13, 2020, 02:03:38 pm »
It wasn't worth the effort..
Instead I adapted a small php handler for the file upload I found on the web and it worked right away.
I put it into a /php dir on the server and used it as the target in the file upload form.
The php ends by redirecting to the page that called it so the new file list after the upload is shown via the webapp.
Works fine but is not as integrated as I would have wished.
Code: PHP  [Select]
  1. <?php
  2.     /* Upload command file to filesystem, then re-display the cmd list page */
  3.     $uploadDirectory = "/agi/monitorctrl/cmd/";
  4.     $cmdlist = "/cgi-bin/getwebpage?command=lst_cmd";
  5.  
  6.     $errors = []; // Store errors here
  7.  
  8.     $fileExtensionsAllowed = ['cmd','txt']; // These will be the only file extensions allowed
  9.  
  10.     $fileName = $_FILES['the_file']['name'];
  11.     $fileSize = $_FILES['the_file']['size'];
  12.     $fileTmpName  = $_FILES['the_file']['tmp_name'];
  13.     $fileType = $_FILES['the_file']['type'];
  14.     $fileExtension = strtolower(end(explode('.',$fileName)));
  15.  
  16.     $uploadPath = $uploadDirectory . basename($fileName);
  17.  
  18.     if (isset($_POST['submit'])) {
  19.  
  20.       if (! in_array($fileExtension,$fileExtensionsAllowed)) {
  21.         $errors[] = "This file extension is not allowed.";
  22.       }
  23.  
  24.       if ($fileSize > 100000) {
  25.         $errors[] = "File exceeds maximum size (100kB)";
  26.       }
  27.  
  28.       if (empty($errors)) {
  29.         $didUpload = move_uploaded_file($fileTmpName, $uploadPath);
  30.  
  31.         if ($didUpload) {
  32.           header("location: $cmdlist ");
  33.           die();
  34.         } else {
  35.           echo "An upload error occurred.";
  36.         }
  37.       }
  38.     }
  39. ?>
--
Bo Berglund
Sweden

lucamar

  • Hero Member
  • *****
  • Posts: 2380
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #13 on: February 13, 2020, 02:30:27 pm »
It wasn't worth the effort..
Instead I adapted a small php handler for the file upload I found on the web and it worked right away.

Well, I was planning on making a (relatively) small example using fcl-web this weekend but if you've already solved it I've better uses for my time ;D

I might make nevertheless; it wold be a good exercise. I'm a little rusty on web programming and need some (small) challenges to get up to date :P
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.4/2.0.6  - FPC 3.0.4 on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

PascalDragon

  • Hero Member
  • *****
  • Posts: 958
  • Compiler Developer
Re: FPC cgi-bin handler, how to read content of posted files?
« Reply #14 on: February 13, 2020, 10:51:43 pm »
Regarding frameworks I have looked at some and they are overwhelmingly big and complex for my simple use. I am just doing a very limited configuration app interacted with via Apache on the RPi4...

It's really not that complex to use fpWeb. Just look at the following, simple example:

Code: Pascal  [Select]
  1. program webupload;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   SysUtils, fphttpapp, HTTPDefs, httproute;
  7.  
  8. procedure HandleRequest(aRequest: TRequest; aResponse: TResponse);
  9. var
  10.   f: TUploadedFile;
  11.   s: String;
  12. begin
  13.   if ARequest.Method <> 'POST' then begin
  14.     AResponse.Content := '<html><head><title>File Test</title><head>' +
  15.       '<body><form action="/" method="POST" enctype="multipart/form-data"><input type="text" name="theedit" /><br />' +
  16.       '<label for="thefile">Select file:</label><input type="file" name="thefile" /><br />' +
  17.       '<input type="submit" value="Send" /></form></body></html>';
  18.     AResponse.SendResponse;
  19.   end else begin
  20.     f := ARequest.Files.FileByName('thefile');
  21.     if not Assigned(f) then
  22.       s := 'No file uploaded'
  23.     else
  24.       s := 'Size of file: ' + IntToStr(f.Size);
  25.     AResponse.Content := '<html><head><title>File Test Done</title><head>' +
  26.       '<body>Value of <i>theedit</i>: ' + ARequest.ContentFields.Values['theedit'] + '<br />' + s + '</body></html>';
  27.     AResponse.SendResponse;
  28.   end;
  29. end;
  30.  
  31. begin
  32.   HTTPRouter.RegisterRoute('*', @HandleRequest);
  33.   Application.Title:='httpproject1';
  34.   Application.Port:=4567;
  35.   Application.Initialize;
  36.   Application.Run;
  37. end.
  38.  

This is not CGI, but is its own server, but you could rather easily change that to CGI as well. And the posted file's data is available in TUploadedFile.Stream.

For more information see this tutorial.
« Last Edit: February 13, 2020, 10:55:58 pm by PascalDragon »