Send a stream to another program with piping

[aradeonas]

Hi,

I read wiki about TProcess but I couldn't understand is there a way for me to send a stream to another program and how?

[eny]

http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess

[Leledumbo]

Hi,

I read wiki about TProcess but I couldn't understand is there a way for me to send a stream to another program and how?

Read TProcess documentation. Input, Output and StdErr are the 3 standard streams that each program has. You read the program output by reading Output, and sending input by writing Input. In case of errors where you didn't supply poStdErrToOutput, StdErr will contain error output if any.

[Basile B.]

The way is the following. Traditionally it's more an IPC technic used under Linux systems (the fact to pipe a chain of process)

i add the following sample because sending input is not well explained in the wiki (the need to call 'CloseInput' is never mentioned !).

Code: [Select]

proc := TProcess.create;
proc.executable := 'stuff';
proc.options := proc.options + [posUsePipes];
proc.execute;
// if you know that the target will directly read:
proc.Input.Write(stuff, stuff_length);
// very important.
proc.CloseInput;
// wait manually: do not use poWaitonExit on large output size
while proc.running do sleep(10);
// reading the output is well explained in the wiki...


[aradeonas]

Thank you.
CloseInput was mentioned in http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess but as you said even after reading Wiki and before making this topic ,it is not clear for me how it work so I asked for maybe a simple demo as you said but I want to know after we call CloseInput the stream goes to process or while we right?
And also if it goes while we right if we have data back how can we get output while we feed it?
Im not a Linux user so I cant understand Piping very easily because I didnt saw it in Windows until now.

[derek.john.evans]

... it is not clear for me how it work so I asked for maybe a simple demo ....

Maybe explain what you are actually trying to achieve.

What is the process you are calling?
Is it one you have written, or is it a 3rd party console tool?
What data are you trying to give to the process?
And what data are you trying to receive?

There are other ways to send data to a process, which might be more suitable (reliable) if you are writing the process you are executing.
If its a 3rd party process, then what is it, and then someone can write up a quick example of how to communicate with the process.

[derek.john.evans]

o, and, is the process constantly running? ie: Is this a process that is constantly running, and therefore you need to handshake the communication?

[aradeonas]

I used it to run ExifTool o extract image info and no it isnt constantly running.
its a dirty code that what Im doing :

Code: [Select]

const
  READ_BYTES = 65536;
  BUF_SIZE = 2048;
var
  numbytes, bytesread: integer;
  s, cmdline: ansistring;
  i: integer;
  b: boolean;
  c: integer;
  p: TProcess;
  exitstatus: integer;
  outputstring: string;
  sa1: TStringArray;
  st: TFileStream;//should be in memory and in parts
  Buffer: array[0..127] of byte;
  ReadCount: integer;
  ReadSize: integer;
  AFileName: string;
  Buffer2: array[1..BUF_SIZE] of byte;
  OutputStream : TStream;
begin
  outputstring := '';
  exitstatus := 0;
  bytesread := 0;
  AFileName := ExtractFilePath(ParamStr(0)) + 'img1.jpg';
  try
    St := TFileStream.Create(AFileName, fmOpenRead);
     OutputStream := TMemoryStream.Create;

    p := TProcess.Create(nil);
    cmdline := ExtractFilePath(ParamStr(0)) + 'exiftool.exe';
    p.Executable := cmdline;
    p.Options := [poUsePipes];
    //p.Parameters.add('-s');  //tag name
    //p.Parameters.add('-j');  //json
    p.Parameters.add('-');
    p.Execute;
    while(st.Position<>st.Size) do begin
      ReadCount := st.Read(Buffer, Sizeof(buffer));
      p.Input.Write(Buffer, ReadCount);
      if (p.Output.NumBytesAvailable>0) then break;
    end;
     {
       BytesRead := p.Output.Read(Buffer2, BUF_SIZE);
    OutputStream.Write(Buffer2, BytesRead)
    }


    p.CloseInput;
    while p.Running do
    begin
      Setlength(outputstring, BytesRead + READ_BYTES);
      NumBytes := p.Output.Read(outputstring[1 + bytesread], READ_BYTES);
      if NumBytes > 0 then
        Inc(BytesRead, NumBytes)
      else
        Sleep(100);
    end;
    repeat
      Setlength(outputstring, BytesRead + READ_BYTES);
      NumBytes := p.Output.Read(outputstring[1 + bytesread], READ_BYTES);
      if NumBytes > 0 then
        Inc(BytesRead, NumBytes);
    until NumBytes <= 0;
    setlength(outputstring, BytesRead);
    exitstatus := p.exitstatus;
  except
    on e: Exception do
    begin
      setlength(outputstring, BytesRead);
    end;

  end;
  St.Free;

  if exitstatus <> 0 then
    b := False;
  if b then
  begin
    Memo1.Append(outputstring);
  end;   

it now work but its not clean and safe as it should as I explained.
For example I can use -fast param but in this case I should control sending and receiving data better so no need for extra data sending.

Take a look at here.

[Leledumbo]

i add the following sample because sending input is not well explained in the wiki (the need to call 'CloseInput' is never mentioned !).

Because it's not actually needed. Depending on the program being executed, you might need to send #13, #10 or close the input stream (effectively making the program think that the input stream is already EOF). Most interactive programs wait for #10, but I usually just go with LineEnding to ensure consistency with underlying platform.

[derek.john.evans]

Because it's not actually needed. Depending on the program being executed, you might need to send #13, #10 or close the input stream (effectively making the program think that the input stream is already EOF).

In the case of ExifTool, CloseFile seems to be required. Ive written a general function for piping using standard TStream's

Code: Pascal  [Select][+][-]

1. procedure ProcessExecute(const AExecutable, AParameters: String; const AInput, AOutput: TStream);
2. var
3.   LBuffer: array[word] of byte;
4. begin
5.   Assert(Assigned(AInput) and Assigned(AOutput));
6.   with TProcess.Create(nil) do begin
7.     try
8.       Options := [poUsePipes];
9.       ShowWindow := swoHIDE;
10.       Executable := AExecutable;
11.       Parameters.CommaText := AParameters;
12.       Execute;
13.       while Running and (Input.Write(LBuffer, AInput.Read(LBuffer, SizeOf(LBuffer))) > 0) do begin
14.         Sleep(10);
15.       end;
16.       CloseInput;
17.       while Running or (Output.NumBytesAvailable > 0) do begin
18.         if Output.NumBytesAvailable > 0 then begin
19.           AOutput.Write(LBuffer, Output.Read(LBuffer, SizeOf(LBuffer)));
20.         end;
21.         Sleep(10);
22.       end;
23.     finally
24.       Free;
25.     end;
26.   end;
27. end;
28.  

A little helper function for testing:

Code: Pascal  [Select][+][-]

1. function ProcessExecute(const AExecutable, AParameters: String; const AInput: TFileName): String;
2. var
3.   LInput: TFileStream;
4.   LOutput: TStringStream;
5. begin
6.   LInput := TFileStream.Create(AInput, fmOpenRead);
7.   try
8.     LOutput := TStringStream.Create(EmptyStr);
9.     try
10.       ProcessExecute(AExecutable, AParameters, LInput, LOutput);
11.       Result := LOutput.DataString;
12.     finally
13.       FreeAndNil(LOutput);
14.     end;
15.   finally
16.     FreeAndNil(LInput);
17.   end;
18. end;  
19.  

And a test case:

Code: Pascal  [Select][+][-]

1. Memo1.Text := ProcessExecute('exiftool.exe', '-', 'test.jpg');    
2.  

[Basile B.]

Yes, i observed the same, and this explains my previous answer.

I've tried 3 or 4 times during 4 or 6 monthes to refactor some code that executes an external program that can either take a file name as first parameter or read the file content from stdin when no params are passed.

It had never worked, i've tried everything: different line endings, TAsyncProcess.OnReadData event, wait a bit, process messages...really everything.

Than a few days ago i tried CloseInput after writing to input and it worked ! Even without writing an additional line ending! I think this is required when you want to write just after execution...a handle problem or a sharing mode issue or well...I don't know.

I'll be interested to have an expertise about the subject.

[derek.john.evans]

I think this is required when you want to write just after execution...a handle problem or a sharing mode issue or well...I don't know.

Maybe the process is reading until EOF (ie: while (!feof(stdin) { .... }), and the only way to send a eof to the process is to close the pipe. CR/LF don't indicate a end of file.

[Leledumbo]

In the case of ExifTool, CloseFile seems to be required. Ive written a general function for piping using standard TStream's

Just checked myself, yep it's true. When reading from stdin, it waits for EOF.