Executing external programs

[vitronix]

Hello everybody,

I found some code on the  wiki pages which works fine under Windows but hangs, after a while under Linux.
My operating systems are Windows 7 and Linux Mint. I tried to run the code under Debian and Ubuntu but it doesn't work.
The processor uses is an Intel I5 with 16GB RAM.
Here's the code:

Code: [Select]

program LargeOutputDemo;
 
{$mode objfpc}{$H+}
 
uses
  Classes, SysUtils, Process; // Process is the unit that holds TProcess
 
const
  BUF_SIZE = 2048; // Buffer size for reading the output in chunks
 
var
  AProcess     : TProcess;
  OutputStream : TStream;
  BytesRead    : longint;
  Buffer       : array[1..BUF_SIZE] of byte;
 
begin
  // Set up the process; as an example a recursive directory search is used
  // because that will usually result in a lot of data.
  AProcess := TProcess.Create(nil);
 
  // The commands for Windows and *nix are different hence the $IFDEFs
  {$IFDEF Windows}
    // In Windows the dir command cannot be used directly because it's a build-in
    // shell command. Therefore cmd.exe and the extra parameters are needed.
    AProcess.Executable := 'c:\windows\system32\cmd.exe';
    AProcess.Parameters.Add('/c');
    AProcess.Parameters.Add('dir /s c:\windows');
  {$ENDIF Windows}
 
  {$IFDEF Unix}
    AProcess.Executable := '/bin/ls';
    AProcess.Parameters.Add('--recursive');
    AProcess.Parameters.Add('--all');
    AProcess.Parameters.Add('-l');
  {$ENDIF Unix}
 
  // Process option poUsePipes has to be used so the output can be captured.
  // Process option poWaitOnExit can not be used because that would block
  // this program, preventing it from reading the output data of the process.
  AProcess.Options := [poUsePipes];
 
  // Start the process (run the dir/ls command)
  AProcess.Execute;
 
  // Create a stream object to store the generated output in. This could
  // also be a file stream to directly save the output to disk.
  OutputStream := TMemoryStream.Create;
 
  // All generated output from AProcess is read in a loop until no more data is available
  repeat
    // Get the new data from the process to a maximum of the buffer size that was allocated.
    // Note that all read(...) calls will block except for the last one, which returns 0 (zero).
    BytesRead := AProcess.Output.Read(Buffer, BUF_SIZE);
 
    // Add the bytes that were read to the stream for later usage
    OutputStream.Write(Buffer, BytesRead)
 
  until BytesRead = 0;  // Stop if no more data is available
 
  // The process has finished so it can be cleaned up
  AProcess.Free;
 
  // Now that all data has been read it can be used; for example to save it to a file on disk
  with TFileStream.Create('output.txt', fmCreate) do
  begin
    OutputStream.Position := 0; // Required to make sure all data is copied from the start
    CopyFrom(OutputStream, OutputStream.Size);
    Free
  end;
 
  // Or the data can be shown on screen
  with TStringList.Create do
  begin
    OutputStream.Position := 0; // Required to make sure all data is copied from the start
    LoadFromStream(OutputStream);
    writeln(Text);
    writeln('--- Number of lines = ', Count, '----');
    Free
  end;
 
  // Clean up
  OutputStream.Free;
end.

I hope that someone finds the bug.

Regards,

Jan Visser

[mdalacu]

I am using this options to process :

Options:=[poUsePipes,poStderrToOutPut];
ShowWindow:=swoHIDE;     

Instead of repeat...until i use

while AProcess.Running do
begin
......
end;

i hope this helps.

[Leledumbo]

Define "doesn't work".

Using BytesRead = 0 as sentinel is not a good idea. A process might buffer a while before flushing its output. Due to it, your repeat-until might exit right away at the first iteration. It's very dependent on the executed programs, but the problem with that condition applies generally.

[eny]

Using BytesRead = 0 as sentinel is not a good idea.

Please explain why this would not be ok.
On Windows it is clearly documented that this is the way to check for received/read data (see https://msdn.microsoft.com/en-us/library/windows/desktop/aa365467%28v=vs.85%29.aspx for more details).
The one thing missing from the simple example is error checking.

[Leledumbo]

explain why this would not be ok.

Read the next 3 sentences that follows.

[eny]

Read the next 3 sentences that follows.

The barber shaves all men in town that do not shave themselves.
Your assumptions are not correct as can be read from the M$ specifications.

[Leledumbo]

Your assumptions are not correct as can be read from the M$ specifications.

Apart from the need to remove Windows from your brain, you're right. In fact, the original code works for me (Manjaro Linux x86_64), I run it on a path without folder for simplicity. I should never run it from my home directory as it has millions of files scattered over thousands of folders and that could make ls runs for a loooong time.

[eny]

Apart from the need to remove Windows from your brain

There is no such need.
If you mean that a more 'platform independent' reference would be nice, I agree.
However reading pipes is a blocking action hence checking BytesRead works (is supposed to work) on any platform.

you're right. In fact, the original code works for me (Manjaro Linux x86_64), I run it on a path without folder for simplicity. I should never run it from my home directory as it has millions of files scattered over thousands of folders and that could make ls runs for a loooong time.

Well, it is just an example so any path would do to illustrate the point.

[eny]

My operating systems are Windows 7 and Linux Mint. I tried to run the code under Debian and Ubuntu but it doesn't work.

Tried to do a quick test on Ubuntu to verify your results, but unfortunately the latest lazarus cannot be installed on a vanilla Ubuntu desktop box.

[Cyrax]

Hello everybody,

I found some code on the  wiki pages which works fine under Windows but hangs, after a while under Linux.
My operating systems are Windows 7 and Linux Mint. I tried to run the code under Debian and Ubuntu but it doesn't work.
The processor uses is an Intel I5 with 16GB RAM.
Here's the code:

Code: [Select]

program LargeOutputDemo;
 
{$mode objfpc}{$H+}
 
uses
  Classes, SysUtils, Process; // Process is the unit that holds TProcess
 
const
  BUF_SIZE = 2048; // Buffer size for reading the output in chunks
 
var
  AProcess     : TProcess;
  OutputStream : TStream;
  BytesRead    : longint;
  Buffer       : array[1..BUF_SIZE] of byte;
 
begin
  // Set up the process; as an example a recursive directory search is used
  // because that will usually result in a lot of data.
  AProcess := TProcess.Create(nil);
 
  // The commands for Windows and *nix are different hence the $IFDEFs
  {$IFDEF Windows}
    // In Windows the dir command cannot be used directly because it's a build-in
    // shell command. Therefore cmd.exe and the extra parameters are needed.
    AProcess.Executable := 'c:\windows\system32\cmd.exe';
    AProcess.Parameters.Add('/c');
    AProcess.Parameters.Add('dir /s c:\windows');
  {$ENDIF Windows}
 
  {$IFDEF Unix}
    AProcess.Executable := '/bin/ls';
    AProcess.Parameters.Add('--recursive');
    AProcess.Parameters.Add('--all');
    AProcess.Parameters.Add('-l');
  {$ENDIF Unix}
 
  // Process option poUsePipes has to be used so the output can be captured.
  // Process option poWaitOnExit can not be used because that would block
  // this program, preventing it from reading the output data of the process.
  AProcess.Options := [poUsePipes];
 
  // Start the process (run the dir/ls command)
  AProcess.Execute;
 
  // Create a stream object to store the generated output in. This could
  // also be a file stream to directly save the output to disk.
  OutputStream := TMemoryStream.Create;
 
  // All generated output from AProcess is read in a loop until no more data is available
  repeat
    // Get the new data from the process to a maximum of the buffer size that was allocated.
    // Note that all read(...) calls will block except for the last one, which returns 0 (zero).
    BytesRead := AProcess.Output.Read(Buffer, BUF_SIZE);
 
    // Add the bytes that were read to the stream for later usage
    OutputStream.Write(Buffer, BytesRead)
 
  until BytesRead = 0;  // Stop if no more data is available
 
  // The process has finished so it can be cleaned up
  AProcess.Free;
 
  // Now that all data has been read it can be used; for example to save it to a file on disk
  with TFileStream.Create('output.txt', fmCreate) do
  begin
    OutputStream.Position := 0; // Required to make sure all data is copied from the start
    CopyFrom(OutputStream, OutputStream.Size);
    Free
  end;
 
  // Or the data can be shown on screen
  with TStringList.Create do
  begin
    OutputStream.Position := 0; // Required to make sure all data is copied from the start
    LoadFromStream(OutputStream);
    writeln(Text);
    writeln('--- Number of lines = ', Count, '----');
    Free
  end;
 
  // Clean up
  OutputStream.Free;
end.

I hope that someone finds the bug.

Regards,

Jan Visser

You should use  TProcess.Output.NumBytesAvailable property to check if there is data avaible in output buffer. Also you should do some calls to Sleep procedure so process which you have been executing with TProcess, have time to fill output buffer with data.

[eny]

Also you should do some calls to Sleep procedure so process which you have been executing with TProcess, have time to fill output buffer with data.

That is not the best idea in case the child process starts generating a lot of data.
Then you want to capture/process it as quickly as possible.
And since the read is blocking anyways there is no need for additional sleep's.