watching TProcess output

[AnthonyTekatch]

Is there a way to watch the TProcess output while an external command is running?

I have tried this simplified code below (modified from the wiki https://wiki.freepascal.org/Executing_External_Programs#Redirecting_input_and_output_and_running_under_root) and am having two problems:
 

• The output only shows up after the command completes. I would like to watch the arp command return its results as they come through like when I execute arp from the Linux command line.
• There is "An unhandled exception" crash when I include the commented-out stderr code. It has something to do with getting Proc.Stderr.NumBytesAvailable in the while loop but no problems with the first while loop. I am using Lazarus v3.6

Code: Pascal  [Select][+][-]

1. program BlockingExecute;
2. {$mode objfpc}{$H+}
3.  
4. uses
5.   Classes,Math,Process, sysutils;
6.  
7. var
8.     Proc: TProcess;
9.     CharBuffer: array [0..511] of char;
10.     ReadCount: integer;
11.     ExitCode: integer;
12.   begin
13.     Proc := TProcess.Create(nil);
14.  
15.       Proc.Options := [poUsePipes, poRunIdle, poStderrToOutPut, poNoConsole];
16.       Proc.CommandLine := '/usr/bin/arp';
17.       Proc.Execute;
18.  
19.       while Proc.Running or (Proc.Output.NumBytesAvailable > 0) or (Proc.Stderr.NumBytesAvailable > 0) do
20.       begin
21.         while Proc.Output.NumBytesAvailable > 0 do
22.         begin
23.           ReadCount := Min(512, Proc.Output.NumBytesAvailable);
24.           Proc.Output.Read(CharBuffer, ReadCount);
25.           Write(StdOut, Copy(CharBuffer, 0, ReadCount));
26.         end;
27.  
28.         //while (Proc.Stderr.NumBytesAvailable > 0) do
29.         //begin
30.           //ReadCount := Min(512, Proc.Stderr.NumBytesAvailable);
31.           //Proc.Stderr.Read(CharBuffer, ReadCount);
32.           //Write(StdOut, Copy(CharBuffer, 0, ReadCount));
33.         //end;
34.  
35.       end;
36.       ExitCode := Proc.ExitStatus;
37.       Proc.Free;
38.       Halt(ExitCode);
39. end.

[marcov]

The exception is probably because you include poStderrToOutPut. Then there is no stderr.

Try to flush stdout after writing.

[AnthonyTekatch]

The exception is probably because you include poStderrToOutPut. Then there is no stderr.

Try to flush stdout after writing.

Thank you for your suggestions.

There is an exception because I included poStderrToOutPut without uncommenting the handler, but if I include the handler then there is also an exception at the "while (Proc.Stderr.NumBytesAvailable > 0) do" line. That second check of Proc.Stderr.NumBytesAvailable causes a crash.

Flushing Stdout does not help since the while "Proc.Output.NumBytesAvailable > 0 do" line does not become true until the process is completely finished.

Here is the updated code that still only puts out the results when the program is completed instead of while it is running:

Code: Pascal  [Select][+][-]

1. program BlockingExecute;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   Classes,Math,Process, sysutils;
7.  
8. var
9.     Proc: TProcess;
10.     CharBuffer: array [0..511] of char;
11.     ReadCount: integer;
12.     ExitCode: integer;
13.   begin
14.     Proc := TProcess.Create(nil);
15.  
16.       Proc.Options := [poUsePipes, poRunIdle,
17.       //poStderrToOutPut,
18.       poNoConsole];
19.       Proc.CommandLine := '/usr/bin/arp';
20.       Proc.Execute;
21.  
22.       while Proc.Running or (Proc.Output.NumBytesAvailable > 0) or (Proc.Stderr.NumBytesAvailable > 0) do
23.       begin
24.         while Proc.Output.NumBytesAvailable > 0 do
25.         begin
26.           ReadCount := Min(512, Proc.Output.NumBytesAvailable);
27.           Proc.Output.Read(CharBuffer, ReadCount);
28.           Write(StdOut, Copy(CharBuffer, 0, ReadCount));
29.           Flush(StdOut);
30.         end;
31.  
32.         //while (Proc.Stderr.NumBytesAvailable > 0) do
33.         //begin
34.           //ReadCount := Min(512, Proc.Stderr.NumBytesAvailable);
35.           //Proc.Stderr.Read(CharBuffer, ReadCount);
36.           //Write(StdOut, Copy(CharBuffer, 0, ReadCount));
37.         //end;
38.  
39.       end;
40.       ExitCode := Proc.ExitStatus;
41.       Proc.Free;
42.       Halt(ExitCode);
43. end.

[Warfley]

You only check Output with:

Code: Pascal  [Select][+][-]

1. while Proc.Output.NumBytesAvailable > 0 do

But what about stderr?

You can take a look at this: https://github.com/Warfley/LazSetup/blob/master/utility/asyncprogress.pas#L312

[AnthonyTekatch]

You only check Output with:

Code: Pascal  [Select][+][-]

1. while Proc.Output.NumBytesAvailable > 0 do

But what about stderr?

You can take a look at this: https://github.com/Warfley/LazSetup/blob/master/utility/asyncprogress.pas#L312

Thank you for your suggestion.

I tried that code sample and altered it for running my desired program "arp". The output still does not show up until arp is completely finished instead of while it is running. Here is my altered version of your suggestion:

Code: Pascal  [Select][+][-]

1. program BlockingExecute;
2. // altered from https://github.com/Warfley/LazSetup/blob/master/utility/asyncprogress.pas#L312
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   Classes,Math,Process, sysutils;
7.  
8. var
9.   Proc: TProcess;
10.   buff: String = '';
11.   Arg: String;
12.   LastProgress: Double;
13. begin
14.   LastProgress := -1;
15.   Proc := TProcess.Create(nil);
16.   try
17.     proc.Executable := '/usr/bin/arp';
18.     Proc.Options := Proc.Options + [poUsePipes, poNoConsole];
19.  
20.     Proc.Execute;
21.     while Proc.Running do
22.     begin
23.       if Proc.Output.NumBytesAvailable > 0 then
24.       begin
25.         SetLength(buff, Proc.Output.NumBytesAvailable);
26.         Proc.Output.Read(buff[1], buff.Length);
27.         writeln(buff);
28.       end
29.       else if Proc.Stderr.NumBytesAvailable > 0 then
30.       begin
31.         SetLength(buff, Proc.Stderr.NumBytesAvailable);
32.         Proc.Stderr.Read(buff[1], buff.Length);
33.         writeln(buff);
34.       end
35.       else
36.         Sleep(10);
37.     end;
38.     if Proc.Output.NumBytesAvailable > 0 then
39.     begin
40.       SetLength(buff, Proc.Output.NumBytesAvailable);
41.       Proc.Output.Read(buff[1], buff.Length);
42.       writeln(buff);
43.     end;
44.     if Proc.Stderr.NumBytesAvailable > 0 then
45.     begin
46.       SetLength(buff, Proc.Stderr.NumBytesAvailable);
47.       Proc.Stderr.Read(buff[1], buff.Length);
48.       writeln(buff);
49.     end;
50.   finally
51.     Proc.Free;
52.   end;
53. end.
54.  

[AnthonyTekatch]

When I add the poStderrToOutPut option to Proc.Options, there is a crash caused by this line:

Code: Pascal  [Select][+][-]

1. else if Proc.Stderr.NumBytesAvailable > 0 then

Regardless of the crash, there is no way I can seem to get TProcess to produce asynchronous output while the external program /usr/bin/arp is running.

The /usr/bin/arp program does not produce an error according to:

Code: Pascal  [Select][+][-]

1. arp 2> error.log

There is no need to even trap StdErr in this case for testing which is why I disabled it to prove that TProcess is blocking the output until the command is complete.

[Roland57]

The output still does not show up until arp is completely finished instead of while it is running.

On my computer, the output of the arp command has only two lines. So I tried your program with another command:

Code: Pascal  [Select][+][-]

1.     proc.Executable := '/usr/bin/sh';
2.     proc.Parameters.Add('-c');
3.     proc.Parameters.Add('find . -name "*.pas" -print');

It works as expected: I see the output in real time. But (by the way) I see several lines at a time, and some lines are cut. So, if you wish to capture separately each line of the command output (as I imagine that you wish), you will have to search line endings in command output (and maybe to repair broken lines).

[marcov]

It could be that the pipe detection of ARP fails. You could try to execute it over the shell.

[AnthonyTekatch]

On my computer, the output of the arp command has only two lines.

The arp command will only display the cache of previously found IPs. Using nmap -n 192.168.1.0/24 will fill the cache, and arp will then display your network.

So I tried your program with another command:

Code: Pascal  [Select][+][-]

1.     proc.Executable := '/usr/bin/sh';
2.     proc.Parameters.Add('-c');
3.     proc.Parameters.Add('find . -name "*.pas" -print');

It works as expected: I see the output in real time. But (by the way) I see several lines at a time, and some lines are cut. So, if you wish to capture separately each line of the command output (as I imagine that you wish), you will have to search line endings in command output (and maybe to repair broken lines).

Your find example did work for me too. Thank you, that helps.

I will likely work with nmap by itself as arp does not play well with TProcess, and nmap provides all the details I need about the network anway.

[AnthonyTekatch]

It could be that the pipe detection of ARP fails. You could try to execute it over the shell.

Executing over the shell did not work any better, the output only comes after arp is completely finished. I think you are right about pipe detection with ARP. I will use nmap instead as it seems to be faster and provides the details I need.

[Warfley]

It's probably just the flushing behavior. Looking into the source code of ARP, it does not modify the file descriptor for STDOUT at all, so it's not that ARP is specifically doing something here.

My guess would be that the terminal sets the file descriptor to _IOLBF, which means the OS will buffer writes line by line and print out as soon as a new line was printed.
The pipes used by TProcess are afaik non configured so it depends on the OS how they are setup, but I'd assume that it's either _IOFBF, which buffers all writes until a flush comes or BUFSIZ, which buffers to a certain threshold (256 characters at least), before flushing.

[AnthonyTekatch]

It's probably just the flushing behavior. Looking into the source code of ARP, it does not modify the file descriptor for STDOUT at all, so it's not that ARP is specifically doing something here.

My guess would be that the terminal sets the file descriptor to _IOLBF, which means the OS will buffer writes line by line and print out as soon as a new line was printed.
The pipes used by TProcess are afaik non configured so it depends on the OS how they are setup, but I'd assume that it's either _IOFBF, which buffers all writes until a flush comes or BUFSIZ, which buffers to a certain threshold (256 characters at least), before flushing.

Interesting. Thanks.

[marcov]

My guess would be that the terminal sets the file descriptor to _IOLBF, which means the OS will buffer writes line by line and print out as soon as a new line was printed.
The pipes used by TProcess are afaik non configured so it depends on the OS how they are setup, but I'd assume that it's either _IOFBF, which buffers all writes until a flush comes or BUFSIZ, which buffers to a certain threshold (256 characters at least), before flushing.

Isn't that only a libc buffering system rather than OS (kernel?).

[Jurassic Pork]

Hello,
to do that , you can use TProcessEx from processutils unit of the fpcupdeluxe project.
In attachment project showing a ping .
arp works (see attachment)
Friendly, J.P

[AnthonyTekatch]

to do that , you can use TProcessEx from processutils unit of the fpcupdeluxe project.

Thank you for your ping example, it does show output as it occurs.

There must be something else wrong with arp, as the results only show when the arp command is completely done.

I have many network devices, and it takes about 20 seconds for arp to show them all. If you only have one device listed, you can run the nmap command to populate the arp cache so that more are discovered, then run arp again.