Capturing output of OS command in 'reatime'

[fatmonk]

I'm using the following to capture the output of a OS console command.

My problem is that this code buffers all output until the command is complete before displaying it and I'd like to display the output in realtime, or at least line by line (while still being able to track when the command finishes execution if possible.

I'm thinking that I might be able to do this by appending the last chunk of buffer to the TMemo (mOutput) within the while...do, but I just can't figure it out.

Code: Pascal  [Select][+][-]

1. procedure TmainForm.ActionButtonClick(Sender: TObject);
2. const
3.   READ_BYTES = 2048;
4. var
5.   OutputLines: TStringList;
6.   MemStream: TMemoryStream;
7.   OurProcess: TProcess;
8.   NumBytes: LongInt;
9.   BytesRead: LongInt;
10.   userInput: String;
11. begin
12.  
13.   // A temp Memorystream is used to buffer the output
14.   MemStream := TMemoryStream.Create;
15.   BytesRead := 0;
16.  
17.   OurProcess := TProcess.Create(nil);
18.  
19.   OurProcess.Executable := 'netstat';
20.   OurProcess.Parameters.Add('-a');
21.  
22.   OurProcess.Options := [poUsePipes, poNoConsole];
23.   OurProcess.Execute;
24.  
25.   while True do
26.     begin
27.       // make sure we have room
28.       MemStream.SetSize(BytesRead + READ_BYTES);
29.  
30.       // try reading it
31.       NumBytes := OurProcess.Output.Read((MemStream.Memory + BytesRead)^, READ_BYTES);
32.       if NumBytes > 0 // All read() calls will block, except the final one.
33.         then begin
34.           Inc(BytesRead, NumBytes);
35.         end else
36.           BREAK // Program has finished execution.
37.     end;
38.  
39.   MemStream.SetSize(BytesRead);
40.  
41.   OutputLines := TStringList.Create;
42.   OutputLines.LoadFromStream(MemStream);
43.  
44.   mOutput.Lines.AddStrings(OutputLines);
45.   OutputLines.Free;
46.   OurProcess.Free;
47.   MemStream.Free;
48.  
49. end;

Am I on the right track? Is there a better way to do this? (Note, netstat is just used as an example here.. this could be any command that takes a while to run but outputs its progress on screen sequentially).

Thanks,

FM

[EDIT: Removed spurious end; and reformatted to make levels clearer]

[taazz]

you have one end to many the code to add the lines to the control (moutput.... etc)  is executed outside the loop that reads output from the process. also change the "while true" to "while ourProcess.Running" or something along those lines.

[fatmonk]

The extra end is because I have cut out a bunch of distracting code... I'll see if I can correct that so that the question makes more sense...

The code actually works fine with commands that return quickly - the output is correctly displayed in the TMemo. In fact it works with comands that take a long time to return as well but the application hangs while it is waiting for the command to complete before displaying anything.

I've changed the while true - it makes no difference in terms of the problem I'm asking about, but does seem a more correct way of doing things.

-FM

[taazz]

The extra end is because I have cut out a bunch of distracting code... I'll see if I can correct that so that the question makes more sense...

The code actually works fine with commands that return quickly - the output is correctly displayed in the TMemo. In fact it works with comands that take a long time to return as well but the application hangs while it is waiting for the command to complete before displaying anything.

I've changed the while true - it makes no difference in terms of the problem I'm asking about, but does seem a more correct way of doing things.

-FM

As I said, pay close attention to where you do form updates. Your code updates the control on the form only after the while loop has finished. you need to put it inside the loop and call the form.repaint as well.

[Thaddy]

In this case I would use a - ahum - timer for display, since the network throughput is probably much higher than the refresh rate of the monitor and either way human eyes are too slow to keep up with either. But indeed, your loop needs to post a message from within. If you do not want to loose information you will need to buffer... no other way.

[fatmonk]

@taazz As per my original question, I thought that writing to the TMemo from within the while...do was one way to do this /but/ I cannot clear and rewrite the whole of the TMemo content each time as it already contains previous output which needs to be kept.

As each 'chunk' of output is collected in the buffer MemStream I would like to append that new chunk to the end of the TMemo content. This would obviously need to be within the while...do loop but I cannot figure out how to get just the last chunk of what was read into the MemStream.

As you can see in the code I am using at the moment, the whole of MemStream is currently loaded into a TSTringList called OutputLines. I then use .AddLines() from TMemo to append these to mOutput.

Rather than loading the whole of MemStream into the TMemo at the end of the process, I'd liek to do it line by line within the while...do loop.

So.. what you are suggesting is, I believe, exactly what I want to do but I can't figure out how to do it (without overwriting the previous content of the TMemo).

-FM

[fatmonk]

OK, slightly different approach...

Code: Pascal  [Select][+][-]

1.       PreviousOutput:=mOutput.Lines
2.  
3.       OutputLines := TStringList.Create;
4.  
5.       OurProcess.Options := [poUsePipes, poNoConsole];
6.       OurProcess.Execute;
7.  
8.       while OurProcess.Running do
9.       begin
10.         // make sure we have room
11.         MemStream.SetSize(BytesRead + READ_BYTES);
12.  
13.         // try reading it
14.         NumBytes := OurProcess.Output.Read((MemStream.Memory + BytesRead)^, READ_BYTES);
15.         if NumBytes > 0 // All read() calls will block, except the final one.
16.           then
17.             begin
18.               Inc(BytesRead, NumBytes);
19.  
20.               MemStream.SetSize(BytesRead);
21.  
22.               OutputLines.LoadFromStream(MemStream);
23.  
24.               mOutput.Lines:=PreviousOutput;
25.               mOutput.Lines.AddStrings(OutputLines);
26.             end
27.         else
28.           BREAK // Program has finished execution.
29.       end;
30.  

But I get a runtime SIGSEGV crash which seems to be to do with trying to set the content of the TMemo to the PreviousOutput. (PreviousOutput is a TStrings).

-FM

[taazz]

1) lines.addstrings does not clear the existing data it only appends the new you do not have to back up the existing.
2) previousoutput is a pointer to the existing lines not a backup it is a waste of cpu cycles to assign it to back to the lines.
3)after each succesfull addstrings call you need to clear the memstream otherwise you will keep on appending the same data over and over again.

[fatmonk]

1) I understand that that is what lines.addstrings does - I am appending the output from the new command to the output from the previous command - because...
3) I do not want to clear the memstream in case I miss some data (as per Thaddy's comment). I want to buffer all of the new command's output incrementally so as not to miss anything.
2) I always trip up on this one... how do I get the content rather than a pointer?

-FM

[taazz]

1) I understand that that is what lines.addstrings does - I am appending the output from the new command to the output from the previous command - because...
3) I do not want to clear the memstream in case I miss some data (as per Thaddy's comment). I want to buffer all of the new command's output incrementally so as not to miss anything.

use a static sized buffer something along the lines of

Code: Pascal  [Select][+][-]

1. var
2.   .........
3.   vBuffer : array[0..read_bytes] of byte;
4. begin
5.    .......
6.     NumBytes := OurProcess.Output.Read(vBuffer[0], READ_BYTES);
7.    vbuffer[numbytes] := 0;//make sure that the string is null terminated.
8.    memStream.write(vbuffer[0],numbytes);
9.    OutputLines.Text := string(pchar(@vBuffer[0]));//break the string in to lines.
10.    memo1.lines.addstrings(outputlines);//append those lines to the memo
11.    oputputlines.clear; // clear the inserted lines from the next insert.
12.  

you get the point I hope.

2) I always trip up on this one... how do I get the content rather than a pointer?

simple you create a tstringlist and call its addstrings method eg

Code: Pascal  [Select][+][-]

1. var
2.   vPreviousOutput :TStringlist;
3. begin
4.   vPreviousOutput :=TStringlist.Create;
5.   try
6.     PreviousOutput.assign(mOutput.Lines);
7.     ......
8.   finally
9.     vPreviousOutput.free;
10.   end;
11.  

When you do something like memo1.Lines := stringlist; you actually call the properties setter method that it autocalls the memo1.lines.assign(stringlist);

[Edson]

Maybe you can check https://github.com/t-edson/UnTerminal

That library can capture the output of a process usign events in four different ways.

[ASerge]

Code: Pascal  [Select][+][-]

1.   OurProcess.Executable := 'netstat';
2.   OurProcess.Parameters.Add('-a');

netstat has problems with real-time redirection. Use additional utilities like https://www.codeproject.com/Articles/16163/Real-Time-Console-Output-Redirection or program switches that do not cause a delay: -na.
To check the operation of your program in real time, use "cmd /c dir c:\  /s". This is a long operation, and if your program sees output immediately, then it works correctly.