Recent

Author Topic: Capturing output of OS command in 'reatime'  (Read 4177 times)

fatmonk

  • Full Member
  • ***
  • Posts: 232
Capturing output of OS command in 'reatime'
« on: March 27, 2018, 04:01:41 pm »
I'm using the following to capture the output of a OS console command.

My problem is that this code buffers all output until the command is complete before displaying it and I'd like to display the output in realtime, or at least line by line (while still being able to track when the command finishes execution if possible.

I'm thinking that I might be able to do this by appending the last chunk of buffer to the TMemo (mOutput) within the while...do, but I just can't figure it out.

Code: Pascal  [Select][+][-]
  1. procedure TmainForm.ActionButtonClick(Sender: TObject);
  2. const
  3.   READ_BYTES = 2048;
  4. var
  5.   OutputLines: TStringList;
  6.   MemStream: TMemoryStream;
  7.   OurProcess: TProcess;
  8.   NumBytes: LongInt;
  9.   BytesRead: LongInt;
  10.   userInput: String;
  11. begin
  12.  
  13.   // A temp Memorystream is used to buffer the output
  14.   MemStream := TMemoryStream.Create;
  15.   BytesRead := 0;
  16.  
  17.   OurProcess := TProcess.Create(nil);
  18.  
  19.   OurProcess.Executable := 'netstat';
  20.   OurProcess.Parameters.Add('-a');
  21.  
  22.   OurProcess.Options := [poUsePipes, poNoConsole];
  23.   OurProcess.Execute;
  24.  
  25.   while True do
  26.     begin
  27.       // make sure we have room
  28.       MemStream.SetSize(BytesRead + READ_BYTES);
  29.  
  30.       // try reading it
  31.       NumBytes := OurProcess.Output.Read((MemStream.Memory + BytesRead)^, READ_BYTES);
  32.       if NumBytes > 0 // All read() calls will block, except the final one.
  33.         then begin
  34.           Inc(BytesRead, NumBytes);
  35.         end else
  36.           BREAK // Program has finished execution.
  37.     end;
  38.  
  39.   MemStream.SetSize(BytesRead);
  40.  
  41.   OutputLines := TStringList.Create;
  42.   OutputLines.LoadFromStream(MemStream);
  43.  
  44.   mOutput.Lines.AddStrings(OutputLines);
  45.   OutputLines.Free;
  46.   OurProcess.Free;
  47.   MemStream.Free;
  48.  
  49. end;

Am I on the right track? Is there a better way to do this? (Note, netstat is just used as an example here.. this could be any command that takes a while to run but outputs its progress on screen sequentially).

Thanks,

FM

[EDIT: Removed spurious end; and reformatted to make levels clearer]
« Last Edit: March 27, 2018, 05:17:23 pm by fatmonk »

taazz

  • Hero Member
  • *****
  • Posts: 5365
Re: Capturing output of OS command in 'reatime'
« Reply #1 on: March 27, 2018, 04:56:05 pm »
you have one end to many the code to add the lines to the control (moutput.... etc)  is executed outside the loop that reads output from the process. also change the "while true" to "while ourProcess.Running" or something along those lines.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

fatmonk

  • Full Member
  • ***
  • Posts: 232
Re: Capturing output of OS command in 'reatime'
« Reply #2 on: March 27, 2018, 05:13:41 pm »
The extra end is because I have cut out a bunch of distracting code... I'll see if I can correct that so that the question makes more sense...

The code actually works fine with commands that return quickly - the output is correctly displayed in the TMemo. In fact it works with comands that take a long time to return as well but the application hangs while it is waiting for the command to complete before displaying anything.

I've changed the while true - it makes no difference in terms of the problem I'm asking about, but does seem a more correct way of doing things.

-FM

taazz

  • Hero Member
  • *****
  • Posts: 5365
Re: Capturing output of OS command in 'reatime'
« Reply #3 on: March 27, 2018, 05:20:09 pm »
The extra end is because I have cut out a bunch of distracting code... I'll see if I can correct that so that the question makes more sense...

The code actually works fine with commands that return quickly - the output is correctly displayed in the TMemo. In fact it works with comands that take a long time to return as well but the application hangs while it is waiting for the command to complete before displaying anything.

I've changed the while true - it makes no difference in terms of the problem I'm asking about, but does seem a more correct way of doing things.

-FM
As I said, pay close attention to where you do form updates. Your code updates the control on the form only after the while loop has finished. you need to put it inside the loop and call the form.repaint as well.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Thaddy

  • Hero Member
  • *****
  • Posts: 10568
Re: Capturing output of OS command in 'reatime'
« Reply #4 on: March 27, 2018, 05:32:08 pm »
In this case I would use a - ahum - timer for display, since the network throughput is probably much higher than the refresh rate of the monitor and either way human eyes are too slow to keep up with either. But indeed, your loop needs to post a message from within. If you do not want to loose information you will need to buffer... no other way.

fatmonk

  • Full Member
  • ***
  • Posts: 232
Re: Capturing output of OS command in 'reatime'
« Reply #5 on: March 27, 2018, 05:58:30 pm »
@taazz As per my original question, I thought that writing to the TMemo from within the while...do was one way to do this /but/ I cannot clear and rewrite the whole of the TMemo content each time as it already contains previous output which needs to be kept.

As each 'chunk' of output is collected in the buffer MemStream I would like to append that new chunk to the end of the TMemo content. This would obviously need to be within the while...do loop but I cannot figure out how to get just the last chunk of what was read into the MemStream.

As you can see in the code I am using at the moment, the whole of MemStream is currently loaded into a TSTringList called OutputLines. I then use .AddLines() from TMemo to append these to mOutput.

Rather than loading the whole of MemStream into the TMemo at the end of the process, I'd liek to do it line by line within the while...do loop.

So.. what you are suggesting is, I believe, exactly what I want to do but I can't figure out how to do it (without overwriting the previous content of the TMemo).

-FM

fatmonk

  • Full Member
  • ***
  • Posts: 232
Re: Capturing output of OS command in 'reatime'
« Reply #6 on: March 27, 2018, 06:16:32 pm »
OK, slightly different approach...

Code: Pascal  [Select][+][-]
  1.       PreviousOutput:=mOutput.Lines
  2.  
  3.       OutputLines := TStringList.Create;
  4.  
  5.       OurProcess.Options := [poUsePipes, poNoConsole];
  6.       OurProcess.Execute;
  7.  
  8.       while OurProcess.Running do
  9.       begin
  10.         // make sure we have room
  11.         MemStream.SetSize(BytesRead + READ_BYTES);
  12.  
  13.         // try reading it
  14.         NumBytes := OurProcess.Output.Read((MemStream.Memory + BytesRead)^, READ_BYTES);
  15.         if NumBytes > 0 // All read() calls will block, except the final one.
  16.           then
  17.             begin
  18.               Inc(BytesRead, NumBytes);
  19.  
  20.               MemStream.SetSize(BytesRead);
  21.  
  22.               OutputLines.LoadFromStream(MemStream);
  23.  
  24.               mOutput.Lines:=PreviousOutput;
  25.               mOutput.Lines.AddStrings(OutputLines);
  26.             end
  27.         else
  28.           BREAK // Program has finished execution.
  29.       end;
  30.  

But I get a runtime SIGSEGV crash which seems to be to do with trying to set the content of the TMemo to the PreviousOutput. (PreviousOutput is a TStrings).

-FM

taazz

  • Hero Member
  • *****
  • Posts: 5365
Re: Capturing output of OS command in 'reatime'
« Reply #7 on: March 27, 2018, 06:22:05 pm »
1) lines.addstrings does not clear the existing data it only appends the new you do not have to back up the existing.
2) previousoutput is a pointer to the existing lines not a backup it is a waste of cpu cycles to assign it to back to the lines.
3)after each succesfull addstrings call you need to clear the memstream otherwise you will keep on appending the same data over and over again.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

fatmonk

  • Full Member
  • ***
  • Posts: 232
Re: Capturing output of OS command in 'reatime'
« Reply #8 on: March 27, 2018, 07:05:07 pm »
1) I understand that that is what lines.addstrings does - I am appending the output from the new command to the output from the previous command - because...
3) I do not want to clear the memstream in case I miss some data (as per Thaddy's comment). I want to buffer all of the new command's output incrementally so as not to miss anything.
2) I always trip up on this one... how do I get the content rather than a pointer?

-FM
« Last Edit: March 27, 2018, 07:10:05 pm by fatmonk »

taazz

  • Hero Member
  • *****
  • Posts: 5365
Re: Capturing output of OS command in 'reatime'
« Reply #9 on: March 28, 2018, 01:35:50 am »
1) I understand that that is what lines.addstrings does - I am appending the output from the new command to the output from the previous command - because...
3) I do not want to clear the memstream in case I miss some data (as per Thaddy's comment). I want to buffer all of the new command's output incrementally so as not to miss anything.
use a static sized buffer something along the lines of
Code: Pascal  [Select][+][-]
  1. var
  2.   .........
  3.   vBuffer : array[0..read_bytes] of byte;
  4. begin
  5.    .......
  6.     NumBytes := OurProcess.Output.Read(vBuffer[0], READ_BYTES);
  7.    vbuffer[numbytes] := 0;//make sure that the string is null terminated.
  8.    memStream.write(vbuffer[0],numbytes);
  9.    OutputLines.Text := string(pchar(@vBuffer[0]));//break the string in to lines.
  10.    memo1.lines.addstrings(outputlines);//append those lines to the memo
  11.    oputputlines.clear; // clear the inserted lines from the next insert.
  12.  
you get the point I hope.
2) I always trip up on this one... how do I get the content rather than a pointer?
simple you create a tstringlist and call its addstrings method eg
Code: Pascal  [Select][+][-]
  1. var
  2.   vPreviousOutput :TStringlist;
  3. begin
  4.   vPreviousOutput :=TStringlist.Create;
  5.   try
  6.     PreviousOutput.assign(mOutput.Lines);
  7.     ......
  8.   finally
  9.     vPreviousOutput.free;
  10.   end;
  11.  
When you do something like memo1.Lines := stringlist; you actually call the properties setter method that it autocalls the memo1.lines.assign(stringlist);
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Edson

  • Hero Member
  • *****
  • Posts: 1109
Re: Capturing output of OS command in 'reatime'
« Reply #10 on: March 28, 2018, 04:55:07 am »
Maybe you can check https://github.com/t-edson/UnTerminal

That library can capture the output of a process usign events in four different ways.
Lazarus 2.0.10 - FPC 3.2.0 - x86_64-win64 on Windows 8

ASerge

  • Hero Member
  • *****
  • Posts: 1692
Re: Capturing output of OS command in 'reatime'
« Reply #11 on: March 28, 2018, 02:29:56 pm »
Code: Pascal  [Select][+][-]
  1.   OurProcess.Executable := 'netstat';
  2.   OurProcess.Parameters.Add('-a');
netstat has problems with real-time redirection. Use additional utilities like https://www.codeproject.com/Articles/16163/Real-Time-Console-Output-Redirection or program switches that do not cause a delay: -na.
To check the operation of your program in real time, use "cmd /c dir c:\  /s". This is a long operation, and if your program sees output immediately, then it works correctly.

 

TinyPortal © 2005-2018