Recent

Author Topic: Polling TProcess whilst running  (Read 1982 times)

TheFifth

  • New member
  • *
  • Posts: 5
Polling TProcess whilst running
« on: May 07, 2021, 04:18:59 pm »
Hi all,

First let me say that I'm completely new to Lazarus and Pascal, so please be gentle with me!

I'm currently writing a GUI front end to TZXTools.  It's a CLI tool that allows the playback and manipulation of cassette images from retro 8bit machines.

I'm using a TProcess to start playback and am running it in a thread using the code I found here:  https://www.sigmdel.ca/michel/program/fpl/yweather/process_thread_en.html

All works great and I can playback the TZX file and clicking stop works and gracefully terminates the TProcess and thread.

The problem I have is that I need to poll the output of the TProcess as it is running.  The CLI provides updates on its progress that I need to parse to update my UI.

Having looked at the 'Reading large output' example, I see that reading the process output is blocking.  I was hoping I could read a smaller buffer amount from the buffer within the thread's 'while PlayProc.Running do' loop, however placing any sort or output polling in here only returns content after the process has finished.  Also, putting any sort of output read in the loop prevents the stop button being able to terminate the process early.

I currently have the following code:

Code: Pascal  [Select][+][-]
  1. unit playunit;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.   Classes, SysUtils;
  9.  
  10. var
  11.   strOutput: String;
  12.   lstStatus: TStringList;
  13.  
  14.  
  15. procedure Play(filename: string; PlayDone: TNotifyEvent = nil; TerminateProcess: boolean = true; CommandLineApp: string = '');
  16.  
  17. procedure StopPlaying;
  18.  
  19. implementation
  20.  
  21. uses
  22.   process, TZXTools;
  23.  
  24. var
  25.   StopPlay: boolean = false;
  26.   AppClosing: boolean = false;
  27.  
  28. type
  29.   TPlayThread = class(TThread)
  30.   private
  31.     procedure Done;
  32.     procedure ShowStatus;
  33.   protected
  34.     procedure Execute; override;
  35.   public
  36.     FOnPlayDone: TNotifyEvent;
  37.     FFilename: string;
  38.     FCommandLineApp: string;
  39.     FTerminateProcess: boolean;
  40.     constructor Create(const filename: string; PlayDone: TNotifyEvent = nil;
  41.        TerminateProcess: boolean = true; CommandLineApp: string = '');
  42.   end;
  43.  
  44. { TPlayThread }
  45.  
  46. constructor TPlayThread.Create(const filename: string;  PlayDone: TNotifyEvent;
  47.    TerminateProcess: boolean; CommandLineApp: string);
  48. begin
  49.   FOnPlayDone := PlayDone;
  50.   FTerminateProcess := TerminateProcess;
  51.   FFilename := filename;
  52.   FCommandLineApp := CommandLineApp;
  53.   inherited create(false);
  54.   FreeOnTerminate := true;
  55. end;
  56.  
  57. procedure TPlayThread.Done;
  58. begin
  59.   if assigned(FOnPlayDone) then
  60.     FOnPlayDone(self);
  61. end;
  62.  
  63. // Will play back the TZX in a thread to not block the UI
  64. procedure TPlayThread.ShowStatus;
  65. begin
  66.   TZXTools.strLastStatus := strOutput;
  67. end;
  68.  
  69. procedure TPlayThread.Execute;
  70. var
  71.   PlayProc: TProcess;
  72.   OutputStream : TStream;
  73.   BytesRead    : longint;
  74.   Buffer       : array[1..2048] of byte;
  75. begin
  76.   PlayProc := TProcess.create(nil);
  77.   try
  78.     PlayProc.executable := FCommandLineApp;
  79.     PlayProc.parameters.add('-v');
  80.     PlayProc.parameters.add(FFilename);
  81.     PlayProc.Options := [poUsePipes, poNoConsole];
  82.     PlayProc.execute;
  83.  
  84.     // Create a stream object to store the generated output in
  85.     OutputStream := TMemoryStream.Create;
  86.     lstStatus := TStringList.Create;
  87.        
  88.     while PlayProc.Running do begin
  89.       if StopPlay or AppClosing then begin
  90.         if StopPlay or FTerminateProcess then
  91.           PlayProc.terminate(1);
  92.         exit;
  93.       end
  94.       else
  95.        
  96.         // Poll the buffer
  97.         BytesRead := PlayProc.Output.Read(Buffer, 2048); // Read the buffer
  98.    
  99.         // Add the bytes that were read to the stream for later usage
  100.         OutputStream.Write(Buffer, BytesRead);
  101.  
  102.         // Read what's in the buffer
  103.         OutputStream.Position := 0;
  104.         lstStatus.LoadFromStream(OutputStream);
  105.         strOutput := lstStatus.Text;
  106.        
  107.         // Update status
  108.         synchronize(@Showstatus);
  109.    
  110.         sleep(1);
  111.     end;
  112.   finally
  113.    
  114.     // *** Below works fine to get output after playback has finished
  115.     // *** However I need to get output as playback proceeds
  116.    
  117.     // Create a stream object to store the generated output in
  118.     {*OutputStream := TMemoryStream.Create;
  119.     lstStatus := TStringList.Create;
  120.     repeat
  121.       // Poll the buffer
  122.         BytesRead := PlayProc.Output.Read(Buffer, 2048); // Read the buffer
  123.  
  124.         // Add the bytes that were read to the stream for later usage
  125.         OutputStream.Write(Buffer, BytesRead);
  126.     until BytesRead = 0;
  127.    
  128.     // Read what's in the buffer
  129.     OutputStream.Position := 0;
  130.     lstStatus.LoadFromStream(OutputStream);
  131.     strOutput := lstStatus.Text;
  132.    
  133.     // Update status
  134.     synchronize(@Showstatus);*}
  135.    
  136.     // Clean up
  137.     PlayProc.free;
  138.     OutputStream.Free;
  139.     lstStatus.Free;
  140.     if assigned(FOnPlayDone) and not AppClosing then
  141.       synchronize(@Done);
  142.   end;
  143. end;
  144.  
  145. procedure Play(filename: string; PlayDone: TNotifyEvent; TerminateProcess: boolean; CommandLineApp: string);
  146. begin
  147.   StopPlay := false;
  148.   TPlayThread.create(filename, PlayDone, TerminateProcess, CommandLineApp);
  149. end;
  150.  
  151. procedure StopPlaying;
  152. begin
  153.   StopPlay := true;
  154. end;
  155.  
  156. finalization
  157.   AppClosing := true;
  158. end.


As you can see, I have also added the buffer reading in the 'finally' section (currently commented out), which works perfectly if I want to return all output in one go after the process has run or the stop button has been clicked.

Is it possible to continuously poll a running process, or is it only possible to return the output in one go at the end?

Thanks in advance for any help.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 9462
  • FPC developer.
Re: Polling TProcess whilst running
« Reply #1 on: May 07, 2021, 04:39:40 pm »
Look at the tprocess.RunCommandLoop  (fpc/packages/fcl-process/src/processbody.inc in 3.2.0+) and tprocess.ReadInputStream.

Note how TInputPipeStream.NumBytesAvailable is checked before reading from output.

This is a trick to check how many bytes are available before reading them, so you never read more than there are, and thus don't block.

TheFifth

  • New member
  • *
  • Posts: 5
Re: Polling TProcess whilst running
« Reply #2 on: May 07, 2021, 05:43:39 pm »
Thanks for your reply, but I'm still struggling a little.

Your suggestion has fixed one of my issues, in that the stop button will now terminate the process early.  So that's progress!

I have change my loop in the thread to be:

Code: Pascal  [Select][+][-]
  1. while PlayProc.Running do begin
  2.       if StopPlay or AppClosing then begin
  3.         if StopPlay or FTerminateProcess then
  4.           PlayProc.terminate(1);
  5.         exit;
  6.       end
  7.       else
  8.        
  9.         // Work out if any bytes are available
  10.         BytesAvail := PlayProc.Output.NumBytesAvailable;
  11.        
  12.         if BytesAvail > 0 then begin
  13.           PrevLen := Length(strOutput);
  14.           SetLength(strOutput, PrevLen + BytesAvail);
  15.           PlayProc.Output.Read(strOutput[PrevLen + 1], BytesAvail);
  16.          
  17.           // Update status
  18.           synchronize(@Showstatus);
  19.        
  20.         end;
  21.      
  22.         sleep(1);
  23.     end;

Unfortunately this still only returns all of the CLI output after the process has finished.

So although I am calling 'synchronize(@Showstatus);' whenever there is data available, it seems to only be running once after the process has finished.

If I run the process manually in Terminal (MacOS), the following is output during playback:

00:00.000   0 Standard Speed Data Block      Program: TEST_PROG  (955 bytes)
00:06.095   1 Standard Speed Data Block      955 bytes of data
00:14.010   2 Standard Speed Data Block      Bytes: o          (start: 27392, 4800 bytes)
00:20.098   3 Standard Speed Data Block      4800 bytes of data
00:48.608     End of Recording

As you can see, the process takes a little over 48 seconds to run and outputs data at the start of each block it is playing.  However, TProcess only seems to be able to return any output after the process has finished.  It does return it all, but it's all in one go after it has finished.  I even added a counter within the 'Showstatus' procedure to check how often it is fired and it's definitely only once, after the process finishes.

Is it possible to grab this output as it happens?

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 9462
  • FPC developer.
Re: Polling TProcess whilst running
« Reply #3 on: May 07, 2021, 10:20:14 pm »
I have no idea. I haven't used OS X in years, but it is possible that some programs react to being piped. (e.g. to show colors etc)

Note that under Windows there is sometimes a problem with not polling stderr while polling stdout, though that shouldn't be a problem with such small output.

TheFifth

  • New member
  • *
  • Posts: 5
Re: Polling TProcess whilst running
« Reply #4 on: May 08, 2021, 01:49:39 pm »
Thanks again for your reply.

I tried also polling StdErr just to be sure and it doesn't make any difference.  The pipe definitely only contains data once the process has finished.  I can tell this because if I terminate the process early (and add a final output poll in the clean up code), there is never anything returned.

The interesting things is that StdErr does return data immediately, during the process run, so that is working fine.

I've also tested on Linux (Ubuntu 21.04) and it's the same there.  Data is only ever returned from the output after the process has completely finished.

I've not checked Windows, but I will install Lazarus on my Windows machine and test there too.

I'll also try running some different processes, to see if it's something specific to TZXTools that's causing the issue.

Thanks again for your input.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 9462
  • FPC developer.
Re: Polling TProcess whilst running
« Reply #5 on: May 08, 2021, 03:30:13 pm »
instead of an external program, try to execute something liek the following program using the code

Code: [Select]
begin
    for i:=0 to 49 do
       begin
          writeln('this is iteration ',i);
          flush(output);
          sleep(500);
       end;
end.

and see if that works incremental.

TheFifth

  • New member
  • *
  • Posts: 5
Re: Polling TProcess whilst running
« Reply #6 on: May 08, 2021, 04:39:06 pm »
Yes that works, so it's definitely something specific to TZXTools by the look of it.  TZXTools is written in Python, so not sure if there's something odd about the way it handles piped output.

Thanks for all your input.  I can probably work around the issue by keeping track of the play time and comparing it against the start time of each block.  So it's not the end of the world.

Thanks again for all your help.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2877
Re: Polling TProcess whilst running
« Reply #7 on: May 08, 2021, 07:50:41 pm »
Yes that works, so it's definitely something specific to TZXTools by the look of it.  TZXTools is written in Python, so not sure if there's something odd about the way it handles piped output.

Check whether it's buffering stdout. I'm by no means a Python guru, but certainly when I was doing a lot of Perl there was a bit of mandatory magic to make sure that buffering was minimised...

Code: Perl  [Select][+][-]
  1.   if (! $|) {
  2.     open(STDERR, '>&STDOUT') or
  3.          die "cannot redirect error messages to standard output - $!\n";
  4.     $| = 1;                             # Unbuffered output
  5.   }
  6.   print "Content-type: text/html\n\n";
  7.  

Since Python has to such a large extent supplanted Perl, I'd be surprised if such a thing were not available. I'd also /not/ be surprised if somebody who didn't expect his program to be run inside a wrapper didn't know about it.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

TheFifth

  • New member
  • *
  • Posts: 5
Re: Polling TProcess whilst running
« Reply #8 on: May 10, 2021, 12:38:55 pm »
Check whether it's buffering stdout. I'm by no means a Python guru, but certainly when I was doing a lot of Perl there was a bit of mandatory magic to make sure that buffering was minimised...

Thanks for the info, I'm also not that familiar with Python, so I'll see if I can get hold of the author and ask them.

 

TinyPortal © 2005-2018