How to run an external command and capture its output the "right" way

[arraybolt3]

Been banging my head against the wall for way too long at this point, figured it was finally time to ask for help.

I'm currently writing an application that does things with VirtualBox VMs, which means that I need to be able to run the VBoxManage command from within my application and then read whatever output it outputs to stdout. For this reason, I'm trying to find a cross-platform, decently performant way to run an external command and capture everything it prints to stdout into a string. This sounds pretty simple, but this has proven to be surprisingly difficult.

The first approach I tried was the seemingly perfect RunCommand() function from the FCL, in a way similar to this:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   {$IFDEF UNIX}
7.   cthreads,
8.   {$ENDIF}
9.   Interfaces, // this includes the LCL widgetset
10.   Forms, Process, Dialogs, Unit1
11.  
12. {$R *.res}
13.  
14. var
15.   VBoxVMListStr: String;
16. begin
17.   RunCommand('/usr/bin/VBoxManage', ['list', 'vms'], VBoxVMListStr);
18.   ShowMessage(VBoxVMListStr);
19. end.

This works perfectly fine on Linux - running this gives me a dialog box listing all of the VMs on the system. This seems to be a flop on Windows though; all that pops up is an empty dialog box. I tried several different combinations of TProcessOptions (poWaitOnExit, poUsePipes, poStderrToOutPut, and poNoConsole), all of which failed in the same way.

After having been thoroughly defeated with this attempt, I then proceeded to write a rather involved function to do what I thought RunCommand() should have done:

Code: Pascal  [Select][+][-]

1. function RunCaptureCommand(ProgramName: String;
2.   Parameters: array of String; out CmdResult: String): Boolean;
3. type
4.   T4KReadBuf = array[0..4095] of Byte;
5. var
6.   CmdProcess: TProcess;
7.   CmdParam: String;
8.   CmdReadBuf: T4KReadBuf;
9.   CmdReadLen: Integer;
10.   CmdOutput: TBytes = ();
11. begin
12.   CmdReadBuf := Default(T4KReadBuf);
13.   CmdProcess := TProcess.Create(nil);
14.   CmdProcess.Executable := ProgramName;
15.   with CmdProcess.Parameters do
16.   begin
17.     for CmdParam in Parameters do
18.     begin
19.       Add(CmdParam);
20.     end;
21.   end;
22.   CmdProcess.Options := [poUsePipes, poNoConsole];
23.   try
24.     CmdProcess.Execute;
25.   except
26.     CmdProcess.Free;
27.     RunCaptureCommand := False;
28.     Exit;
29.   end;
30.  
31.   while True do
32.   begin
33.     CmdReadLen := CmdProcess.Output.Read(CmdReadBuf, 4096);
34.     if (CmdReadLen = 0) and (not CmdProcess.Running) then break;
35.     SetLength(CmdOutput, Length(CmdOutput) + CmdReadLen);
36.     Move(CmdReadBuf, CmdOutput[Length(CmdOutput) - CmdReadLen], CmdReadLen);
37.   end;
38.   CmdResult := StringOf(CmdOutput);
39.  
40.   if CmdProcess.ExitCode = 0 then RunCaptureCommand := True
41.     else RunCaptureCommand := False;
42.   CmdProcess.Free;
43. end;

This one technically works on Linux, but it's horribly slow (for some commands it may take multiple seconds, it takes about a second just to list VMs like shown above). I fiddled with it in the debugger for a bit, and it appears to be hanging on the CmdProcess.Output.Read() call. I see in the documentation for TProcess.Output that it warns "reading from the stream may cause the calling process to be suspended when the created process is not writing anything to standard output, or to cause errors when the process has terminated." The former bit makes sense to me for a long-running external command, but the commands I'm running are very short-lived. Once they terminate, I don't understand why my application would block trying to read the remaining data in the stream. I really don't understand why trying to read from a terminated process's stream can result in errors; how is one supposed to read the last bits of text printed by a command before it exits if reading after the process terminates is unsafe?

I'm not finding anything in the docs that looks like it will actually work on both Linux and Windows, and that isn't potentially dangerous or slow. What's the right way to do what I'm trying to do?

(Edit: Not sure why but the forum software decided to turn all my paragraph breaks into double paragraph breaks. Fixed.)

[CM630]

Try this, it works for me:

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender:TObject);
2. var
3.   VBoxVMListStr: String;
4. begin
5.   RunCommand('cmd.exe', ['/c', 'C:\Progra~1\Oracle\VirtualBox\VBoxManage.exe list vms'], VBoxVMListStr); //or      RunCommand('cmd.exe', ['/c', '"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" list vms'], VBoxVMListStr);
6.   Memo1.Append (VBoxVMListStr);
7. end;  

[n7800]

This wiki article contains a wealth of useful information about running external programs. Pay particular attention to the "Reading large output" section.

[marcov]

For me simply

Code: Pascal  [Select][+][-]

1. {$mode delphi}
2. uses process;
3.  
4. var VBoxVMListStr : string;
5.  
6. begin
7.  RunCommand('C:\Progra~1\Oracle\VirtualBox\VBoxManage.exe',['list','vms'], VBoxVMListStr);  
8.  writeln(VBoxVMListStr );
9. end.

works, and simple

Code: [Select]

RunCommand('C:\Program files\Oracle\VirtualBox\VBoxManage.exe',['list','vms'], VBoxVMListStr); 

too.

HMM. If I compile with 3.2.4rc1 or 3.3.1 it works, with 3.2.2 not. So it seems to be a bug that is already fixed since 3.2.2

[n7800]

HMM. If I compile with 3.2.4rc1 or 3.3.1 it works, with 3.2.2 not. So it seems to be a bug that is already fixed since 3.2.2

It's strange, for me too. Although with Git, for example, there are no problems.

[arraybolt3]

Hmm, well that's unfortunate. The version of FPC in Debian 13 is 3.2.2 (the version with the bug), and the latest binaries for Lazarus for Windows also come with 3.2.2 bundled. I guess I'll have to use a manual workaround.

Thanks for the wiki article n7800, I'll dig into it.

[cdbc]

Hi
I recommend you try out FpcUpDeluxe It'll install pretty much any version of FPC & Lazarus available, for you... 
It's really good.
Regards Benny

[arraybolt3]

After a bit of experimentation, it looks like VBoxManage is just... weird, at least when running under Debian 13.

I read through the relevant parts of the https://wiki.freepascal.org/Executing_External_Programs wiki page (i.e. everything up to and including "Reading large output" in the TProcess section, the rest of it didn't look like it had much bearing on what I was trying to do), and carefully re-implemented the large output example, using VBoxManage rather than ls:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   {$IFDEF UNIX}
7.   cthreads,
8.   {$ENDIF}
9.   Interfaces, Forms, Unit1, Classes, SysUtils, Process;
10.  
11. {$R *.res}
12.  
13. const
14.   BUF_SIZE = 2048;
15.  
16. var
17.   AProcess: TProcess;
18.   OutputStream: TStream;
19.   BytesRead: longint;
20.   Buffer: array[1..BUF_SIZE] of byte;
21.  
22. begin
23.   AProcess := TProcess.Create(nil);
24.   AProcess.Executable := '/usr/bin/vboxmanage';
25.   AProcess.Parameters.Add('list');
26.   AProcess.Parameters.Add('vms');
27.   AProcess.Options := [poUsePipes];
28.   AProcess.Execute;
29.  
30.   OutputStream := TMemoryStream.Create;
31.   repeat
32.     BytesRead := AProcess.Output.Read(Buffer, BUF_SIZE);
33.     OutputStream.Write(Buffer, BytesRead);
34.   until BytesRead = 0;
35.  
36.   AProcess.Free;
37.   with TStringList.Create do
38.   begin
39.     OutputStream.Position := 0;
40.     LoadFromStream(OutputStream);
41.     writeln(Text);
42.     writeln('--- Number of lines = ', Count, '----');
43.   end;
44.  
45.   OutputStream.Free;
46. end.

Similar to my original code, this hangs for multiple seconds when run:

Code: Pascal  [Select][+][-]

1. [user ~/ks-dev/TestProject]% time ./project1
2. WARNING: The vboxdrv kernel module is not loaded. Either there is no module
3.          available for the current kernel (6.12.64-1.qubes.fc41.x86_64) or it failed to
4.          load. Please recompile the kernel module and install it by
5.  
6.            sudo /sbin/vboxconfig
7.  
8.          You will not be able to start VMs until this problem is fixed.
9. "TestVM1" {50d5bc49-8ced-4c6f-976d-e006d8e18708}
10. "TestVM2" {6deea8ae-951b-4981-b267-8043ddcf2f4d}
11. "TestVM3" {52505be3-3f79-4613-b664-a1a5605ed02d}
12.  
13. --- Number of lines = 10----
14. ./project1  0.01s user 0.01s system 0% cpu 5.141 total

(Ignore the bit about kernel modules; it's an artifact of my work environment and doesn't affect what I'm working on. In case it's useful, I'm running Lazarus in a Kicksecure 18 VM on a Qubes OS R4.3 host. Kicksecure 18 is based on Debian 13.)

However, if I replace "vboxmanage list vms" with "ls --recursive --all", the command executes very quickly as I would hope. I initially thought that maybe this was because VBoxManage was returning too little info and maybe that was getting Read() stuck, but that doesn't seem to be the case; if I remove the "--recursive" and just run "ls --all", it only returns a small bit of data (around 500 bytes in my instance) and still returns quickly.

After a bit more digging (and noticing the warning about handling stderr in the wiki"), I tried rewriting things to use TProcess.ReadInputStream instead, since that seems to be effectively a non-blocking variant of TProcess.Output.Read():

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   {$IFDEF UNIX}
7.   cthreads,
8.   {$ENDIF}
9.   Interfaces, Forms, Unit1, Classes, SysUtils, Process;
10.  
11. {$R *.res}
12.  
13. const
14.   BUF_SIZE = 2048;
15.  
16. var
17.   AProcess: TProcess;
18.   OutputStream: TStream;
19.   ErrStream: TStream;
20.   ReadStdout: Boolean;
21.   ReadStderr: Boolean;
22.   Buffer: array[1..BUF_SIZE] of byte;
23.  
24. begin
25.   AProcess := TProcess.Create(nil);
26.   AProcess.Executable := '/usr/bin/vboxmanage';
27.   AProcess.Parameters.Add('list');
28.   AProcess.Parameters.Add('vms');
29.   AProcess.Options := [poUsePipes];
30.   AProcess.Execute;
31.  
32.   OutputStream := TMemoryStream.Create;
33.   ErrStream := TMemoryStream.Create;
34.   repeat
35.     ReadStdout := AProcess.ReadInputStream(AProcess.Output, OutputStream, 1);
36.     ReadStderr := AProcess.ReadInputStream(AProcess.Stderr, ErrStream, 1);
37.   until (not ReadStdout) and (not ReadStderr) and (not AProcess.Running);
38.  
39.   AProcess.Free;
40.   with TStringList.Create do
41.   begin
42.     OutputStream.Position := 0;
43.     LoadFromStream(OutputStream);
44.     writeln(Text);
45.     writeln('--- Number of lines = ', Count, '----');
46.   end;
47.  
48.   OutputStream.Free;
49. end.

This version works quite fast ("time" says it takes about 0.137 seconds to run "vboxmanage list vms" and print the results), but it also comes at the cost of being very CPU-intensive. If I replace "vboxmanage list vms" with "sleep 10", the process naturally takes about 10 seconds to return, but it also consumes 100% of one CPU core the entire time it runs.

I also don't think the reason TProcess.Output.Read() was hanging for me is because I was failing to process stderr, since I can comment out the "ReadStderr := AProcess.ReadInputStream(AProcess.Stderr, ErrStream, 1);" line in my code and it still returns quickly.

Is there any way in Free Pascal to do the equivalent of C's select() function, but on a TStream? Or is there any way to get this to return fast without it also eating 100% of a CPU core until the process exits?

[arraybolt3]

Hi
I recommend you try out FpcUpDeluxe It'll install pretty much any version of FPC & Lazarus available, for you... 
It's really good.
Regards Benny

It looks neat, but unfortunately due to policies I don't set, the program I'm working on has to be build-able using a toolchain and libraries present in Debian 13. (The end goal is for the application to run on Windows, but we want to cross-compile the Windows binary in a trusted Debian 13 environment with no third-party dependencies. The only reason the Windows version of Lazarus got involved is because I was using it to test things more quickly.)

[marcov]

Some binaries try to detect if they are running from a terminal or not. (e.g. to show colors when on terminal and not when piped).

It is quite common in VM related programs:
https://gitlab.com/freepascal.org/fpc/source/-/issues/41146

[arraybolt3]

True. I tried going back to the TProcess.Output.Read() code and adding "AProcess.CloseInput" and "AProcess.CloseStderr" immediately after the process is executed to see if that would make any difference. It seems to have done nothing unfortunately.

I'll probably resort to using fpSelect (and whatever its Windows equivalent is, will cross that bridge when I get to it) on the underlying OS handle to wait for the fd to be readable.

[arraybolt3]

Minor update, I discovered that it is very non-intuitive to do the equivalent of an fpSelect() on a Windows file handle. However, I found that the TProcess-based solution works just fine on Windows (no five seconds of delay) unlike on Linux. Since the RunCommand() solution works on Linux but not on Windows, I now have two platform-specific ways of doing what I want, so I can now use ifdef's to get what I want. My code currently looks like this:

Code: Pascal  [Select][+][-]

1. {$ifdef windows}
2. function RunCaptureCommand(ProgramName: String;
3.   Parameters: array of String; out CmdResult: String): Boolean;
4. type
5.   T4KReadBuf = array[0..4095] of Byte;
6. var
7.   CmdProcess: TProcess;
8.   CmdParam: String;
9.   CmdReadBuf: T4KReadBuf;
10.   CmdReadLen: Integer;
11.   CmdOutput: TStringStream;
12. begin
13.   CmdReadBuf := Default(T4KReadBuf);
14.   CmdProcess := TProcess.Create(nil);
15.   CmdProcess.Executable := ProgramName;
16.   with CmdProcess.Parameters do
17.   begin
18.     for CmdParam in Parameters do
19.     begin
20.       Add(CmdParam);
21.     end;
22.   end;
23.   CmdProcess.Options := [poUsePipes, poNoConsole];
24.   try
25.     CmdProcess.Execute;
26.   except
27.     CmdProcess.Free;
28.     RunCaptureCommand := False;
29.     Exit;
30.   end;
31.  
32.   CmdOutput := TStringStream.Create;
33.   repeat
34.     CmdReadLen := CmdProcess.Output.Read(CmdReadBuf, 4096);
35.     CmdOutput.Write(CmdReadBuf, CmdReadLen);
36.   until CmdReadLen = 0;
37.   CmdResult := CmdOutput.DataString;
38.  
39.   if CmdProcess.ExitCode = 0 then RunCaptureCommand := True
40.   else RunCaptureCommand := False;
41.   CmdProcess.Free;
42.   CmdOutput.Free;
43. end;
44. {$else}
45. function RunCaptureCommand(ProgramName: String;
46.   Parameters: array of String; out CmdResult: String): Boolean;
47. begin
48.   RunCaptureCommand := RunCommand(ProgramName, Parameters, CmdResult,
49.     [poNoConsole]);
50. end;
51. {$endif}

Hopefully that will be enough, at least to begin with.

[marcov]

I'll probably resort to using fpSelect (and whatever its Windows equivalent is, will cross that bridge when I get to it) on the underlying OS handle to wait for the fd to be readable.

Do it directly on both the stderr and stdout handles. But easiest is to simply find out the differences between 3.2.2 and 3.2.4 and see if there is a workaround without coding your own.

Runcommand/Tprocess have had many fixes for such detail behaviour and options over the years, and starting your own will only make you hit one after the other each time again.

[Thaddy]

You can also simply just copy and compile the process unit from 3.2.3 fixes.
There are no dependencies on new features.

On a side note: I had quite often speed troubles with VirtualBox vm's under Windows 11 and related to stdin/out/err. I migrated most of them to WSL2 where possible. This may be unrelated, though.

[valdir.marcos]

HMM. If I compile with 3.2.4rc1 or 3.3.1 it works, with 3.2.2 not. So it seems to be a bug that is already fixed since 3.2.2

It's strange, for me too. Although with Git, for example, there are no problems.

Hmm, well that's unfortunate. The version of FPC in Debian 13 is 3.2.2 (the version with the bug), and the latest binaries for Lazarus for Windows also come with 3.2.2 bundled. I guess I'll have to use a manual workaround.

Thanks for the wiki article n7800, I'll dig into it.

So many years without a new FPC stable release has become a nightmare.

Minor update, I discovered that it is very non-intuitive to do the equivalent of an fpSelect() on a Windows file handle. However, I found that the TProcess-based solution works just fine on Windows (no five seconds of delay) unlike on Linux. Since the RunCommand() solution works on Linux but not on Windows, I now have two platform-specific ways of doing what I want, so I can now use ifdef's to get what I want. My code currently looks like this:

Code: Pascal  [Select][+][-]

1. {$ifdef windows}
2. function RunCaptureCommand(ProgramName: String;
3.   Parameters: array of String; out CmdResult: String): Boolean;
4. type
5.   T4KReadBuf = array[0..4095] of Byte;
6. var
7.   CmdProcess: TProcess;
8.   CmdParam: String;
9.   CmdReadBuf: T4KReadBuf;
10.   CmdReadLen: Integer;
11.   CmdOutput: TStringStream;
12. begin
13.   CmdReadBuf := Default(T4KReadBuf);
14.   CmdProcess := TProcess.Create(nil);
15.   CmdProcess.Executable := ProgramName;
16.   with CmdProcess.Parameters do
17.   begin
18.     for CmdParam in Parameters do
19.     begin
20.       Add(CmdParam);
21.     end;
22.   end;
23.   CmdProcess.Options := [poUsePipes, poNoConsole];
24.   try
25.     CmdProcess.Execute;
26.   except
27.     CmdProcess.Free;
28.     RunCaptureCommand := False;
29.     Exit;
30.   end;
31.  
32.   CmdOutput := TStringStream.Create;
33.   repeat
34.     CmdReadLen := CmdProcess.Output.Read(CmdReadBuf, 4096);
35.     CmdOutput.Write(CmdReadBuf, CmdReadLen);
36.   until CmdReadLen = 0;
37.   CmdResult := CmdOutput.DataString;
38.  
39.   if CmdProcess.ExitCode = 0 then RunCaptureCommand := True
40.   else RunCaptureCommand := False;
41.   CmdProcess.Free;
42.   CmdOutput.Free;
43. end;
44. {$else}
45. function RunCaptureCommand(ProgramName: String;
46.   Parameters: array of String; out CmdResult: String): Boolean;
47. begin
48.   RunCaptureCommand := RunCommand(ProgramName, Parameters, CmdResult,
49.     [poNoConsole]);
50. end;
51. {$endif}

Hopefully that will be enough, at least to begin with.

I can understand the necessity or your temporary solution until a new FPC stable release.