* * *

Author Topic: Why TProcess gives deadlock, whereas the same thing can be run in the terminal ?  (Read 2304 times)

comingnine

  • New member
  • *
  • Posts: 25
I would like to use TProcess to run a simple series of shell commands and collect the tiny amount of output.

As can be shown from ~/test.sh or ~/test_oneline.sh, this series of commands start script1 and then look for script2, and reports script2's PID and stops when script2 is found.

Both script file runs well in the terminal. However, when using with TProcess as shown below, the executable hangs for ever, indicating certain deadlock. However, the output is really small and the output pipe should not be full. I am really lost about the reason why the deadlock occurs here... Can you help me to solve this problem ? Many thanks ! :)

Code: Bash  [Select]
  1. [vagrant@localhost ~]$ ps aux | grep [^]]script
  2. [vagrant@localhost ~]$ ps aux | grep [^]]sleep
  3. [vagrant@localhost ~]$ cat ~/script1
  4. #!/bin/bash -x
  5. sleep 20
  6. ~/script2
  7. [vagrant@localhost ~]$ cat ~/script2
  8. #!/bin/bash -x
  9. sleep 4320
  10. [vagrant@localhost ~]$ cat ~/test.sh
  11. #!/bin/bash -x
  12.  
  13. nohup ~/script1 & bg_id=$!
  14. disown -a
  15. while true ; do
  16.   if ! ps -p $bg_id > /dev/null 2>&1 ; then
  17.     echo "script1 terminated"
  18.     break
  19.   fi
  20.   str=$(ps aux | grep [^]]script2)
  21.   BashRegex='^[[:alnum:]_]+\s+([[:alnum:]_]+)'
  22.   if [[ $str =~ $BashRegex ]] ; then
  23.     for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do
  24.       echo ${BASH_REMATCH[$i]}
  25.     done
  26.     break
  27.   else
  28.     sleep 5
  29.   fi
  30. done
  31. [vagrant@localhost ~]$ cat ~/test_oneline.sh
  32. #!/bin/bash -x
  33. nohup ~/script1 & bg_id=$! ; disown -a ; while true ; do if ! ps -p $bg_id > /dev/null 2>&1 ; then echo "script1 terminated" ; break; fi ; str=$(ps aux | grep [^]]script2) ; BashRegex='^[[:alnum:]_]+\s+([[:alnum:]_]+)' ; if [[ $str =~ $BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ; fi ; done
  34. [vagrant@localhost ~]$
  35.  


Code: Pascal  [Select]
  1. program test_freepascal;
  2. {$IFDEF FPC} {$MODE Delphi} {$ELSE} {$APPTYPE CONSOLE} {$ENDIF}
  3. uses SysUtils, Classes, Process;
  4.  
  5. procedure Test_TProcess_Executable_parameters; // http://wiki.freepascal.org/Executing_External_Programs#Reading_large_output
  6. const BUF_SIZE = 2048; // Buffer size for reading the output in chunks
  7. var
  8.   Proc: TProcess;
  9.   OutStream, ErrStream: TStream;
  10.   OutLines, ErrLines: TStrings;
  11.   BytesRead: LongInt;
  12.   Buffer: array[1..BUF_SIZE] of Byte;
  13. begin
  14.   Proc := TProcess.Create(nil);
  15.   try
  16.     Proc.Executable := '/bin/bash';
  17.     Proc.Parameters.Add('-c');
  18.     Proc.Parameters.Add('nohup ~/script1 & bg_id=$! ; disown -a ; while true ; do if ! ps -p $bg_id > /dev/null 2>&1 ; then echo "script1 terminated" ; break; fi ; str=$(ps aux | grep [^]]script2) ; BashRegex=''^[[:alnum:]_]+\s+([[:alnum:]_]+)'' ; if [[ $str =~ $BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ; fi ; done');
  19.     //Proc.Options := [poUsePipes, poWaitOnExit]; // Does not work...
  20.     Proc.Options := [poUsePipes]; // Does not work...
  21.     Proc.Execute;
  22.  
  23.     OutStream := nil; ErrStream := nil; OutLines := nil; ErrLines := nil;
  24.     try
  25.       OutStream := TMemoryStream.Create; ErrStream := TMemoryStream.Create; OutLines := TStringList.Create; ErrLines := TStringList.Create;
  26.  
  27.       repeat
  28.         BytesRead := Proc.Output.Read(Buffer, BUF_SIZE);
  29.         OutStream.Write(Buffer, BytesRead)
  30.       until BytesRead = 0;
  31.       OutStream.Position := 0;
  32.       OutLines.LoadFromStream(OutStream);
  33.  
  34.       repeat
  35.         BytesRead := Proc.StdErr.Read(Buffer, BUF_SIZE);
  36.         ErrStream.Write(Buffer, BytesRead)
  37.       until BytesRead = 0;
  38.       ErrStream.Position := 0;
  39.       ErrLines.LoadFromStream(ErrStream);
  40.  
  41.       Writeln('##### Out #####' + sLineBreak + OutLines.Text + sLineBreak + '##### Err #####' + sLineBreak + ErrLines.Text);
  42.     finally
  43.       OutStream.Free; ErrStream.Free; OutLines.Free; ErrLines.Free;
  44.     end;
  45.  
  46.   finally
  47.     Proc.Free;
  48.   end;
  49. end;
  50.  
  51. begin
  52.   Test_TProcess_Executable_parameters;
  53. end.
  54.  

tudi_x

  • Hero Member
  • *****
  • Posts: 524
for mere mortals would you mind uploading the two script files and the app folder.
i think i have a mint box.
Lazarus 1.8.4 64b on MX Linux "Horizon"

comingnine

  • New member
  • *
  • Posts: 25
Many thanks for your efforts !
The binary file is rather large and could not be uploaded as attachment.
The two script and the .lpr/.lpi files are attached.   :)


tudi_x

  • Hero Member
  • *****
  • Posts: 524
i would place the code in the scripts instead of TProcess parameters.
cat ~/test.sh in your initial post gives a lot of content whereas the scripts you loaded are very few lines.

if you have several commands maybe loop them in a TProcess execution for each.
Lazarus 1.8.4 64b on MX Linux "Horizon"

comingnine

  • New member
  • *
  • Posts: 25
As shown in the .lpr content below, the RunCommand version works as expected.   :)
However, it seems that RunCommand can only provide merged StdOut & StdErr, and I would like to retrieve StdErr content separately to see more clearly if something goes wrong.

There must be some root difference between RunCommand internal and the wiki usage of TProcess. Your insights will be appreciated very much  :o

Code: Pascal  [Select]
  1. program test_freepascal_runcommand;
  2. {$IFDEF FPC} {$MODE Delphi} {$ELSE} {$APPTYPE CONSOLE} {$ENDIF}
  3. uses SysUtils, Classes, Process;
  4.  
  5. // http://wiki.freepascal.org/Executing_External_Programs
  6. // http://forum.lazarus.freepascal.org/index.php/topic,38398.0.html
  7. procedure Test_RunCommand;
  8. var
  9.   Output: string;
  10. begin
  11.   RunCommand(
  12.     '/bin/bash',
  13.     ['-c',
  14.      'nohup ~/script1 & bg_id=$! ; disown -a ; while true ; do if ! ps -p $bg_id > /dev/null 2>&1 ; then echo "script1 terminated" ; break; fi ; str=$(ps aux | grep [^]]script2) ; BashRegex=''^[[:alnum:]_]+\s+([[:alnum:]_]+)'' ; if [[ $str =~ $BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ; fi ; done'
  15.     ],
  16.     Output,
  17.     []
  18.   );
  19.   Writeln(Output);
  20. end;
  21.  
  22. begin
  23.   Test_RunCommand;
  24. end.  
  25.  

comingnine

  • New member
  • *
  • Posts: 25
i would place the code in the scripts instead of TProcess parameters.
cat ~/test.sh in your initial post gives a lot of content whereas the scripts you loaded are very few lines.

Both ~/test.sh and ~/test_oneline.sh works as expected.
Furthermore, calling directly /home/vagrant/test.sh as TProcess.Executable does not work either. I am confused. :o

if you have several commands maybe loop them in a TProcess execution for each.

Multiple TProcess.... Two TProcess instances in the wiki page already look complicated enough.

comingnine

  • New member
  • *
  • Posts: 25
A working version which mimics the Process.internalRuncommand procedure...
Code: Pascal  [Select]
  1. procedure MoreTest_TProcess_Executable_Parameters_ReadOnlyIfAvailable; // Try to mimic Process.internalRuncommand
  2. const BUF_SIZE = 65536; // Read at most BUF_SIZE bytes at one time
  3. var
  4.   Proc: TProcess;              
  5.   OutMemStream, ErrMemStream: TMemoryStream;
  6.   OutNumBytes, ErrNumBytes: LongInt;
  7.   OutLines, ErrLines: TStrings;
  8.   Available, BytesRead: LongInt;
  9. begin
  10.   Proc := TProcess.Create(nil);
  11.   try
  12.     //Proc.Executable := '/home/vagrant/test.sh';
  13.     //(*
  14.     Proc.Executable := '/bin/bash';
  15.     Proc.Parameters.Add('-c');
  16.     Proc.Parameters.Add('nohup ~/script1 & bg_id=$! ; disown -a ; while true ; do if ! ps -p $bg_id > /dev/null 2>&1 ; then echo "script1 terminated" ; break; fi ; str=$(ps aux | grep [^]]script2) ; BashRegex=''^[[:alnum:]_]+\s+([[:alnum:]_]+)'' ; if [[ $str =~ $BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ; fi ; done');
  17.     //*)
  18.     //Proc.Options := [poUsePipes, poWaitOnExit]; // Also works...
  19.     Proc.Options := [poUsePipes];
  20.     Proc.Execute;
  21.  
  22.     OutMemStream := nil;
  23.     ErrMemStream := nil;
  24.     OutLines := nil;
  25.     ErrLines := nil;
  26.     try
  27.       OutMemStream := TMemoryStream.Create;
  28.       ErrMemStream := TMemoryStream.Create;
  29.       OutLines := TStringList.Create;
  30.       ErrLines := TStringList.Create;
  31.  
  32.       OutNumBytes := 0;
  33.       ErrNumBytes := 0;
  34.       while Proc.Active or (Proc.Output.NumBytesAvailable > 0) or (Assigned(Proc.Stderr) and (Proc.Stderr.NumBytesAvailable > 0)) do
  35.       begin
  36.         Available := Proc.Output.NumBytesAvailable;
  37.         while Available > 0 do // Only read if data from corresponding stream is available. Otherwise, read blocks on linux
  38.         begin
  39.           if Available > BUF_SIZE then
  40.           begin
  41.             Available := BUF_SIZE; // Read at most BUF_SIZE bytes at one time
  42.           end;
  43.           OutMemStream.Size := OutNumBytes + Available; // Allocate storage in advance
  44.           BytesRead := Proc.Output.Read((OutMemStream.Memory + OutNumBytes)^, Available); // Read data into stream storage
  45.           Assert(BytesRead = Available);
  46.           if BytesRead > 0 then
  47.           begin
  48.             Inc(OutNumBytes, BytesRead);
  49.           end;
  50.           Available := Proc.Output.NumBytesAvailable;
  51.         end;
  52.  
  53.         if Assigned(Proc.Stderr) then // Check for assigned(P.stderr) to prevent AV if poStderrToOutput in p.Options.
  54.         begin          
  55.           Available := Proc.Stderr.NumBytesAvailable;
  56.           while Available > 0 do
  57.           begin      
  58.             if Available > BUF_SIZE then
  59.             begin
  60.               Available := BUF_SIZE; // Read at most BUF_SIZE bytes at one time
  61.             end;
  62.             ErrMemStream.Size := ErrNumBytes + Available; // Allocate storage in advance
  63.             BytesRead := Proc.StdErr.Read((ErrMemStream.Memory + ErrNumBytes)^, Available); // Read data into stream storage
  64.             Assert(BytesRead = Available);
  65.             if BytesRead > 0 then
  66.             begin
  67.               Inc(ErrNumBytes, BytesRead);
  68.             end;    
  69.             Available := Proc.Stderr.NumBytesAvailable;
  70.           end;
  71.         end;
  72.       end;          
  73.  
  74.       OutMemStream.Size := OutNumBytes;
  75.       ErrMemStream.Size := ErrNumBytes;
  76.  
  77.       OutMemStream.Position := 0;
  78.       OutLines.LoadFromStream(OutMemStream);
  79.  
  80.       ErrMemStream.Position := 0;
  81.       ErrLines.LoadFromStream(ErrMemStream);
  82.  
  83.       Writeln('##### Out #####' + sLineBreak + OutLines.Text + sLineBreak + '##### Err #####' + sLineBreak + ErrLines.Text);
  84.     finally
  85.       OutMemStream.Free;
  86.       ErrMemStream.Free;
  87.       OutLines.Free;
  88.       ErrLines.Free;
  89.     end;
  90.   finally
  91.     Proc.Free;
  92.   end;
  93. end;  
  94.  

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus