Lazarus

Programming => General => Topic started by: TCH on January 03, 2018, 11:48:10 pm

Title: TProcess.Output is permanently empty
Post by: TCH on January 03, 2018, 11:48:10 pm
I have a small C program which writes a 8-10 byte sized line to the stdout at roughly every 150-200 millisecond.

I have a Lazarus program what runs this program as a TProcess and tries to process it's output, but the Output pipe stream is always empty.

I create the process:
Code: Pascal  [Select][+][-]
  1. proc := TProcess.Create(nil);
  2. proc.Options := [poUsePipes];
  3. proc.CommandLine := '/usr/local/bin/cprog arg1 arg2 ';
  4. proctimer.Enabled := True;
And there is a timer which fires every 100 millisecond:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.proctimerTimer(Sender: TObject);
  2. var
  3.         i, bc: integer;
  4.         buffer: array[0..15] of char;
  5. begin
  6.         bc := proc.Output.NumBytesAvailable;
  7.         proc.Output.read(buffer[0], bc);
  8.         i := 0;
  9.         while (i < bc) do
  10.         begin
  11.                 // process the incoming data
  12.         end;
  13. end;
And the variable bc is always zero, the Output pipe is always empty.

What's the problem with this code?
Title: Re: TProcess.Output is permanently empty
Post by: Jurassic Pork on January 04, 2018, 01:48:11 am
hello,
have you forgotten the proc.Execute ?  :-X

And not sure that you can use a Timer.
Other solution use a loop like this for example :
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button9Click(Sender: TObject);
  2. var Proc: TProcess;
  3.            CharBuffer: array [0..511] of char;
  4.           ReadCount: integer;
  5. begin
  6. try
  7. Proc := TProcess.Create(nil);
  8. Proc.Options := [poUsePipes];
  9. Proc.CommandLine := '/bin/ping -c 5 127.0.0.1';
  10. Proc.Execute();
  11. while (Proc.Running) or (Proc.Output.NumBytesAvailable > 0) or
  12.        (Proc.Stderr.NumBytesAvailable > 0) do
  13.      begin
  14.        // read stdout and write to our stdout
  15.        while Proc.Output.NumBytesAvailable > 0 do
  16.        begin
  17.          ReadCount := Min(512, Proc.Output.NumBytesAvailable); //Read up to buffer, not more
  18.          Proc.Output.Read(CharBuffer, ReadCount);
  19.          Memo1.Lines.Append(Copy(CharBuffer, 0, ReadCount));
  20.        end;
  21.        // read stderr and write to our stderr
  22.        while Proc.Stderr.NumBytesAvailable > 0 do
  23.        begin
  24.          ReadCount := Min(512, Proc.Stderr.NumBytesAvailable); //Read up to buffer, not more
  25.          Proc.Stderr.Read(CharBuffer, ReadCount);
  26.           Memo1.Lines.Append(Copy(CharBuffer, 0, ReadCount));
  27.        end;
  28.         application.ProcessMessages;
  29.         sleep(200);
  30.      end;
  31.      ExitCode := Proc.ExitStatus;
  32.    finally
  33.      Proc.Free;
  34.    end;
  35. end;

Friendly, J.P
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 05:57:32 pm
have you forgotten the proc.Execute ?  :-X

Only in my example code, not in the real one of course.

Code: Pascal  [Select][+][-]
  1.         application.ProcessMessages;
  2.         sleep(200);
This is practically what a TTimer does: do its job, let the GUI do its, then waits for some time.
And reading from the Output is working with TTimer-s, i already did it before; for example this program does it (http://oscomp.hu/?details/SIDPlay2_Frontend_1.1.1_for_Linux_x86_64_Linux_1460) and works.

I think, the C program outputs its output too slow; i did another program, which just omits garbage to the console with much more speed and the Pascal code suddenly begun to work. But the one which only prints out data after 150-200 ms will give an empty Output buffer.
Title: Re: TProcess.Output is permanently empty
Post by: Thaddy on January 04, 2018, 06:05:19 pm
Don't use the timer. that's silly. Use TAsyncProcess if you are not satisfied, The timer will get you into trouble at one time. Timers are for timing, not to solve sync problems. That's bad programming. Delay and Sleep are too...
Sync doesn't rely on timing: it is unpredictable by nature.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 06:22:06 pm
I don't care how i can use, but i need to run this program along with mine because it's outputs are determining what my program should do.
If i cannot use TTimer and sleep / delay then what should i do? A TThread would also need some timing, or else it would consume all CPU time.
Title: Re: TProcess.Output is permanently empty
Post by: Thaddy on January 04, 2018, 06:23:46 pm
Use TAsynchProcess. And - of course - a timeout value, but chosen so high it should never occur, like 1 whole second.

Just curious: can you solve this in C.... :D :D :D :D Does the C code have a timer?  8-) >:D :P
Well then,write it in Pascal syntax.....

You may need some lessons in blocking vs non-blocking code but then file this under beginners.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 06:33:51 pm
I replaced the declarations to TAsynchProcess nothing has changed.

Solve what in C? I did not wrote the C program, i just have to use it.
Title: Re: TProcess.Output is permanently empty
Post by: Thaddy on January 04, 2018, 06:39:37 pm
read my last remark - that crossed your answer -.
Forget about in what language the program you are communicating with is written in. That does not matter at all. You are trying to get reliable communication between yours and some other program. Right?
If I am right I will help you further. If the program or process written in C or whatever behaves well and writes to pipes or stdout we can fix this (without a timer)
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 06:48:32 pm
Okay, so what should i do? I just rewrote it to use a TThread with sleep instead of TTimer and it still does not work.
Title: Re: TProcess.Output is permanently empty
Post by: Thaddy on January 04, 2018, 06:54:02 pm
Then we/I need some code first.. (Don't worry,I am outspoken but harmless)
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 07:07:34 pm
The example code is in the opening post. I cannot give you the real code, since it's company code and it's not mine.

The basic idea is just this: An external program dumps out a line with some chars every 150-200 ms to the stdout. I already have a Pascal program which does a lot of thing, but among them, i need to track these written lines. The starting/stopping/etc. of the external program works just fine, but the Output buffer is always empty.
Title: Re: TProcess.Output is permanently empty
Post by: Thaddy on January 04, 2018, 07:09:33 pm
Ok, will have a look. Have to eat first.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 07:14:49 pm
Приятного аппетита!
Title: Re: TProcess.Output is permanently empty
Post by: Leledumbo on January 04, 2018, 10:44:48 pm
As you have abstracted the actual program and parameters, and not willing to give small, compilable and runnable program that demonstrates the behavior, I cannot really know what is wrong so this is just a wild guess.

CommandLine is deprecated, use Executable and Parameters to form the command to execute. Improper quoting is enough to ruin everything.
Are you sure that the process is even executing? Perhaps it stops already? Check Running and ExitStatus property.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 10:56:55 pm
The C program is closed, so i cannot even give the source of it. As for the Pascal, as i said, it's not mine, i cannot distribute the sources.

The external program is running fine, it communicates via serial port and i see the LED blinking. If the programs tops, the LED stops blinking. And it does not matter if i use CommandLine or the new methods, i've checked the parameters and all is fine...
Title: Re: TProcess.Output is permanently empty
Post by: Edson on January 04, 2018, 11:03:18 pm
Maybe you can use my library: https://github.com/t-edson/UnTerminal

It uses events to indicate when there is output from a process.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 04, 2018, 11:22:58 pm
I see the license is LGPL. Can we use it in a closed source commercial product? If yes, then i'll try it on tomorrow, or on the weekend.
Title: Re: TProcess.Output is permanently empty
Post by: Leledumbo on January 05, 2018, 01:23:22 am
The C program is closed, so i cannot even give the source of it. As for the Pascal, as i said, it's not mine, i cannot distribute the sources.

The external program is running fine, it communicates via serial port and i see the LED blinking. If the programs tops, the LED stops blinking. And it does not matter if i use CommandLine or the new methods, i've checked the parameters and all is fine...
You don't have to give the same source code, create yourself one that demonstrates the behavior, as simple as possible.

In that case, the program might really be running, but buffers its output. Again this is just a guess. One more thing, it could actually be writing to stderr instead of stdout, so you might want to add poStderrToOutPut to Options property.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 05, 2018, 10:24:13 am
You don't have to give the same source code, create yourself one that demonstrates the behavior, as simple as possible.
I already did. It's in the opening post.
In that case, the program might really be running, but buffers its output. Again this is just a guess. One more thing, it could actually be writing to stderr instead of stdout, so you might want to add poStderrToOutPut to Options property.
No, it's writing to stdout. I tried poStderrToOutPut, but no avail. I tried to redirect the program's output to a file from console and 1> did the redirect, 2> did not. What do you mean by "it buffers its output"? It's just do a lot of printf.
Title: Re: TProcess.Output is permanently empty
Post by: marcov on January 05, 2018, 11:12:43 am
You don't have to give the same source code, create yourself one that demonstrates the behavior, as simple as possible.
I already did. It's in the opening post.

No, that is only a fragment, and as already mentioned misses execute or anything else that activates the tprocess instance.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 05, 2018, 11:20:39 am
But it contains the essential part and as i already told, i only missed the Execute from this example code; it is in the original.
Title: Re: TProcess.Output is permanently empty
Post by: Leledumbo on January 06, 2018, 01:32:47 am
But it contains the essential part and as i already told, i only missed the Execute from this example code; it is in the original.
Uncompilable, therefore cannot be used as a base to reproduce the behavior. You first create one that all of us can compile, using your real C executable where it demonstrates the behavior that you said, where the output stream is always empty. You then rename the C executable (in the Pascal code) to whatever you wish, keeping the rest of the code intact. Then post or attach here.

Read this stackoverflow post (https://stackoverflow.com/help/mcve) for better explanation. In short, help us help you.
Title: Re: TProcess.Output is permanently empty
Post by: engkin on January 06, 2018, 02:42:19 am
@TCH
What OS/CPU and Laz/FPC version?

Does the example in this post (http://forum.lazarus.freepascal.org/index.php/topic,39535.msg271659.html#msg271659) work for you?
If yes, replace the CommandLine with yours. Does it work?

Your first post shows that the C prog takes two arguments. What type of arguments? Numbers?
Title: Re: TProcess.Output is permanently empty
Post by: Jurassic Pork on January 06, 2018, 06:19:07 am
hello,
i have tried my example with the option -i to 200 ms for the ping ( -i = Wait interval seconds between sending each packet ) :
Code: Pascal  [Select][+][-]
  1. Proc := TProcess.Create(nil);
  2. Proc.Options := [poUsePipes];
  3. Proc.Executable := '/bin/ping';
  4. Proc.Parameters.Add('-c 100');
  5. Proc.Parameters.Add('-i 0.2');
  6. Proc.Parameters.Add('127.0.0.1');
  7. Proc.Execute();

it still works with Output every 200 milliseconds on Ubuntu 16.04 64 bits Lazarus 1.8 !

Friendly, J.P
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 12, 2018, 03:13:07 pm
Uncompilable, therefore cannot be used as a base to reproduce the behavior. You first create one that all of us can compile, using your real C executable where it demonstrates the behavior that you said, where the output stream is always empty. You then rename the C executable (in the Pascal code) to whatever you wish, keeping the rest of the code intact. Then post or attach here.
Okay, here is the real one:
Code: [Select]
constructor TEMP860Thread.Create(CreateSuspended : boolean);
begin
  inherited Create(CreateSuspended);
  Self.Stop := False;
end;

procedure TEMP860Thread.ErrorSet;
begin
     set_error_bits(ERROR_CASHHANDLER_DIED);
end;

procedure TEMP860Thread.ErrorClear;
begin
     set_error_bits(ERROR_CASHHANDLER_DIED);
end;

procedure TEMP860Thread.CoinDrop;
begin
     add_coin(Self.Money);
end;

procedure TEMP860Thread.FinishTransaction;
begin
     finish_transaction;
end;

procedure TEMP860Thread.ClearTransaction;
begin
     clear_transaction;
end;

procedure TEMP860Thread.Execute;
var
    bc, i, a, code, tries: longint;
    cs: array[0..3] of string;
begin
     while (not Self.Stop) do
     begin
{$IFDEF CPUARMHF}
         if (not emp860.Running) then
         begin
              if (emp860_fails < 16) then
              begin
                   emp860.Execute;
                   writeln('EMP860 executed.');
                   inc(emp860_fails);
              end
              else
      begin
         Synchronize(@ErrorSet);
              end;
         end
         else
         begin
              writeln('READ');
              if (emp860_fails > 15) then
              begin
         Synchronize(@ErrorClear);
              end;
              emp860_fails := 0;
              bc := 0;
              tries := 0;
              while ((bc = 0) and (tries < 20)) do
              begin
                   sleep(10);
                 bc := emp860.Output.NumBytesAvailable;
                   inc(tries);
              end;
              writeln('bc = ' + inttostr(bc));
              if (bc = 0) then
              begin
                   inc(emp860_nosignal);
                   if (emp860_nosignal > 15) then
                   begin
                        emp860_nosignal := 0;
                emp860.Terminate(0);
                        writeln('EMP860 terminated.');
                   end;
              end
              else
              begin
                   emp860.Output.Read(pipe_buffer[0], bc);
                   emp860_nosignal := 0;
     i := 0;
                   a := 0;
                   cs[0] := '';
                   cs[1] := '';
                   cs[2] := '';
                   cs[3] := '';
                   while (i < bc) do
                   begin
                         if ((pipe_buffer[i] <> ';') and (pipe_buffer[i] <> #10)) then
                         begin
                            cs[a] := cs[a] + pipe_buffer[i];
                         end
                         else
                         begin
                              if (pipe_buffer[i] = ';') then
                              begin
                                   inc(a);
                              end
                              else
                              begin
                                   code := strtoint(cs[0]);
                                   if (code = 10) then
                                   begin
                Self.Money := strtoint(cs[2]);
                                        Synchronize(@CoinDrop);
                                   end;
                                   if (code = 12) then
                                   begin
                                        Synchronize(@FinishTransaction);
                                   end;
                                   if (code = 13) then
                                   begin
                                        Synchronize(@ClearTransaction);
                                   end;
                                   a := 0;
                                   cs[0] := '';
                                   cs[1] := '';
                                   cs[2] := '';
                                   cs[3] := '';
                              end;
                         end;
                         inc(i);
                   end;
              end;
         end;
{$ENDIF}
sleep(100);
WriteLn('EMP860 TICK');
     end;
end;
and in the startup:
Code: [Select]
{$IFDEF CPUARMHF}
     WriteLn('EMP860 process initialization...');
     emp860 := TProcess.Create(nil);
     emp860.Options := [poUsePipes, poStderrToOutPut];
     emp860.CommandLine := '/usr/local/bin/emp860 /dev/ttyUSB0 ' + generate_bitmask(AcceptableCoins, 6);
     emp860_fails := 0;
     emp860_nosignal := 0;
{$ENDIF}
     WriteLn('Creating EMP860 thread...');
     emp860thread := TEMP860Thread.Create(false);
And the declaration:
Code: [Select]
  TEMP860Thread = class(TThread)
    protected
      procedure Execute; override;
          procedure ErrorSet;
          procedure ErrorClear;
          procedure CoinDrop;
          procedure FinishTransaction;
          procedure ClearTransaction;
    public
          var Stop: Boolean;
    var Money: Word;
      Constructor Create(CreateSuspended : boolean);
  end;
@TCH
What OS/CPU and Laz/FPC version?

Does the example in this post (http://forum.lazarus.freepascal.org/index.php/topic,39535.msg271659.html#msg271659) work for you?
If yes, replace the CommandLine with yours. Does it work?

Your first post shows that the C prog takes two arguments. What type of arguments? Numbers?
Linux on ARMHF, Lazarus is 1.6.4 and FPC is 3.0.2.

The linked example did not worked for me.
hello,
i have tried my example with the option -i to 200 ms for the ping ( -i = Wait interval seconds between sending each packet ) :
Code: Pascal  [Select][+][-]
  1. Proc := TProcess.Create(nil);
  2. Proc.Options := [poUsePipes];
  3. Proc.Executable := '/bin/ping';
  4. Proc.Parameters.Add('-c 100');
  5. Proc.Parameters.Add('-i 0.2');
  6. Proc.Parameters.Add('127.0.0.1');
  7. Proc.Execute();

it still works with Output every 200 milliseconds on Ubuntu 16.04 64 bits Lazarus 1.8 !

Friendly, J.P
Nope, did not work for me.
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 13, 2018, 04:44:00 pm
Solved in a different way. I got the C++ sources of the external program, took the needed parts out and integrated the object file into my Lazarus program.
Title: Re: TProcess.Output is permanently empty
Post by: Thaddy on January 13, 2018, 04:58:35 pm
If you needed to do that, the C++ written program does not use any known standard regarding IO. Because those are pretty well defined. Let me guess: direct access.... by-passing cout...
Title: Re: TProcess.Output is permanently empty
Post by: marcov on January 13, 2018, 07:05:58 pm
Code: Pascal  [Select][+][-]
  1. Proc := TProcess.Create(nil);
  2. Proc.Options := [poUsePipes];
  3. Proc.Executable := '/bin/ping';
  4. Proc.Parameters.Add('-c 100');
  5. Proc.Parameters.Add('-i 0.2');
  6. Proc.Parameters.Add('127.0.0.1');
  7. Proc.Execute();

This is probably wrong. You need to add e.g. -c and 100 as separate parameters. It matters most on *nix, less on Windows.
 
Title: Re: TProcess.Output is permanently empty
Post by: TCH on January 14, 2018, 01:45:53 pm
If you needed to do that, the C++ written program does not use any known standard regarding IO. Because those are pretty well defined. Let me guess: direct access.... by-passing cout...
The C++ program used printf, but now i omitted the console writing, i changed the main function to another which has two additional parameters besides the original (i mean original expected, not the argc/argv, but what arrives in them): one is responsible for allowing the function to continue and the other is a procedure's address. So practically i run the C++ function from Pascal, give them an address of a boolean which i use for halting the cycle and a Pascal procedure's address which handles the arriving result.
TinyPortal © 2005-2018