"RunCommand extensions" - broken Wiki documentation?

[colo]

Hi,

I'm trying (and failing) to solve an important use-case when wrapping a CLI command. On this Windows host using Lazarus 3.2.2, I would like to

• capture the output and error stream of the program while it executes
• update the GUI with progress information about the inferior process while it's running

Researching various options, I have come to believe that

Code: [Select]

TProcess.RunCommandLoop is the "proper" way to do this in state-of-the-art Lazarus/FPC, and I would very much like to use it to implement my solution, but I just... utterly fail to pull it off.

(That is to say, I've had a number of *partial* solutions working where I could get StdOut and StdErr into TStringlists after the process terminates and have the GUI display a spinner while waiting for the spawned process to finish, but once the text produced on either fd exceeds a certain limit (I'm guessing Windows has one for pipes, too), everything just hangs indefinitely, as expected. Yeah, I get it that this is discussed in the Wiki article about Executing Processes. HOWEVER...)

The wiki has an abbreviated example available at https://wiki.freepascal.org/Executing_External_Programs#RunCommand_extensions - but when I try to wrap that into a complete and working reproduction of this supposed example, the compiler barfs at me with

Code: [Select]

unit1.pas(58,26) Error: Wrong number of parameters specified for call to "LocalnIdleSleep"
unit1.pas(35,27) Error: Found declaration: LocalnIdleSleep(TObject;TObject;TRunCommandEventCode;const AnsiString);

I've never created a class of my own in FPC/Lazarus before, so I might be missing some context, but... is this example just plain incorrect or at least incomplete?


I would also like to express my ample frustration with this particular part/page of the Wiki documentation - if I had the knowledge, I would happily and readily rewrite it, but I don't, hence I can't. What is wrong with it, in my humble opinion, is there there is gobs and gobs and GOBS of paragraphs that deal with this (I think) relatively common use-case, but nowhere does it state clearly what is the sanest way to do it TODAY, instead citing numerous ways for readers who are interested in the historic recounting of Pascal-esque execve() variants and "this is a way to do it, but it's wrong, and breaks if there is too much output or the moon is in the 7th house"-non-solutions. I think it is in desperate need of expert attention. I would happily donate EUR 50,- to a charity or Lazarus-affiliated organization of the author's choosing if it were to be weeded out and made clear, concise, sufficiently complete and wholly correct.

[PascalDragon]

The wiki has an abbreviated example available at https://wiki.freepascal.org/Executing_External_Programs#RunCommand_extensions - but when I try to wrap that into a complete and working reproduction of this supposed example, the compiler barfs at me with

Code: [Select]

unit1.pas(58,26) Error: Wrong number of parameters specified for call to "LocalnIdleSleep"
unit1.pas(35,27) Error: Found declaration: LocalnIdleSleep(TObject;TObject;TRunCommandEventCode;const AnsiString);

I've never created a class of my own in FPC/Lazarus before, so I might be missing some context, but... is this example just plain incorrect or at least incomplete?

Just guessing: the example is written for mode Delphi (look at the top of the example), however your code uses mode ObjFPC and thus you need to use p.OnRunCommandEvent:=@p.LocalnIdleSleep; (note the @).

I would also like to express my ample frustration with this particular part/page of the Wiki documentation - if I had the knowledge, I would happily and readily rewrite it, but I don't, hence I can't. What is wrong with it, in my humble opinion, is there there is gobs and gobs and GOBS of paragraphs that deal with this (I think) relatively common use-case, but nowhere does it state clearly what is the sanest way to do it TODAY, instead citing numerous ways for readers who are interested in the historic recounting of Pascal-esque execve() variants and "this is a way to do it, but it's wrong, and breaks if there is too much output or the moon is in the 7th house"-non-solutions. I think it is in desperate need of expert attention. I would happily donate EUR 50,- to a charity or Lazarus-affiliated organization of the author's choosing if it were to be weeded out and made clear, concise, sufficiently complete and wholly correct.

All the examples are still valid today and each has their use case, especially if one has special needs that aren't covered by TProcess or its convenience wrappers.

[marcov]

A demo that is maybe more in the direction of your problem is in http://www.stack.nl/~marcov/files/processmemodemo.zip  that try to instrument another program and put the output in TMemo.

[colo]

@PascalDragon: Thanks very much for turning my attention at that obviosuly very important detail that I seem to have missed when migrating the example code into a Lazarus project created for the purpose. I guess it will turn out that you are just right, but I can't stomach going back to the other machine and fire up the IDE once more today...

In reply to your second statement - yeah, that very well may be so, but the way the content is organized and presented, to a reader with only very cursory familiarity with Pascal syntax and practises, turns it into a minefield of (mis)information and a source of terrible confusion.

Given the fact that that Wiki page - and this small fragment of information here: https://www.freepascal.org/docs-html/current/fcl/process/tprocess.runcommandloop.html - is the ONLY documentation and example for "RunCommandLoop" usage on the Web as indexed by Google and Bing to date, I hereby renew my plea to any subject matter expert with an inclination to improving documentation to please do so.


@marcov: Thanks, I will check that out as soon as my eyes and grey matter feel up to the task!

[KodeZwerg]

indexed by Google and Bing to date

Let me open your mind for other search engine that has a lot of hits
https://duckduckgo.com/?q=lazarus+RunCommandLoop

[colo]

Let me open your mind for other search engine that has a lot of hits
https://duckduckgo.com/?q=lazarus+RunCommandLoop

The less fuzzy search term https://duckduckgo.com/?q=lazarus+%22RunCommandLoop%22&ia=web yields 11 hits here, with only one being a a concrete example - and that is the Wiki page I linked earlier.

(DDG is actually my main search engine, and it shares a lot with Bing in terms of indexed content.)


Somewhat ironically (and also unfortunately), https://forum.lazarus.freepascal.org/index.php?topic=59975.0 is not in that particular search result, which seems like a very good and practical example. But that does not change the fact that the Wiki page mentioned is a mess that needs attention.

[marcov]

In the beginning there only was TProcess. But TProcess was unwieldy to use, so several people like me, Reinier Olislagers  (original author FPCDELUXE) and Ludo Brands all had their TProcess wrappers. All of them were based or a continuation of the large output scheme of the wiki.

However , in practice we constantly saw people only wanting to "simply" execute a binary and capture the output using the basic example in the wiki, the simple loadstream method , though it was known that wouldn't scale, and people would face hard to diagnose issues at some point

So I tried to integrate the various usages and make a final version of that in wrappers called Runcommand for basic usage.

Fast forward two major versions, and people complained that this was a bit "my way or the highway". Either you use  "runcommand", or got thrown back to the raw TProcess interface, and do everything yourself

So that's how runcommandloop emerged (3.2.0?). Basically the loop from runcommand made public and  further parametrized. So that people could reuse it rather than do anything (usually wrong) from scratch.

As far as documentation and examples goes. As something that is very pluggable, it is hard to abstractly document, and relies on examples. Probably I'll put these examples on my gitlab account in time, since FPC doesn't have a spot for Lazarus using examples.

[colo]

Thanks very much for the elaboration, and also the implementation of runcommand et al.

With the additional example code discovered on the forums, and a freshened mind, I managed to implement a wrapper class and associated machinery of my own that works as expected/required. In hindsight, I do find the interface provided very practical, but it's hard to wrap one's head around several new things at once (as a Pascal newcomer) with so much additional information around the subject floating about.

[marcov]

If you have any observations or examples please report them back.

I specially would appreciate reports of non-trivial use on *nix. I myself only use the most basic runcommand on it (to get git logs), the rest of my use is under Windows.

If you have problems on Windows executing binaries that spawn shells or something similar, try to use popassinput (it doesn't duplicate the handles), the only example I know now is powertoy's psexec.

[colo]

So I've been trying to produce a rather minimal application for educational purposes (primarily to the benefit of myself but I will readily share it once it's working as I need it to) that deals with executing another process under GNU/Linux, while handling both STDOUT and STDERR pipes/fds, and also triggering a callback procedure when an event is handled.

I haven't got it right yet, and I need to invest a fair number of additional brainwaves into understanding how the overridden TProcess-derived ReadInputStream and the ProcessEvent procedure are supposed to interact... but what I've been wondering the most is this:
With the current implementation of RunCommandLoop, (how) can the OnRunCommandEvent callback determine if it was data flowing in via either STDOUT or STDERR that caused the callback to be run?

[marcov]

The main reason for the override of ReadInputStream is that the lengths of the strings are not fields, IOW it just tries to intercept the variables in order to store/manipulate them.

The necessity to do that could be considered a flaw.

[colo]

So I find myself with some time again to look into this today, and afaict, the only events my project/code is ever able to catch are RunCommandIdle and RunCommandFinished. I don't see why, and where the material differences between my clumsy attempt and your processmemodemo project lie. Can you explain to me what I am doing wrong?

I was initially suspecting stdio buffering to play a role, but ample experimentation in that direction (with using unbuffered fds and having lots more output to exceed glibc buffer capacity) yielded nothing


Also, I am not sure what the difference between RunCommandReadOutputString and RunCommandReadOutputStream are, and when which one is supposed to fire. Some elaboration on that would me very much appreciated - even more so if I were to be taught to fish, instead of the fish simply being given to me Where/how can I learn how this works internally?


(Caveat: The two .sh scripts in the attached and severly incomplete project need to have the executable bit set (`chmod +x *.sh`) after the ZIP archive has been extracted for this example of a failure to even slightly work.)

[marcov]

So I find myself with some time again to look into this today, and afaict, the only events my project/code is ever able to catch are RunCommandIdle and RunCommandFinished. I don't see why, and where the material differences between my clumsy attempt and your processmemodemo project lie. Can you explain to me what I am doing wrong?

I'm on a windows laptop. So the .sh won't do much anyway, and can't quickly test. Also I'm generally using trunk, and not 3.2.2/fixes.

Yes, those events are limited, which why readinputstream must be overriden. From what I can see you use both stderr and stdout without using postderrtostdout.

But you treat all input in readinputstream as stdout, like an own postderrtostdout.

Probably you have to compare the input stream to tprocess.output and tprocess.stderr and then go into a loop for either stdout or stderr string buffer.

I was initially suspecting stdio buffering to play a role, but ample experimentation in that direction (with using unbuffered fds and having lots more output to exceed glibc buffer capacity) yielded nothing

Also, I am not sure what the difference between RunCommandReadOutputString and RunCommandReadOutputStream are, and when which one is supposed to fire.

TProcess originally only worked with streams, and Runcommand was meant to simplify that capturing the output as a string. Later Readinputstream was factored out, and because this functionality is also usable without runcommandloop, I added both variants to suit all purposes.

Runcommandloop however only uses the string variant.

Some elaboration on that would me very much appreciated - even more so if I were to be taught to fish, instead of the fish simply being given to me Where/how can I learn how this works internally?

Single step and debug ? How else do you think I do it ? :-)

(Caveat: The two .sh scripts in the attached and severly incomplete project need to have the executable bit set (`chmod +x *.sh`) after the ZIP archive has been extracted for this example of a failure to even slightly work.)

I'm on a windows laptop. So the .sh won't do much anyway, at least not in combo with GUI apps (I can SSH into a *nix machine) and can't quickly test. Maybe when I get back home I can fire up some machine.

[marcov]

Ok, I got home and the computer parts I ordered arrived, so my Linux box is working again.

For me running .sh's didn't seem to work until I added

Code: Pascal  [Select][+][-]

1. Interactiveprocess.executable:='/bin/sh';
2. Interactiveprocess.arguments.add('-c');
3. Interactiveprocess.arguments.add('./scriptname.sh');
4. // and then .add the other parameters.

My hunch about separating stdout and stderr worked, I

• Defined two new methods readinputstdout and readinputstrerr
• Both get a copy of the existing readinputstream code
• The stderr variant gets modified to use "stderr" instead of "stdout"variables
• readinpustream becomes something like

Code: Pascal  [Select][+][-]

1. begin
2.    if ProcPipe=output then
3.      readinputstdout (arguments.....)
4. else
5.      readinputstderr (arguments.....)
6. end
7.  

And I also added porunidle to the process options so that porunidle gets run.

However the application is not responsive, and I don't know why. Some LCL/GTK guru must comment on that.


It might have something to do with the not draw without paint event problem.

[colo]

Thanks very much for your input! I've gotten a bit further on my own differentiating between stdout and stderr streams by comparing ProcPipe.Handle to output.Handle and stderr.Handle, but your explanation makes everything a bit more clearer yet. I will see if I can work out the GUI responsiveness problem somehow, and report back with any results


The `/bin/sh`-requiring problem COULD be related to the unzipped shellscript not having the eXecutable bit set.