* * *

Author Topic: [SOLVED] Bounty: Automate external process with prompts  (Read 11315 times)

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
[SOLVED] Bounty: Automate external process with prompts
« on: September 26, 2013, 10:37:18 pm »
I'm really interested in solving the general case of automating a command line application (cross-platform) that has some level of prompting/user interaction. I'm happy doing the parsing and command logic but the thing I've got really stuck with is interacting with a prompting command.

I've read http://wiki.freepascal.org/Executing_External_Programs
and run the example https://svn.code.sf.net/p/lazarus-ccr/svn/examples/process/
I've read some good threads here about the issues, including the comments that TProcess isn't a shell.

I written a basic shell app using TAsyncProcess (which works on linux) by using the /bin/sh trick like so:

Code: [Select]
 

  FProcess.Executable := '/bin/sh';
  FProcess.Parameters.Add('-i'); //makes the shell interactive
  FLastRunning := true;
  FProcess.Execute;
 
  //then doing the following to send a command
  FProcess.Input.Write(inputstr[1],Length(inputstr));     

  //and extracting the output for display
    if assigned(aproc.Output) then
      tempStrings.LoadFromStream(aproc.Output);
    Result := tempStrings.Text; 

But it's not good enough as it doesn't catch prompts due to TProcess using pipes.

What I really want is a class that can wrap a terminal emulator/command line interpreter so that I can detect when a prompt has occurred (by watching it and seeing if the text matches an expected prompt? or any other mechanism) and then send the response in code. The end result is that we could use Lazarus to write cross-platform GUI layers on top of command line tools such as ftp, maven or whatever which would be awesome.

I'll offer a bounty of US$400 / £250 for solution that:
- allows invocation of an arbitrary command line application
- catches all textual output (stdout and stderr)
- catches or somehow detects applications raising prompts
- allows sending a response to prompts and continuing execution
- doesn't create a visible cli/xterm console window
- works on linux, mac and windows platforms (32/64)

I think this is possible because when I'm using Lazarus on Linux/Mac to debug the IDE can catch the prompts my app is missing and display them in the "Terminal Output" window, and I can even type responses in there.
« Last Edit: December 08, 2016, 11:54:56 pm by macmike »

sam707

  • Sr. Member
  • ****
  • Posts: 432
Re: Bounty: Automate external process with prompts
« Reply #1 on: September 27, 2013, 12:46:11 am »
I don't understand well your problem , but i made some tests on linux and windows about terminals

- why can't you (1) read periodically the piped ToutputSream of the process and '''if you receive a line ending with '~$'+#$0D (on linux), ending with '>'+#$0D (on windows) then record no activity for 0.250 sec''' ... assume that the Process is in a standby mode waiting commands. it's a way to 'catch prompt', maybe

- send the next command to the piped TinputStream , and loop to the (1) til the end of your commands list

you could build and maintain this commands list in a stringlist with menus, checkboxes, edit fields, buttons, etc , inside a mainform, then launch the jobs following the list... a kind of cooked batch
« Last Edit: September 27, 2013, 01:00:56 am by sam707 »
CodeTyphon 5.8, FPC 3.1.1, Emulators = Radio Shack TRS-80, basic, Pascal 80 , Misosys C pro, Turbo Pascal from version 2 to 5 on amtrad cpc6128 CP/M also a bunch of C compilers. Pascal, C, C++ all versions windows/linux. also assembler from Zilog Z-80 to Pentium MMX

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
Re: Bounty: Automate external process with prompts
« Reply #2 on: September 27, 2013, 01:01:27 am »
Thanks for the response Sam707, unfortunately if you try that you'll see the problem. You can do as you say (which is almost the code I posted above) and you'll get the cmd line output but not the prompts and the input pipe to TProcess won't send commands to the prompt.

Here's the full code of my simple shell app that does what you suggest but doesn't meet my tests (e.g. try running "ftp" or similar with this kind of approach).

Code: [Select]
unit ufrmSimpleShell;

{$mode objfpc}{$H+}

interface

uses
  Classes, SysUtils, process, FileUtil, Forms, Controls, Graphics, Dialogs,
  StdCtrls, ExtCtrls, AsyncProcess, LCLType;

type

  { TfrmShell }

  TfrmShell = class(TForm)
    btnExec : TButton;
    edtInput : TEdit;
    FProcess : TAsyncProcess;
    mmLog : TMemo;
    procedure btnExecClick(Sender : TObject);
    procedure edtInputKeyUp(Sender : TObject; var Key : Word;
      Shift : TShiftState);
    procedure FormClose(Sender : TObject; var CloseAction : TCloseAction);
    procedure FormCreate(Sender : TObject);
    procedure FProcessReadData(Sender : TObject);
    procedure FProcessTerminate(Sender : TObject);
  private
    function ReadOutput(const aproc : TAsyncProcess) : string;
    procedure Log(const s : string);
    procedure Log(Const Fmt : String; const Args : Array of const);
  public
    { public declarations }
  end;

var
  frmShell : TfrmShell;

implementation

uses LCLProc;

{$R *.lfm}

{ TfrmShell }



procedure TfrmShell.FormCreate(Sender : TObject);
begin
  FProcess.Executable := '/bin/sh';
  FProcess.Parameters.Add('-i');
  FLastRunning := true;
  FProcess.Execute;
  FProcessReadData(Sender);
end;

procedure TfrmShell.btnExecClick(Sender : TObject);
var
  inputstr : TCaption;
begin
  inputstr := edtInput.Text;
  Log('Sending input: "%s"',[inputstr]);
  inputstr := inputstr + #10;
  FProcess.Input.Write(inputstr[1],Length(inputstr));
end;



procedure TfrmShell.edtInputKeyUp(Sender : TObject; var Key : Word;
  Shift : TShiftState);
begin
  if Key = VK_RETURN then
    btnExecClick(Sender);
end;



procedure TfrmShell.FormClose(Sender : TObject; var CloseAction : TCloseAction);
begin
  FProcess.Terminate(0);
end;

procedure TfrmShell.FProcessReadData(Sender : TObject);
var newOutput : string;
begin
  newOutput :=  ReadOutput(FProcess);
  if newOutput <> '' then
    Log(newOutput);
end;

procedure TfrmShell.FProcessTerminate(Sender : TObject);
begin
  Log('Process Terminate');
end;

function TfrmShell.ReadOutput(const aproc : TAsyncProcess) : string;
var tempStrings : TStringList;
begin
  tempStrings := TStringList.Create;
  try
    if assigned(aproc.Output) then
      tempStrings.LoadFromStream(aproc.Output);
    Result := tempStrings.Text;
  finally
    tempStrings.Free;
  end;
end;

procedure TfrmShell.Log(const s : string);
begin
  mmLog.Lines.Add(s);
end;

procedure TfrmShell.Log(const Fmt : String; const Args : array of const);
begin
  Log(Format(Fmt,Args));
end;

end.


This kind of thing can't even see the prompt lines ending % or > or whatever as they're not part of the output piped to the stream, they go straight to the outer console if there is one.
« Last Edit: September 27, 2013, 01:03:12 am by macmike »

sam707

  • Sr. Member
  • ****
  • Posts: 432
Re: Bounty: Automate external process with prompts
« Reply #3 on: September 27, 2013, 01:16:35 am »
Sir ,

may I suggest you to Free and Create a process for each command in ur list if possible.

cutting problems into little pieces often helps to light a solution and yes I tried inject a command line on a TProcess Input stream, it didn't work on my OSes grrr.

so I did what I mentioned above (cutting into lil pieces) at the moment , it works : fill the executable, parameters, command and THEN run commands one by one on a recreated TProcess descendant.

Sorry , that's the only way I found out until now. If your problem really can't be turned into multiples Freeed and recreated processes, if you need to "inject" commands into a terminal , then.... I don't know  :(
CodeTyphon 5.8, FPC 3.1.1, Emulators = Radio Shack TRS-80, basic, Pascal 80 , Misosys C pro, Turbo Pascal from version 2 to 5 on amtrad cpc6128 CP/M also a bunch of C compilers. Pascal, C, C++ all versions windows/linux. also assembler from Zilog Z-80 to Pentium MMX

sam707

  • Sr. Member
  • ****
  • Posts: 432
Re: Bounty: Automate external process with prompts
« Reply #4 on: September 27, 2013, 03:11:01 am »
found this thread around :

http://forum.lazarus.freepascal.org/index.php?topic=5986.0

as you can see, a TCmdBox component exists for Lazarus that attempts to "emulate" a terminal , ... and encounter almost same issues.

Not sure , but i think that "Injecting" orders in a console process from outside would probably have security issues (I am also thinking about keyboard hooking, and many antivirus would yell at ya) on many platforms nowadays. This might be practically not possible  %)

That is why , I'll keep the solution I gave before : 1 Process per command , and the results redirected to a only one display , mimicing but not emulating a terminal.

In the hope that my old programmer's brain helped.

Ty
« Last Edit: September 27, 2013, 03:28:11 am by sam707 »
CodeTyphon 5.8, FPC 3.1.1, Emulators = Radio Shack TRS-80, basic, Pascal 80 , Misosys C pro, Turbo Pascal from version 2 to 5 on amtrad cpc6128 CP/M also a bunch of C compilers. Pascal, C, C++ all versions windows/linux. also assembler from Zilog Z-80 to Pentium MMX

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
Re: Bounty: Automate external process with prompts
« Reply #5 on: September 27, 2013, 09:24:10 am »
Thanks again, I've played with the CmdBox component but it doesn't solve the problem it's just a different way of scraping the output (which is probably better for longer lasting processes).

I've tried creating/freeing per command in fact my first version executed commands as a static function on a class doing exactly that, the problem then is you've got no chance to respond to a prompt because the process wrapper either freezes because it is waiting for input (which you can't send even if you guess it's needed) or if you fire and forget it's been freed before the prompt happens and a new Tprocess is executing a new command not providing input to the previous.

avra

  • Hero Member
  • *****
  • Posts: 821
    • Additional info
Re: Bounty: Automate external process with prompts
« Reply #6 on: September 27, 2013, 10:28:24 am »
Shouldn't you be able to start a process, get it's PID, and then monitor if process with such a PID is still alive? If it's not - you have a command prompt waiting for input. That will do only if you don't run processes in a way that they return command prompt while they are still running. Processes that show some message and require user input can be handled by simply parsing their output and reacting accordingly.

If I remember well, Krusader and Double Commander have embedded terminal, so maybe looking at their code you might get some better idea...
« Last Edit: September 27, 2013, 10:34:02 am by avra »
ct2laz - Easily convert components and projects between Lazarus and CodeTyphon

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
Re: Bounty: Automate external process with prompts
« Reply #7 on: September 27, 2013, 07:44:56 pm »
The problem with that approach (or one of executing commands one at a time via TProcess) is that, as you say, it doesn't deal with processes that have user interaction on the console while they're still running. The idea of looking at Krusader and Double Commander is good, I'm not familiar with them but I'll have a look.

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
Re: Bounty: Automate external process with prompts
« Reply #8 on: September 28, 2013, 12:17:10 am »
Looks like both DoubleCommander and Krusader just invoke a shell (set by the user in options) along the lines of my first example although they don't even capture the output.

sam707

  • Sr. Member
  • ****
  • Posts: 432
Re: Bounty: Automate external process with prompts
« Reply #9 on: September 28, 2013, 12:19:49 am »
deal with processes that have user interaction on the console while they're still running.

= data injection = forbidden = NO

I told you
CodeTyphon 5.8, FPC 3.1.1, Emulators = Radio Shack TRS-80, basic, Pascal 80 , Misosys C pro, Turbo Pascal from version 2 to 5 on amtrad cpc6128 CP/M also a bunch of C compilers. Pascal, C, C++ all versions windows/linux. also assembler from Zilog Z-80 to Pentium MMX

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
Re: Bounty: Automate external process with prompts
« Reply #10 on: September 28, 2013, 12:23:45 am »
Then how does the Lazarus IDE do it?

sam707

  • Sr. Member
  • ****
  • Posts: 432
Re: Bounty: Automate external process with prompts
« Reply #11 on: September 28, 2013, 12:26:50 am »
nope !

where ? when ?

you mean the "messages" for compiler results and debugger ?

no "interaction" in them , just logs !
CodeTyphon 5.8, FPC 3.1.1, Emulators = Radio Shack TRS-80, basic, Pascal 80 , Misosys C pro, Turbo Pascal from version 2 to 5 on amtrad cpc6128 CP/M also a bunch of C compilers. Pascal, C, C++ all versions windows/linux. also assembler from Zilog Z-80 to Pentium MMX

macmike

  • Jr. Member
  • **
  • Posts: 83
    • Soft-Practice
Re: Bounty: Automate external process with prompts
« Reply #12 on: September 28, 2013, 12:29:30 am »
Not so, I mentioned it in my first post in this thread. I'm currently looking at the code in TPseudoConsoleDlg that does it.

Leledumbo

  • Hero Member
  • *****
  • Posts: 7487
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Bounty: Automate external process with prompts
« Reply #13 on: September 28, 2013, 12:33:40 am »
Try my damn old OctaveGUI. I didn't wrap any shell, just execute the required process directly. I don't really remember the code, probably there was Windows specific code/treatment, but you can start with that.

sam707

  • Sr. Member
  • ****
  • Posts: 432
Re: Bounty: Automate external process with prompts
« Reply #14 on: September 28, 2013, 12:39:47 am »
https://github.com/alrieckert/lazarus/blob/master/debugger/pseudoterminaldlg.pp

the mentioned TPseudoTerminalDlg uses debugwindows, (c.f. my above messages = keyboard hooking = implies to install a debugger , also implies hard time to make it totally cross-platform)

believe me , lol , threads safe OSes are built to BE threads safe and Processes safe, if you "inject data" by a back door it is

1) considered Hacking
2) not standard
3) unportable
4) breaking thru the security and making happy holes and open doors to pirats
« Last Edit: September 28, 2013, 12:44:35 am by sam707 »
CodeTyphon 5.8, FPC 3.1.1, Emulators = Radio Shack TRS-80, basic, Pascal 80 , Misosys C pro, Turbo Pascal from version 2 to 5 on amtrad cpc6128 CP/M also a bunch of C compilers. Pascal, C, C++ all versions windows/linux. also assembler from Zilog Z-80 to Pentium MMX

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus