Lazarus
Programming => General => Topic started by: macmike on September 26, 2013, 10:37:18 pm
-
I'm really interested in solving the general case of automating a command line application (cross-platform) that has some level of prompting/user interaction. I'm happy doing the parsing and command logic but the thing I've got really stuck with is interacting with a prompting command.
I've read http://wiki.freepascal.org/Executing_External_Programs (http://wiki.freepascal.org/Executing_External_Programs)
and run the example https://svn.code.sf.net/p/lazarus-ccr/svn/examples/process/ (https://svn.code.sf.net/p/lazarus-ccr/svn/examples/process/)
I've read some good threads here about the issues, including the comments that TProcess isn't a shell.
I written a basic shell app using TAsyncProcess (which works on linux) by using the /bin/sh trick like so:
FProcess.Executable := '/bin/sh';
FProcess.Parameters.Add('-i'); //makes the shell interactive
FLastRunning := true;
FProcess.Execute;
//then doing the following to send a command
FProcess.Input.Write(inputstr[1],Length(inputstr));
//and extracting the output for display
if assigned(aproc.Output) then
tempStrings.LoadFromStream(aproc.Output);
Result := tempStrings.Text;
But it's not good enough as it doesn't catch prompts due to TProcess using pipes.
What I really want is a class that can wrap a terminal emulator/command line interpreter so that I can detect when a prompt has occurred (by watching it and seeing if the text matches an expected prompt? or any other mechanism) and then send the response in code. The end result is that we could use Lazarus to write cross-platform GUI layers on top of command line tools such as ftp, maven or whatever which would be awesome.
I'll offer a bounty (http://wiki.lazarus.freepascal.org/Bounties#Multi-platform_bounties) of US$400 / £250 for solution that:
- allows invocation of an arbitrary command line application
- catches all textual output (stdout and stderr)
- catches or somehow detects applications raising prompts
- allows sending a response to prompts and continuing execution
- doesn't create a visible cli/xterm console window
- works on linux, mac and windows platforms (32/64)
I think this is possible because when I'm using Lazarus on Linux/Mac to debug the IDE can catch the prompts my app is missing and display them in the "Terminal Output" window, and I can even type responses in there.
-
I don't understand well your problem , but i made some tests on linux and windows about terminals
- why can't you (1) read periodically the piped ToutputSream of the process and '''if you receive a line ending with '~$'+#$0D (on linux), ending with '>'+#$0D (on windows) then record no activity for 0.250 sec''' ... assume that the Process is in a standby mode waiting commands. it's a way to 'catch prompt', maybe
- send the next command to the piped TinputStream , and loop to the (1) til the end of your commands list
you could build and maintain this commands list in a stringlist with menus, checkboxes, edit fields, buttons, etc , inside a mainform, then launch the jobs following the list... a kind of cooked batch
-
Thanks for the response Sam707, unfortunately if you try that you'll see the problem. You can do as you say (which is almost the code I posted above) and you'll get the cmd line output but not the prompts and the input pipe to TProcess won't send commands to the prompt.
Here's the full code of my simple shell app that does what you suggest but doesn't meet my tests (e.g. try running "ftp" or similar with this kind of approach).
unit ufrmSimpleShell;
{$mode objfpc}{$H+}
interface
uses
Classes, SysUtils, process, FileUtil, Forms, Controls, Graphics, Dialogs,
StdCtrls, ExtCtrls, AsyncProcess, LCLType;
type
{ TfrmShell }
TfrmShell = class(TForm)
btnExec : TButton;
edtInput : TEdit;
FProcess : TAsyncProcess;
mmLog : TMemo;
procedure btnExecClick(Sender : TObject);
procedure edtInputKeyUp(Sender : TObject; var Key : Word;
Shift : TShiftState);
procedure FormClose(Sender : TObject; var CloseAction : TCloseAction);
procedure FormCreate(Sender : TObject);
procedure FProcessReadData(Sender : TObject);
procedure FProcessTerminate(Sender : TObject);
private
function ReadOutput(const aproc : TAsyncProcess) : string;
procedure Log(const s : string);
procedure Log(Const Fmt : String; const Args : Array of const);
public
{ public declarations }
end;
var
frmShell : TfrmShell;
implementation
uses LCLProc;
{$R *.lfm}
{ TfrmShell }
procedure TfrmShell.FormCreate(Sender : TObject);
begin
FProcess.Executable := '/bin/sh';
FProcess.Parameters.Add('-i');
FLastRunning := true;
FProcess.Execute;
FProcessReadData(Sender);
end;
procedure TfrmShell.btnExecClick(Sender : TObject);
var
inputstr : TCaption;
begin
inputstr := edtInput.Text;
Log('Sending input: "%s"',[inputstr]);
inputstr := inputstr + #10;
FProcess.Input.Write(inputstr[1],Length(inputstr));
end;
procedure TfrmShell.edtInputKeyUp(Sender : TObject; var Key : Word;
Shift : TShiftState);
begin
if Key = VK_RETURN then
btnExecClick(Sender);
end;
procedure TfrmShell.FormClose(Sender : TObject; var CloseAction : TCloseAction);
begin
FProcess.Terminate(0);
end;
procedure TfrmShell.FProcessReadData(Sender : TObject);
var newOutput : string;
begin
newOutput := ReadOutput(FProcess);
if newOutput <> '' then
Log(newOutput);
end;
procedure TfrmShell.FProcessTerminate(Sender : TObject);
begin
Log('Process Terminate');
end;
function TfrmShell.ReadOutput(const aproc : TAsyncProcess) : string;
var tempStrings : TStringList;
begin
tempStrings := TStringList.Create;
try
if assigned(aproc.Output) then
tempStrings.LoadFromStream(aproc.Output);
Result := tempStrings.Text;
finally
tempStrings.Free;
end;
end;
procedure TfrmShell.Log(const s : string);
begin
mmLog.Lines.Add(s);
end;
procedure TfrmShell.Log(const Fmt : String; const Args : array of const);
begin
Log(Format(Fmt,Args));
end;
end.
This kind of thing can't even see the prompt lines ending % or > or whatever as they're not part of the output piped to the stream, they go straight to the outer console if there is one.
-
Sir ,
may I suggest you to Free and Create a process for each command in ur list if possible.
cutting problems into little pieces often helps to light a solution and yes I tried inject a command line on a TProcess Input stream, it didn't work on my OSes grrr.
so I did what I mentioned above (cutting into lil pieces) at the moment , it works : fill the executable, parameters, command and THEN run commands one by one on a recreated TProcess descendant.
Sorry , that's the only way I found out until now. If your problem really can't be turned into multiples Freeed and recreated processes, if you need to "inject" commands into a terminal , then.... I don't know :(
-
found this thread around :
http://forum.lazarus.freepascal.org/index.php?topic=5986.0 (http://forum.lazarus.freepascal.org/index.php?topic=5986.0)
as you can see, a TCmdBox component exists for Lazarus that attempts to "emulate" a terminal , ... and encounter almost same issues.
Not sure , but i think that "Injecting" orders in a console process from outside would probably have security issues (I am also thinking about keyboard hooking, and many antivirus would yell at ya) on many platforms nowadays. This might be practically not possible %)
That is why , I'll keep the solution I gave before : 1 Process per command , and the results redirected to a only one display , mimicing but not emulating a terminal.
In the hope that my old programmer's brain helped.
Ty
-
Thanks again, I've played with the CmdBox component but it doesn't solve the problem it's just a different way of scraping the output (which is probably better for longer lasting processes).
I've tried creating/freeing per command in fact my first version executed commands as a static function on a class doing exactly that, the problem then is you've got no chance to respond to a prompt because the process wrapper either freezes because it is waiting for input (which you can't send even if you guess it's needed) or if you fire and forget it's been freed before the prompt happens and a new Tprocess is executing a new command not providing input to the previous.
-
Shouldn't you be able to start a process, get it's PID, and then monitor if process with such a PID is still alive? If it's not - you have a command prompt waiting for input. That will do only if you don't run processes in a way that they return command prompt while they are still running. Processes that show some message and require user input can be handled by simply parsing their output and reacting accordingly.
If I remember well, Krusader and Double Commander have embedded terminal, so maybe looking at their code you might get some better idea...
-
The problem with that approach (or one of executing commands one at a time via TProcess) is that, as you say, it doesn't deal with processes that have user interaction on the console while they're still running. The idea of looking at Krusader and Double Commander is good, I'm not familiar with them but I'll have a look.
-
Looks like both DoubleCommander and Krusader just invoke a shell (set by the user in options) along the lines of my first example although they don't even capture the output.
-
deal with processes that have user interaction on the console while they're still running.
= data injection = forbidden = NO
I told you
-
Then how does the Lazarus IDE do it?
-
nope !
where ? when ?
you mean the "messages" for compiler results and debugger ?
no "interaction" in them , just logs !
-
Not so, I mentioned it in my first post in this thread. I'm currently looking at the code in TPseudoConsoleDlg that does it.
-
Try my damn old OctaveGUI (https://code.google.com/p/octave-gui/downloads/list). I didn't wrap any shell, just execute the required process directly. I don't really remember the code, probably there was Windows specific code/treatment, but you can start with that.
-
https://github.com/alrieckert/lazarus/blob/master/debugger/pseudoterminaldlg.pp (https://github.com/alrieckert/lazarus/blob/master/debugger/pseudoterminaldlg.pp)
the mentioned TPseudoTerminalDlg uses debugwindows, (c.f. my above messages = keyboard hooking = implies to install a debugger , also implies hard time to make it totally cross-platform)
believe me , lol , threads safe OSes are built to BE threads safe and Processes safe, if you "inject data" by a back door it is
1) considered Hacking
2) not standard
3) unportable
4) breaking thru the security and making happy holes and open doors to pirats
-
Try my damn old OctaveGUI (https://code.google.com/p/octave-gui/downloads/list). I didn't wrap any shell, just execute the required process directly. I don't really remember the code, probably there was Windows specific code/treatment, but you can start with that.
Thanks that looks really interesting, I think you might have been lucky in the way Octave works but I'll look into it more deeply :)
-
https://github.com/alrieckert/lazarus/blob/master/debugger/pseudoterminaldlg.pp (https://github.com/alrieckert/lazarus/blob/master/debugger/pseudoterminaldlg.pp)
the mentioned TPseudoTerminalDlg uses debugwindows, (c.f. my above messages = keyboard hooking = implies to install a debugger , also implies hard time to make it totally cross-platform)
believe me , lol , threads safe OSes are built to BE threads safe and Process safe, if you "inject data" by a back door it is
1) considered Hacking
2) not standard
3) unportable
4) breaking thru the security and making happy holes and open doors to pirats
I disagree, there's no thread safety issues here. As for process safety, inter-process communication is a standard feature of OSs, the challenge is doing it in a nice cross platform way using Lazarus. I'm not suggesting using a back door to "inject data" but using a published command line api. I can do what I'm trying to achieve in other technologies (how do you think terminal emulators are written?) - what I'm interested in is using Lazarus to achieve the same result.
-
you said "interaction" = getting answers from , then put orders to = injecting , call it as you want , blue cow if you like , its the same
IPC is the linux version of COM/Ole , its based on binary messages , I know right , I use it TY
-
Furthermore :
As I am an Emulator Fan, I am on my way to make what you call a Terminal emulator :D
AND
so far mixing TCmdBox , TAsyncProcess plus a OEM to UTF8 translation , I really DO NOT encounter any problem.
2nd day of work (let say 10h) and I am here as follow :
1) I built some embeded commands begining with '!!'
like !!help , !!ver, !!cls, !!clh (clear history), !!bye, !!xitc (get exit code from last task) etc etc
2) I can launch any program or any embeded OS command with their parameters from inside My box by typing exactly the same way I would in a terminal
3) any launched task displays exactly what it would show inside a normal terminal, and I can catch the Exit Code
4) when the running task takes longer than 50 milisecond , 2 disabled buttons on the right of My emulated terminal become enabled with these functions : "Pause/Resume" , "Stop" the current task. these buttons return to grayed while no task active.
5) finally I plan to embed either PascalScript engine in the background, either my own simplified using commands like !!if , !!exist, !!fileage, !!echo(on/off), etc
COOL :P
...
type
TMyAsyncProcess = class(TAsyncProcess)
...
end;
{ TMainForm }
TMainForm = class(TForm)
cpEdit: TEdit;
Label2: TLabel;
Label3: TLabel;
StopButton: TButton;
CmdBox: TCmdBox;
Label1: TLabel;
PauseTBox: TToggleBox;
procedure CmdBoxOnEnter(Sender: TObject);
procedure CmdBoxOnInput(ACmdBox: TCmdBox; Input: string);
procedure FormCloseQuery(Sender: TObject; var CanClose: boolean);
procedure FormCreate(Sender: TObject);
private
{ private declarations }
rdbuf: array[0..8191] of Byte;
protected
CanExit: boolean;
console: TMyAsyncProcess;
procedure WriteHelp;
procedure ConsoleRead(Sender: TObject);
procedure ConsoleExec;
procedure cpChange;
..............
procedure TMainForm.CmdBoxOnInput(ACmdBox: TCmdBox; Input: string);
...
{$IF DEFINED(WINDOWS)}
console.Executable:='cmd.exe'; // My AsyncProcessConsole ;)
console.Parameters.Add('/C');
{$ELSEIF DEFINED(LINUX)}
console.Executable:='/bin/sh';
console.Parameters.Add('-i');
{$ELSE}
{$ERROR this must be compiled under windows or linux}
{$ENDIF}
console.CommandLine:=console.Executable+' '+console.Parameters[0]+' '+Input;
......
... to be continued
-
Yep, that's not dissimilar to my example previously. The problem is that technique won't pick up prompts and can't respond to them.
I think that what the IDE does (on linux/mac) is use GDB to reset the current processes tty console. So perhaps a way to do this is to do the same thing, or just redirect /dev/tty on linux/mac?
-
Yep, that's not dissimilar to my example previously. The problem is that technique won't pick up prompts and can't respond to them.
I think that what the IDE does (on linux/mac) is use GDB to reset the current processes tty console. So perhaps a way to do this is to do the same thing, or just redirect /dev/tty on linux/mac?
sorry hehehe, I was wrong, I found out the way to respond to prompts with pipes. I love bounties roflmao
-
try this , read the readme.txt file 1st
It works fine ! on windows at least
-
patch to mainunit , fixing conditional string conversion upon empty codepage :
procedure TMainForm.InjectButtonClick(Sender: TObject);
var
r: integer;
s: string;
p: PChar;
len: longint;
begin
AnsForm.Edit.Text:=EmptyStr;
r:=AnsForm.ShowModal;
if r<>mrOk then Exit;
try
if cpEdit.Text<>EmptyStr then
s:=ConvertEncoding(AnsForm.Edit.Text,GetDefaultTextEncoding,cpEdit.Text)
else
s:=AnsForm.Edit.Text;
Except
On Exception do s:=AnsForm.Edit.Text;
end;
s:=s+#$0D; p:=PChar(s);
len:=strlen(p);
Move(p^,wrbuf,Succ(len));
CmdBox.TextColor(clAqua); CmdBox.Writeln(AnsForm.Edit.Text);
console.Input.Write(wrbuf,len);
end;
and
procedure TMainForm.ConsoleRead(Sender: TObject);
var
nb,rd: dword;
p: PChar;
s: string;
begin
nb:=console.NumBytesAvailable;
rd:=console.Output.Read(rdbuf,nb);
rdbuf[rd]:=0;
p:=@rdbuf;
try
if cpEdit.Text<>EmptyStr then
s:=ConvertEncoding(PChar(@rdbuf),cpEdit.Text,EncodingUTF8)
else
s:=PChar(@rdbuf);
Except
On Exception do s:=PChar(@rdbuf);
end;
CmdBox.Write(s);
end;
-
why do we need a script system ?
simply because we want "automated" answers to prompts instead of giving them manually, using the "Answer" button.
So
with a script command like !!onscan "text","answer" <console script launch command> , my T2 could analyse output text from console scripts and react properly automatically by injecting the proper answer.
even more !!onscan2 "datafile" <console script lauch command> could scan question patterns from the console output and inject the proper answerS
with a "datafile" looking like
enter your name, John
enter your age, 33
want to rebuild or compile, C
...
that sounds easy to do : before the comma, the piped output sentence to scan and check, after the comma, the answer to write to the piped input
-
Thanks Sam, I think you're getting a bit carried away with writing a terminal app, for my requirements at least. I'd love to see a Lazarus terminal app but that's not what my bounty is for. To clarifiy I need a solution (preferably just a class) that:
- allows invocation of an arbitrary command line application
- catches all textual output (stdout and stderr)
- catches or somehow detects applications raising prompts (this is killer requirement #1, think sudo or something to test it)
- allows sending a response to prompts and continuing execution (this is killer requirement #2)
- doesn't create a visible cli/xterm console window
- works on linux, mac and windows platforms (32/64)
I don't really want any other dependencies such as pascalscript, cmdbox etc. I just want a class that can do this stuff. Simple putting a memo and a bunch of buttons on form is enough for me. If someone adds all that stuff it's just work for me to strip it out later.
For those interested on working this that are wary of the Mac requirement, don't worry too much, in my experience solutions that work on linux tend to work on Macs. The bounty would still be up for grabs at 80% for a solution that's just windows and linux.
-
Here's the interface for my ideal solution (details aren't important):
TCmdLineAutomation = class(TObject)
public
property CurrentDirectoy : string read FCurrentDirectoy write SetCurrentDirectoy;
property IsRunning : boolean read FIsRunning write SetIsRunning;
property CommandLine : string read FCommandLine write SetCommandLine; //or cmd + args
//input
property OnPrompt : TNotifyEvent read FOnPrompt write SetOnPrompt;
property LastPrompt : string read FLastPrompt write SetLastPrompt;
procedure SendInput(const inputStr : String);
//output
property OnOutputRecieved : TNotifyEvent read FOnOutputRecieved write SetOnOutputRecieved;
property Output : TOutputStream read FOutput write SetOutput;
end;
-
Yeah, probably needs an execute command too :-[
-
This bounty is still open if anyone is interested.
-
This bounty is now closed.
The experience of running a bounty was a weird one. There were a few time wasters along the way and then finally two serious attempts to win it - in the same two weeks after 2 years of being open!
I created some tests: https://github.com/macmike/console_automation_tests/releases/tag/v0.1
Which were all passed today: https://github.com/macmike/console_automation_tests/releases/tag/v0.2
Thanks to skalogryz :)