Lazarus

Programming => General => Topic started by: toby on January 14, 2023, 01:37:02 am

Title: capture apl_exec stdout
Post by: toby on January 14, 2023, 01:37:02 am
hello

i can run the libapl apl_exec function in an fpc program but need to capture it's stdout and redirect to an output buffer instead of the screen and then covert into ansistring in the fpc program itself not as an external program which runcommand will do

this is example c++ code that does this

std::stringstream outbuffer;
std::streambuf *coutbuf = std::cout.rdbuf();
std::cout.rdbuf(outbuffer.rdbuf());
std::stringstream errbuffer;
std::streambuf *cerrbuf = std::cerr.rdbuf();
std::cerr.rdbuf(errbuffer.rdbuf());
execerr = apl_exec (cmd.toStdString ().c_str ());
std::cout.rdbuf(coutbuf);
std::cerr.rdbuf(cerrbuf);
outString = QString (outbuffer.str ().c_str ());
errString = QString (errbuffer.str ().c_str ());

'while apl_exec is running, standard output and standard error will be redirected to output and error buffers instead of the screen. These are then converted into strings.'

Title: Re: capture apl_exec stdout
Post by: Thaddy on January 14, 2023, 11:02:15 am
Simple example:
Code: Pascal  [Select][+][-]
  1. program redirectstdout;
  2. {$apptype console}
  3. uses streamIO,classes;
  4. var
  5. f: TextFile;
  6. s: TStringStream;
  7. begin
  8.   f:=stdout;
  9.   s := TStringStream.Create;
  10.   AssignStream(f, s);
  11.   Rewrite(f);
  12.   Writeln(f, 'Hello World');
  13.   CloseFile(f);
  14.   writeln('redirected: ', s.datastring);
  15.   s.Free;
  16. end.

But it is better to use Tprocess, simple example:
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$H+}
  2. uses sysutils,classes, process;
  3.  
  4. var
  5.  p:Tprocess;
  6.  l:Tstrings;
  7.  readcount:integer;
  8.  a,b:string;
  9.  i:integer;
  10. begin
  11.   p :=TProcess.Create(nil);
  12.   try
  13.     l := Tstringlist.create;
  14.     try
  15.       try
  16.         P.Options := [];
  17.         p.executable:='D:\fpctrunk\bin\x86_64-win64\ppcx64.exe';
  18.         p.parameters.Add('-h');
  19.         p.runcommandloop(a,b,i);// accumulate stdout in a, stderr in b
  20.         l.add(a);
  21.         l.add(b);
  22.       except
  23.         on E:Exception do writeln(E.Message);
  24.       end;
  25.       writeln(l.text);
  26.     finally
  27.       l.free;
  28.     end
  29.   finally
  30.     p.free;
  31.   end;
  32.   readln;
  33. end.

Side note: I was not aware of TProcess.RunCommandLoop until I wrote the example. It basically avoids messing with pipes yourself and greatly simplifies things. Because I was not aware of that, note you may need a fpc version 3.2.0 or higher, but then again it may have always been there.

If you have Pascal headers or C headers for libapl I can give a much better example, but when I try to find them I ran into many 404s

Title: Re: capture apl_exec stdout
Post by: toby on January 14, 2023, 09:23:46 pm

Hi Thaddy,

your second example is for running an external program which is what i resorted to using my libapl/fpc coding but i need one fpc program that can run apl_exec and bring the results into the fpc program itself - it gets really complicaed having to maintain separate programs for each apl_exec line etc

i used your first code example and it worked for your example but the apl_exec stdout was not the screen output display but only the return value (0)

i would be glad to supply my fpc headers and c headers (libapl) but first do you have a working apl/libapl installed?

there are c/libapl and python3/lib_gnu_apl (and even lua  headers but i don't use them) included with the apl svn source

i compiled apl from svn source to get libapl/c and lib_gnu_apl/python3 headers but don't know if the distro versions have these done

i have c and python3 libapl programs using apl_exec that can test if you have a good apl/libapl installation

my fpc headers are simple and use the c/libapl headers and give good libapl/apl_exec for some very complicated apl expressions (the apl maintainer says that what i am doing shouldn't be doable with what he knows about the libapl - libapl is actually c headers for apl and separately authored from apl itself)

i can do everything i want with apl and fpc using external apl scripting libapl/fpc programs using runcommand but the libapl runs the same code almost 2x faster and

the libapl (whether fpc/c/python3) appears to be giving it's output to something other then stdout

i believe i am the only person working with fpc-3.2.2 and apl/libapl

---
Title: Re: capture apl_exec stdout
Post by: TRon on January 14, 2023, 09:47:22 pm
the libapl (whether fpc/c/python3) appears to be giving it's output to something other then stdout
I/O channels depend on how things were build, see https://gist.github.com/houmei/cfd9e570b8de4d8fd55ada228d5ff004#file-readme-2-configure

Quote
i believe i am the only person working with fpc-3.2.2 and apl/libapl
Official GNU sources for apl do not compile on my setup (debian gcc 12.2). Official GNU apl .deb does not seem to include libapl.
Title: Re: capture apl_exec stdout
Post by: toby on January 14, 2023, 10:42:29 pm
i have been asked if

> Does freepascal start foreign code in a separate process?  Without reopening stdout?

---

Hi TRon

I/O channels depend on how things were build, see https://gist.github.com/houmei/cfd9e570b8de4d8fd55ada228d5ff004#file-readme-2-configure

compiling libapl --with-android    made no difference-

-

'Official GNU sources for apl do not compile on my setup (debian gcc 12.2). Official GNU apl .deb does not seem to include libapl.'

this is what i thought was the situation

I would be glad to help you compile apl and libapl   i am on a 'debian' system with gcc-12.2.0 also

this is the svn line
svn co https://svn.savannah.gnu.org/svn/apl/trunk apl

you need to compile apl and libapl separately

for apl
configure --without-gtk3 --without-sqlite3 --without-postgresql --without-pcre
make
make install

type 'apl'  abd then  2+2    )off   to exit      you will need apl keyboard and apl font we can do them after things work

for libapl
configure --with-libapl --without-gtk3 --without-sqlite3 --without-postgresql --without-pcre
make
make install
check if /usr/local/lib/apl/libapl.[a la so] have been created
you need to edit /etc/ld.so.conf  and put the line   /usr/local/lib/apl     in it
ldconfig

this is example ;ibapl/c code

// libapl.c    compile with : g++ -O2 libapl.c -o libapl -L /usr/local/lib/apl -lapl
#include <stdio.h>
#include <stdlib.h>
#include <apl/libapl.h>
int main (int argc, char * argv[])
{
init_libapl(argv[0], 0);
apl_exec("2+2");
return 0;
}

i had to edit some source files to make libapl work - lets see if your system causes problems before we edit them

i have also edited files to change locations of default apl config files etc - lets see what happens with default compiles and installation and if you want to use what i have done

let me know how this goes

Title: Re: capture apl_exec stdout
Post by: TRon on January 14, 2023, 11:59:20 pm
@toby: can't oblige right now. will try again as soon as time permits.
Title: Re: capture apl_exec stdout
Post by: toby on January 15, 2023, 02:05:07 am
this is the fpc header file i use - it give me complete access to all the apl commands with apl_exec - i left all the test lines (commented) that i have tried to no effect


unit libaplu;

interface

// extern void init_libapl(const char * progname, int log_startup);
procedure init_libapl(progname : ansistring; log_startup : integer); cdecl;

//extern LIBAPL_error apl_exec(const char * line_utf8);
function apl_exec(line_utf8 : ansistring) : longint; cdecl;
//function apl_exec(line_utf8 : pchar) : longint; cdecl;

//extern const char * apl_command(const char * command_utf8);
function apl_command(command_utf8 : pchar) : pchar; cdecl; // good : output is apl result
//function apl_command(command_utf8 : pchar) : longint; cdecl; // good : output is integer
//function apl_command(command_utf8 : pchar) : ansistring; cdecl; // does not worl
//function apl_command(command_utf8 : ansistring) : ansistring; cdecl; // does not work
//function apl_command(command_utf8 : ansistring) : longint; cdecl; // does not work

implementation

procedure init_libapl(progname : ansistring; log_startup : integer); cdecl; external;

function apl_exec(line_utf8 : ansistring) : longint; cdecl; external;
//function apl_exec(line_utf8 : pchar) : longint; cdecl; external;

function apl_command(command_utf8 : pchar) : pchar; cdecl; external;
//function apl_command(command_utf8 : pchar) : longint; cdecl; external;
//function apl_command(command_utf8 : pchar) : ansistring; cdecl; external;
//function apl_command(command_utf8 : ansistring) : ansistring; cdecl; external;
//function apl_command(command_utf8 : ansistring) : longint; cdecl; external;

end.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 15, 2023, 08:30:26 am
AnsiString functions need wrappers because it is a Pascal string type that apl does not know about, e.g:
Code: Pascal  [Select][+][-]
  1. unit libaplu;
  2. {$mode objfpc}{$H+ string=ansistring}
  3. interface
  4. { Pascal function wrappers }
  5. procedure init_libapl(progname : string; log_startup : integer = 0);inline;
  6. function apl_exec(line_utf8 : string) : integer;inline;
  7. function apl_command(command_utf8 : string):string;inline;
  8.  
  9. { external declarations , no need to use them}
  10. procedure init_libapl(progname : PChar; log_startup : longint);cdecl;external;
  11. function apl_exec(line_utf8 : pchar) : longint; cdecl; external;
  12. function apl_command(command_utf8 : pchar) : pchar; cdecl; external;
  13.  
  14. implementation
  15.  
  16. procedure init_libapl(progname : string; log_startup : integer);
  17. begin
  18.   init_libapl(Pchar(progname),log_startup);
  19. end;
  20.  
  21. function apl_exec(line_utf8 : string) : integer;
  22. begin
  23.    Result :=apl_exec(PChar(line_utf8));
  24. end;
  25.  
  26. function apl_command(command_utf8 : string) : string;
  27. var temp:Pchar;
  28. begin
  29.    temp:=apl_command(pchar(command_utf8));
  30.    setstring(Result,temp, strlen(temp));
  31. end;
  32.  
  33. end.
I removed nonsense declarations.
Btw: you were lucky that your own init_libapl (AnsiString) worked!! I have corrected that.
Untested, but compiles and should be OK. The cdecl is not necessary for the wrappers.

Does anybody have a libapl.dll binary? (pref 64 bit, but 32 bit is acceptable)
Title: Re: capture apl_exec stdout
Post by: TRon on January 15, 2023, 12:35:17 pm
Thank you for your header code Thaddy.

Does anybody have a libapl.dll binary? (pref 64 bit, but 32 bit is acceptable)
Does this (https://mirror.koddos.net/gnu/apl/apl-1.8-windows.zip) satisfy your needs ?

@toby:
Thanks. I got trunk compiled now. libapl Is working as expected (even though I took a bit of another route but that is besides the point).

I am able to confirm that attempting to reroute as suggested by Thaddy's assignstream example (which was my first thought as well) does indeed not work as expected. If at all possible, I would need more time to figure out how.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 15, 2023, 02:51:07 pm
Can you confirm my code works, the wrappers that is?

Thank you for your header code Thaddy.

Does anybody have a libapl.dll binary? (pref 64 bit, but 32 bit is acceptable)
Does this (https://mirror.koddos.net/gnu/apl/apl-1.8-windows.zip) satisfy your needs ?
Alas no, did you get it to work? Seems only the keyboard driver.
Note I am missing some unitialize call. I am sure there must be one, because the dll or shared library uses its own memory management.
Can you post the signature for that? Then I can also add that one too.
If you experience any problems with my code plz inform here, but I am quite confident that it works.

About the console output: I probably need to change the APL sourcecode to be able to capture the handles.
But that is doable.
Title: Re: capture apl_exec stdout
Post by: TRon on January 15, 2023, 05:30:58 pm
Can you confirm my code works, the wrappers that is?
Yeah wrappers work, (just some minor stuff see also rest of my answer)

Quote
Alas no, did you get it to work? Seems only the keyboard driver.
Sorry. I was not aware the zipfile only contained the keyboard driver. I simply noticed the dll's in there.

I did get libapl to compile and run as intended (on Linux) but alas the AssignStream was unable to catch the output from the library.

I assume this is because the c compiler's COUT/CERR/CIN and FPC's used handles are (internally) not using the same descriptors.

Sofar, I seem unable to solve that. As last resort it might perhaps be possible to redirect /proc/ IO filehandles but am not sure if that will work.

Quote
Note I am missing some unitialize call. I am sure there must be one, because the dll or shared library uses its own memory management.
Can you post the signature for that? Then I can also add that one too.
So far as I am able to see there simply is no uninitialization provided. library's include file header: https://svn.savannah.gnu.org/viewvc/apl/trunk/src/libapl.h?view=markup

Quote
If you experience any problems with my code plz inform here, but I am quite confident that it works.
The library itself, yeah that is no problem. I've made some minor correction to your (and toby's) code but nothing dramatic (e.g. parameter naming that reflects documentation, and I had to add the name of the library to the external modifier)

Quote
About the console output: I probably need to change the APL sourcecode to be able to capture the handles.
But that is doable.
It uses standard COUT/CERR/CIN.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 15, 2023, 06:26:45 pm
Can you give us the modifications? By the end of the week we have a package  :D!

And yes, on linux everything works as expected (Debian and RaspberryOS tested, just Windows does not play nice).
I should be able to solve the IO issues. (and everything else in that header file, tnks)
It will be a nice addition to the standard packages.

I hope you did not touch the wrappers, though: that is pure pascalification and has almost no speed penalty because I inlined it.
Title: Re: capture apl_exec stdout
Post by: Fred vS on January 15, 2023, 08:00:42 pm
If you want to capture all the output to terminal in Unix OS, you may use FpDup2

https://www.freepascal.org/docs-html/rtl/baseunix/fpdup2.html

Here example how to use it for redirect errors (you may redirect all stdout also, see fpc fpdup2 demo)

Code: Pascal  [Select][+][-]
  1. program redirect-stdout;
  2.  
  3. uses
  4.  BaseUnix;
  5.  
  6. var
  7. fs: TFileStream;
  8. ordir: string;
  9.  
  10. begin
  11.  ordir := ExtractFilePath(ParamStr(0)) +
  12.             directoryseparator + 'log.txt' ;
  13.    
  14.   fs := TFileStream.Create(ordir, fmOpenReadWrite or fmCreate);
  15.  
  16.   FpDup2(fs.Handle, StdErrorHandle);  
  17.  
  18.   // do your things ...
  19.  
  20.  application.run;
  21.  
  22.   fs.Free;
  23.  
  24. end.
Title: Re: capture apl_exec stdout
Post by: toby on January 15, 2023, 10:08:00 pm
Thaddy

your libaplu.pas compiles and works fine   

i don't think there is an uninitialize call - (there is nothing in the /usr/local/include/apl/libapl.h and when working with the apl author on an error from not having init_libapl only needing the init_libapl was mentioned as missing from it

there is no unitialize in any of the c or python3 libapl code either

of interest : for full display when initializing libapl i use
init_libapl(paramstr(0), 1);

Title: Re: capture apl_exec stdout
Post by: toby on January 16, 2023, 01:36:08 am

Tron

in response to your post #10

Thaddy :
About the console output: I probably need to change the APL sourcecode to be able to capture the handles.
But that is doable.

Tron :
It uses standard COUT/CERR/CIN.

according to the link you gave in your post #3
https://gist.github.com/houmei/cfd9e570b8de4d8fd55ada228d5ff004#file-readme-2-configure

which says
-with-android
This option prevents the instantiation of CIN, COUT, and CERR. This maybe needed when GNU APL is compiled for Android, since Android applicationsare using a different I/O model than standard C++ programs.

when i compile my libapl --with-android and then my libapl program with new android libapli get
/usr/local/lib/apl/libapl.so: undefined reference to `CIN'
/usr/local/lib/apl/libapl.so: undefined reference to `UERR'
/usr/local/lib/apl/libapl.so: undefined reference to `COUT'
/usr/local/lib/apl/libapl.so: undefined reference to `CERR'

this is what i was told about these compiler errors
Those variables are C++ variables, so the names are probably mangled. You
can try to change the GNU APL code to declare them as extern "C".


i mistakingly said compiling --with-android made no difference (i forgot to copy the new libapl to /usr/local/lib/apl/libapl.so)  before compiling

would setting COUT to stdout be this solution for the stdout problem in the fpc code using libapl?


Title: Re: capture apl_exec stdout
Post by: Thaddy on January 16, 2023, 05:15:09 am
CIN COUT and CERR already map to stdin, stdout and stderr. Different names, but exactly the same.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 16, 2023, 05:33:17 am
of interest : for full display when initializing libapl i use
init_libapl(paramstr(0), 1);
Then you can change 0 to 1 if that is preferred, e.g.:
Code: Pascal  [Select][+][-]
  1. procedure init_libapl(progname : string; log_startup : integer = 1);inline;
  2. // and subsequently
  3. init_libapl(Paramstr(0));
Since that a.o. means that libapl inherits the process handle (Paramstr(0)) that actually means that redirecting the output is much easier than I initially thought.
Title: Re: capture apl_exec stdout
Post by: TRon on January 16, 2023, 05:34:11 am
Can you give us the modifications? By the end of the week we have a package  :D!
No problem. What I have is attached.

Quote
I hope you did not touch the wrappers, though: that is pure pascalification and has almost no speed penalty because I inlined it.
https://www.youtube.com/watch?v=otCpCn0l4Wo  :P

If you want to capture all the output to terminal in Unix OS, you may use FpDup2
Yes the last resort is to try redirect all terminal output.

Your example works as expected (thank you very much for that example code Fred vS).

The only drawback is the need for a file (or files in this case: err and out)

example 3 in the attachment show the use of your approach.

i mistakingly said compiling --with-android made no difference (i forgot to copy the new libapl to /usr/local/lib/apl/libapl.so)  before compiling

No problem. I was expecting erors like that would surface when compiling for android.

Thanks you for the feedback toby.

Quote
would setting COUT to stdout be this solution for the stdout problem in the fpc code using libapl?
No, unfortunately that would not solve the issue. Its pretty technical, especially on/for Linux. E.g. I am trying to find a better solution but I am also not very well versed in such matters.

More important (e.g. first step) is that the library works as expected  :D


attachment update 2: I have re-arranged the examples, included some original documentation from APL trunk and added the two callback related functionality (with very simple invocation examples)
attachment update 2b: Added Thaddy's additions.
attachment update 3: almost all API calls are now available. Added, extended callback example, redirection example, license (PIF) and ulibapl.pas can now be compiled in mode Delphi as well.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 16, 2023, 05:47:55 am
Thanks for the code.
Btw, I also had strDispose(), but that is really not necessary because the temp PChar is allocated on the stack and so disappears after the function returns (goes out of scope). StrDispose is to free heap memory, see the documentation for strDispose, temp is stack.

I think I also solved the IO problem, stdout redirection works:
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$H+}
  2. uses classes,streamIO, libaplu;
  3. var
  4.   OldOutput,
  5.   f: TextFile;
  6.   s: TStringStream;
  7. begin
  8.   s := TStringStream.Create;
  9.   AssignStream(f, s);
  10.   Rewrite(f);
  11.   OldOutput:=Output;
  12.   Output:=f;
  13.   init_libapl(paramstr(0),1);
  14.   { any output from apl should  be redirected }
  15.   Writeln(Boolean(apl_command('2 < 1')));
  16.   Close(f);
  17.   Output:=OldOutPut;
  18.   writeln('redirected: ', s.datastring);
  19.   s.Free;
  20. end.
AFAICT APL uses the same handles as the pascal code.
Needs some more work for stdin and stderr.
Please test. If it does not work I will send you a patched libapl. (very crude, but also works)
Title: Re: capture apl_exec stdout
Post by: toby on January 16, 2023, 08:41:00 pm
Thaddy

are you using the same libaplu.pas that you posted earlier?

i had to add {$linklib apl} to top before uses ? don't you need it?  what is your compile line or /etc/fpc.cfg change?

and get
stdout.pas(18,15) Hint: Variable "f" does not seem to be initialized
stdout.pas(31,9) Error: Illegal type conversion: "AnsiString" to "Boolean"
stdout.pas(44) Fatal: There were 1 errors compiling module, stopping

if i add f := stdout;  right after begin
i now get
stdout.pas(31,9) Error: Illegal type conversion: "AnsiString" to "Boolean"

if i change
//Writeln(Boolean(apl_command('2 < 1')));      the '2 < 1' is an apl expression not an apl command
Writeln(Boolean(apl_exec('2 < 1')));

i get
0
redirected: FALSE

the result of  '2 < 1' is 0 not false
the false comes from the stdout of '0' and then your boolean(0) that is the stdout from all apl_exec expressions signifying sucdess with current libapl

use '2 < 1 1 1' to clarify if you are really getting apl_exec stdout and eliminate '0' 'false' confusion

---

using writeln(apl_command(')help'));         does indeed work
writeln('length(s.datastring) : ', length(s.datastring));
4506

---
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 16, 2023, 08:58:05 pm
Yes, there were some differences between the code I posted and the code I was working with. Correct that tommorow.
But did the redirect work for you?
Title: Re: capture apl_exec stdout
Post by: toby on January 16, 2023, 09:30:01 pm
I have modified my post 19 since you did post 20

using apl_command(')help'); is redirected
but apl_exec('2 < 1 1 1');   doesn't

Title: Re: capture apl_exec stdout
Post by: marcov on January 16, 2023, 09:44:16 pm
If you use commandline style redirection, either the library function you use needs to execute a shell (like system())) , or you have to specify a shell yourself.
Title: Re: capture apl_exec stdout
Post by: TRon on January 17, 2023, 12:19:15 am
Btw, I also had strDispose(), but that is really not necessary because the temp PChar is allocated on the stack and so disappears after the function returns (goes out of scope). StrDispose is to free heap memory, see the documentation for strDispose, temp is stack.
Sorry for that.

Quote
Please test. If it does not work I will send you a patched libapl. (very crude, but also works)
It works. That is the Output is redirected as expected. Thank you for that as I was not aware that you could 'trick' IO that way.

Unfortunately doing the same with ErrOutput does not. I assume the library uses the special UERR handle for that. (I assume(d) you have figured that yourself )


If you use commandline style redirection, either the library function you use needs to execute a shell (like system())) , or you have to specify a shell yourself.
Skimming posts can be risky, as is in this case  ;)

Your input is appreciated but the used characters have nothing to do with redirection. It is how the APL language itself operates.

TS is trying to retrieve the output from the library (that is emitted to the terminal). I assume he wishes to do that so that the library can be used in a Lazarus GUI program.
Title: Re: capture apl_exec stdout
Post by: Fred vS on January 17, 2023, 03:56:21 am
If you want to capture all the output to terminal in Unix OS, you may use FpDup2
Yes the last resort is to try redirect all terminal output.

Your example works as expected (thank you very much for that example code Fred vS).

The only drawback is the need for a file (or files in this case: err and out)

Hello.

Yes, indeed it needs a file but only one.
If you want to have err and out in one file, in your apl_test3.pas, just use one TFileStream and 2 fpdup2():

Code: Pascal  [Select][+][-]
  1. ...
  2.  
  3.   fs1 := TFileStream.Create(prglog1, fmOpenReadWrite or fmCreate);
  4.  
  5.   fpdup2(fs1.handle, StdErrorHandle);
  6.   fpdup2(fs1.handle, StdOutputHandle);
  7. ...
  8.  

Maybe there is a trick to use a TMemorysteam vs a TFileStream for fpdup2 but I did not find the way.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 08:49:58 am
Maybe there is a trick to use a TMemorysteam vs a TFileStream for fpdup2 but I did not find the way.
What you need is a THandleStream or descendant (that may behave like a memorystream).
That's because fpdup2 needs an OS handle.
The unit process contains TInputPipeStream and TOutputPipeStream, which descent from THandlestream.
These work with fpdup2.

Note that my approach is "a bit"  :D more cross-platform than fpdup2 and works under linux and windows, probably every platform that accepts objfpc. fpdup2 is unix only.
The two streams mentioned are also X-platform and indepent of the TProcess class.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 09:35:13 am
I have added the *_ucs calls and overloaded the pascal wrappers for UCS/Unicode:
Code: Pascal  [Select][+][-]
  1. unit libaplu;
  2. {$mode delphi}{$H+ string=ansistring}
  3. interface
  4. { Pascal function wrappers }
  5. procedure init_libapl(progname : string; log_startup : integer = 1 {based on Toby's suggestion });inline;
  6. function apl_exec(line_utf8 : string) : integer;inline;overload;
  7. function apl_exec(line_utf8 : unicodestring) : integer;inline;overload;
  8. function apl_command(command_utf8 : string):string;inline;overload;
  9. function apl_command(command_utf8 : unicodestring):unicodestring;inline;overload;
  10.  
  11. { external declarations, better use the pascal wrappers }
  12. procedure init_libapl(progname : PChar; log_startup : longint);cdecl;external;
  13. function apl_exec(line_utf8 : pchar) : longint; cdecl; external;
  14. function apl_exec_ucs(line_utf8 : pWidechar) : longint; cdecl; external;
  15. { note if you use the next functions with a global variable for the result,
  16.   you must call dispose. For local variables this is not needed. }
  17. function apl_command(command_utf8 : pchar) : pchar; cdecl; external;
  18. function apl_command_ucs(command_utf8 : pWidechar) : pWidechar; cdecl; external;
  19.  
  20. implementation
  21.  
  22. procedure init_libapl(progname : string; log_startup : integer);inline;
  23. begin
  24.   init_libapl(Pchar(progname),log_startup);
  25. end;
  26.  
  27. function apl_exec(line_utf8 : string) : integer;
  28. begin
  29.    Result :=apl_exec(PChar(line_utf8));
  30. end;
  31.  
  32. function apl_command(command_utf8 : string) : string;
  33. var temp:Pchar;
  34. begin
  35.    temp:=apl_command(pchar(command_utf8));
  36.    setstring(Result,temp, length(temp));
  37. end;
  38.  
  39. function apl_exec(line_utf8 : unicodestring) : integer;
  40. begin
  41.    Result :=apl_exec_ucs(PWideChar(line_utf8));
  42. end;
  43.  
  44. function apl_command(command_utf8 : unicodestring) : unicodestring;
  45. var temp:PWideChar;
  46. begin
  47.    temp:=apl_command_ucs(pWidechar(command_utf8));
  48.    setstring(Result,temp,length(temp));
  49. end;
  50.  
  51. end.

The UCS wrappers may be more comfortable since the APL syntax can be processed more natural.
I have overloaded them since that is more natural to Object Pascal.
(Note UCS is actually UCS2 which is a subset of Unicode16, limited to 16 bit codepoints)
Title: Re: capture apl_exec stdout
Post by: TRon on January 17, 2023, 09:54:11 am
I have edited post #17 to attach an updated zipfile for libapl.

I have re-arranged the examples, included some original documentation from APL trunk and added the two callback related functionality (with very simple invocation examples)

@Thaddy:
Thank you for the update. fwiw I've tried handlestream and got nowhere :-S

I am confused as strace clearly shows it using a handle with a value of 2. e.g. the redirection on stderr (as you did for stdout) should then in theory work ?

Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 09:59:05 am
It *should* work. Will investigate. I will add the redirect options to the unit when written and tested.
Btw I link from the command line during development and use an external linker ld on most platforms except windows.
Title: Re: capture apl_exec stdout
Post by: marcov on January 17, 2023, 10:09:18 am

If you use commandline style redirection, either the library function you use needs to execute a shell (like system())) , or you have to specify a shell yourself.
Skimming posts can be risky, as is in this case  ;)

Your input is appreciated but the used characters have nothing to do with redirection. It is how the APL language itself operates.

TS is trying to retrieve the output from the library (that is emitted to the terminal). I assume he wishes to do that so that the library can be used in a Lazarus GUI program.

I reacted to the Toby post immediately above it. Should have quoted.

Another tidbit that surfaced while reading through this thread: While I do work on TProcess from time to time, I am not that versed in *nix side of things anymore. But iirc some kinds of redirection there worked on glibc ( or c++ stdlib?)  level rather than on handle level. It is possible the C++ code does that.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 10:21:34 am
It is possible the C++ code does that.
The U part error is non-standard user error re-direction and needs a call back afaict. But it is no problem to use a file for that. It is used to report syntax errors etc.
CIN,COUT and CERR have no problems, nor on windows nor on linux, simple standard OS handles. libapl simply inherits the handles of the pascal process.
Much easier than I thought.
(My test environment is windows 10 windows 11 and debian bullseye, all 64 bit)
Title: Re: capture apl_exec stdout
Post by: TRon on January 17, 2023, 10:24:15 am
It *should* work.
See example named redir2b.pas

Quote
Will investigate. I will add the redirect options to the unit when written and tested.
appreciated. :thumbsup:

BTW: I've updated post #17 again with your additions. Thank you Thaddy !
Title: Re: capture apl_exec stdout
Post by: TRon on January 17, 2023, 10:25:18 am
The U part error is non-standard user error re-direction and needs a call back afaict. But it is no problem to use a file for that. It is used to report syntax errors etc.
CIN,COUT and CERR have no problems, nor on windows nor on linux, simple standard OS handles. libapl simply inherits the handles of the pascal process.
Much easier than I thought.
Unfortunately UERR is also used for command output  :'(
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 10:30:11 am
Then we need to duplicate the handles....?
Title: Re: capture apl_exec stdout
Post by: TRon on January 17, 2023, 10:30:22 am
I reacted to the Toby post immediately above it. Should have quoted.
Sorry i should have been more explicit.

I assumed that you "got trigerred" by

using apl_command(')help'); is redirected
but apl_exec('2 < 1 1 1');   doesn't
And I understand why (hence my reaction) :)

Quote
Another tidbit that surfaced while reading through this thread: While I do work on TProcess from time to time, I am not that versed in *nix side of things anymore. But iirc some kinds of redirection there worked on glibc ( or c++ stdlib?)  level rather than on handle level. It is possible the C++ code does that.
Thank you for that insight. For sure there is something odd there as redirecting stderr to a file (as suggested by Fred vS) has the wanted effect alas it seems that can't be replicated by using other means (at least not to my current knowledge). I could perhaps try using pipes or reroute using another (new created) handle.
Title: Re: capture apl_exec stdout
Post by: TRon on January 17, 2023, 10:47:17 am
Then we need to duplicate the handles....?
Sorry, I have to correct myself There. It is not the command output rather the output (apl_exec() result ) from the interpreter that seem to get emitted to UERR.

TBH I have no idea how to approach this atm.

My intial thought is/was to compile libapl with the android option and provide/export the missing variables in FPC. But then I would have no idea how C expects CIN/COUT/CERR and UERR to be initialized and/or how they are stored in memory. My knowledge on c++ internals is rather empty.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 11:10:20 am
No, you do not have to correct yourself, it is correct,  what you stated is to the point. Working on it.
CIN/COUT/CERR is done and was easy. Trying to find a creative solution for UERR, that is not standard for C++ iostreams.
Basic Idea is to clone the handle from stdout to UERR. Possibly the other way around  ;D
May be as easy as moving the apl_init call up in the call chain.
Will report back on progress.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 17, 2023, 11:30:15 am
One important thing to note too, is that when you write a GUI app, for the redirection to work the GUI application needs to be compiled with {$apptype console} NOT {$apptype GUI} where you hide the console in code. This is because {$apptype GUI} does not generate handles for stdin/out/err. (or you need to create the console manually). You will still have a GUI application.

Code will follow, now busy solving UERR.
Title: Re: capture apl_exec stdout
Post by: toby on January 19, 2023, 12:24:42 am
Fred VS

what about assigning what is written to the file to an ansistring instead or better adding it to a tstringlist.add(...)?

it is possble?

Title: Re: capture apl_exec stdout
Post by: Fred vS on January 19, 2023, 12:43:32 am
Fred VS

what about assigning what is written to the file to an ansistring instead or better adding it to a tstringlist.add(...)?

it is possble?

fpdup2 needs an OS handle, like Thaddy explained here:
https://forum.lazarus.freepascal.org/index.php/topic,61841.msg467050.html#msg467050

So instead of a file (if you dont like it) you can try with a THandleStream (you may ask to Thaddy how to do it  ;) )
Title: Re: capture apl_exec stdout
Post by: marcov on January 19, 2023, 10:14:43 am
See https://www.freepascal.org/docs-html/rtl/sysutils/getfilehandle.html to get OS handle from file type.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 19, 2023, 02:10:24 pm
Making progress:

Set WriteErrorsToStdErr to False. This is a FPC system variable which defaults to true.
If it is false everything goes to stdout/Output.
UERR is simply a set of C++11 formatting macro's and relates only stdout and stderr, so we do not need to handle it in any special way.

https://freepascal.org/docs-html/rtl/system/writeerrorstostderr.html

Quite a bit off the beaten track, here... Hmm, should have known that, but I did't... Not even its existance.
Title: Re: capture apl_exec stdout
Post by: TRon on January 19, 2023, 02:17:02 pm
@toby:
afaik there is something wrong with the (default) make/configuration options.

When building the library as you described it generated the libapl.so file.

However, when you try to load that library dynamically you get a: "Error loading library /path/to/library/libapl.so: undefined symbol: main" indicating that this is not a standalone library rather expects the library to be (statically) linked against.

Adding --enabled-shared to the configuration does not help.
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 19, 2023, 02:26:11 pm
@TRon

I prefer to link it statically for the time being. Does that work for you?
If not we three need to synchronize our libraries so we are barking all up to the same tree.. :) O:-)

btw UERR seems fixed here.
Title: Re: capture apl_exec stdout
Post by: TRon on January 19, 2023, 03:17:34 pm
I prefer to link it statically for the time being. Does that work for you?
Yes .. and no  :D

I've extended the header unit with dynamic loading and wanted to test.

Which brings me to the second point. I have not installed APL as intended, therefor required to set the library path in order to link (my prerogative as I wanted to start testing with Lazarus).

Quote
If not we three need to synchronize our libraries so we are barking all up to the same tree.. :) O:-)
No problem for testing purpose I can stick to linking.

Quote
btw UERR seems fixed here.
Then you are a greater man then I am, because it does not seem to matter what I do as I am simply unable to shut off the chatter of libapl. For some reason FPC does not seem to pick up the chatter on StdErrorHandle for me (even though I explicitly tell it to: example redir2b is wrong in that regards).
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 19, 2023, 06:47:44 pm
Did you try my reply at #41? Affter that, all the chatter came nicely in Output/stdOut.
It simply makes sure that stdErr goes to stdOut.

Set WriteErrorsToStdErr to False  // this is from unit system
Title: Re: capture apl_exec stdout
Post by: TRon on January 20, 2023, 11:16:28 am
Thaddy:
Yes, I have seen your reply #41

Thank you for that as I also was not aware of the existence of that variable

Unfortunately it did not had the required effect for me (of redirecting the output on handle 2 from terminal if even to oblivion). The library will continue to emit to the terminal for me.

But a second point (not important right now) is that we actually wish for this to be outputted to the error-handle because it seems the way the library communicates.

For some reason FPC fails (refuses ?) to pick things up on the standard error handle.

I have tried multiple approaches now and I seem to fail to replicate what a filestream create does (as shown by Fred vS) differently when compared to duplicating the handle and feed that with a customized textrec to Assignstream. It should behave the same but it seems it does not.

I have to delve deeper into FPC internals and/or brush up my knowledge on what a Filestream and AssignStream actually does (to a handle) in order to spot the difference.
Title: Re: capture apl_exec stdout
Post by: toby on January 26, 2023, 11:07:32 pm
thaddy and tron : looks like you weren't able to figure it out and just left it hanging

isn't this doable from fpc itself by some combo of thandlestream/fpdup/tmemorystream/tstringstream/loadfromstream/tstream.write programming methods?

surely some hero fpc programmer here who knows this stuff can do it :)
Title: Re: capture apl_exec stdout
Post by: toby on January 27, 2023, 12:23:08 am
this is my attempt to convert the fpdup proigran that writes apl_exec output to file to a stream (and then to tstringlist)

anyone can help get it working properly?

Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. program stdouty;
  4.  
  5. //{$mode objfpc}{$H+}
  6.  
  7. {$linklib apl}
  8.  
  9. uses sysutils, classes, baseunix, libaplu;
  10.  
  11. //var fs : tfilestream;
  12. var fs : thandlestream;
  13.     oldoutputhandle : thandle;
  14.     olderrorhandle : thandle;
  15.     i, j : longint;
  16.     e : array[1..4] of longint;
  17.  
  18. begin
  19.  
  20. for i := 1 to 4 do e[i] := 2;
  21.  
  22. // save original handles
  23. oldoutputhandle := fpdup(textrec(stdout).handle);
  24. olderrorhandle  := fpdup(textrec(stderr).handle);
  25.  
  26. //fs := tfilestream.create('stdout.tmp', fmopenreadwrite or fmcreate);
  27. fs := thandlestream.create(stdoutputhandle);
  28.  
  29. // assign output and error handles to a single file
  30. fpdup2(fs.handle, stderrorhandle);
  31. fpdup2(fs.handle, stdoutputhandle);
  32.  
  33. init_libapl(paramstr(0), 0);
  34. apl_exec('"⍳4"');
  35. j := apl_exec('⍳4');
  36. writeln('j : ', j);
  37.  
  38. writeln('fs.position : ', fs.position);
  39. fs.position := 0;
  40. writeln('fs.position : ', fs.position);
  41. /fpc/current/tstream/tstream.html
  42. writeln('fs.size : ', fs.size);
  43.  
  44. writeln(fs.write(e, 4));
  45.  
  46. for i := 1 to 4 do writeln('i : ', i, '  e[', e[i], ']');
  47.  
  48. fs.free;
  49.  
  50. // restore original handles
  51. fpdup2(oldoutputhandle, textrec(stdout).handle);
  52. fpdup2(olderrorhandle , textrec(stderr).handle);
  53.  
  54. //fpunlink('stdout.tmp');
  55.  
  56. end.
  57.  
Title: Re: capture apl_exec stdout
Post by: 440bx on January 27, 2023, 12:28:28 am
@toby,

you should put your code between tags, i.e, [ code = pascal ] <your code goes here> [ / code ]  (without the spaces between [ and ])

HTH.

ETA:

well done :)



Title: Re: capture apl_exec stdout
Post by: TRon on January 27, 2023, 06:44:15 am
thaddy and tron : looks like you weren't able to figure it out and just left it hanging
not leaving it hanging, just requires more in depth research to figure out exactly what is happening internally.

Quote
isn't this doable from fpc itself by some combo of thandlestream/fpdup/tmemorystream/tstringstream/loadfromstream/tstream.write programming methods?
As you have noticed yourself: no. That is not a problem of FPC perse.

Quote
surely some hero fpc programmer here who knows this stuff can do it :)
Things would have been much easier if the library was a proper shared library (and also not tied to the programming language) so that provision for a normal call-back mechanism would have been in place. We/you are trying to fix something that should not have to be fixed in the first place. You can already make use of a sort of callback mechanism with APL but it still will not catch /all/ output from the library.
Title: Re: capture apl_exec stdout
Post by: TRon on January 27, 2023, 06:59:29 am
anyone can help get it working properly?
Just for the record: combining output and error will not help you in communicating properly with the APL library. One channel is used for normal results, the other for indicating that there is an error and another channel that outputs additional information. And then you have your input channel. The latter can be fixed/circumvented with a call-back. Another callback also work for /one/ of the other used channels.
Title: Re: capture apl_exec stdout
Post by: toby on January 27, 2023, 10:18:23 pm

i don't understand your problem with the libapl.so library
please post your problem code so i have something to understand what is a problem

---

this is information i received from apl guys

from a c++ libapl programmer he uses the line
LIBAPL_error execerr = LAE_NO_ERROR;

from the apl dev
'not sure if this helps, but if I remember correctly (I may not) then the main GNU APL output
goes to *stderr* (fd 2) and not to *stdout* (fd 1). The reason is somewhat historic because *stdout* is
buffered by default while *stderr* is not (which caused some issues with *stdout* when used
interactively that did not occur with *stderr*).'

---
Title: Re: capture apl_exec stdout
Post by: Thaddy on January 28, 2023, 09:13:47 am
Hey, it is weekend. Back on it again.
Btw. I would not worry too much about the stdout vs stderr, that is easily solved as per one of my previous posts. I now ran into trouble with my library compile, but that is likely to be my fault. (setback, only on windows, Linux just fine)
But I got the windows keyboard layout working. (progress)
Title: Re: capture apl_exec stdout
Post by: TRon on January 28, 2023, 11:31:43 am
i don't understand your problem with the libapl.so library
As a reminder: I do not have a problem with libapl.so, you did ask for help  ;D

Quote
please post your problem code so i have something to understand what is a problem
I don't like repeating myself: see post #42

.. and try to brush up on your knowledge concerning shared objects.


Quote
from a c++ libapl programmer he uses the line
LIBAPL_error execerr = LAE_NO_ERROR;
That has no bearings on the current issues.

Quote
from the apl dev
I am aware how the library works. You can also do so by using a simple trace.

I said that the communication the library uses is not how shared libraries are suppose to work (not on any platform) and that the library is tight to the programing language it was designed in (the story of any FPC programmer).

I refuse to write C startup/initialization code or write a wrapper in C. That's all  :)
Title: Re: capture apl_exec stdout
Post by: TRon on January 31, 2023, 02:49:23 pm
@toby:
fwiw: I read the same ML's that you use and you are currently confronted with answers that are the typical response and deliberate misinterpretation (apparently APL itself is written in APL and compiled with the APL compiler) of things that you quoted (which consisted of parts of my statements).

Usually one get such answers when you confront someone with their (broken) design philosophy (please don't get me wrong there as I get it, but I detest when someone tries to deflect the issue with providing bogus responses).

The provided answers did clear something up for me though and that is that you can not solve this problem with Free Pascal. the APL shared library is broken by design so you are forced to brush up on your C(++) knowledge. The provided answer that tells you/us to adjust the C source-code / makefile is exactly what I meant with my statement "the story of any FPC programmer"

The explanation from their ML about the call-back mechanism that /is/ in place is nice but I already knew that (apparently you yourself still do not, despite the fact that I had already expanded the library with that functionality and added two examples to show how to use them).

As said earlier in this post: it will still not solve the underlying problem (e.g. there is a reason (as explained) you are forced to use static linking).
Title: Re: capture apl_exec stdout
Post by: TRon on February 04, 2023, 04:31:01 pm
@toby:
With regards to your return value evaluation of exec call problem:
Code: Pascal  [Select][+][-]
  1. var
  2.   global_exec_callback_result : string;
  3.  
  4. function exec_callback(const value: TAPL_value; committed: cint): cint; cdecl;
  5. var temp:pchar;
  6. begin
  7.   global_exec_callback_result := '';
  8.  
  9.   temp := print_value_to_string(value);
  10.  
  11.   // if value has not been committed
  12.   if committed = 0
  13.     then setstring(global_exec_callback_result,temp,strlen(temp));
  14.  
  15.   // do not commit
  16.   result := 0;
  17. end;
  18.  
  19.  
  20. // wrapper function
  21. function apl_exec(apl_line: string; out ret_value: string): Tlibapl_error;
  22. begin
  23.   res_callback:=@exec_callback;
  24.   result:=ulibapl.apl_exec(apl_line);
  25.   ret_value:=global_exec_callback_result;
  26.   res_callback:=nil;
  27. end;
  28.  

edit: You can use it like:
Code: Pascal  [Select][+][-]
  1. var
  2.   err: Tlibapl_error;
  3.   exec_result: string;
  4.  
  5. const
  6.   example = '2*8';
  7.  
  8. begin
  9.   init_libapl(argv[0], 0);
  10.   err := apl_exec(example, exec_result);
  11.   if err = 0
  12.    then writeln('.APL code "', example, '" was interpreted successful: answer = ', exec_result)
  13.    else writeln('.APL code "', example, '" failed miserably');
  14. end.
  15.  

That is what the good Dr. is trying to tell you.
Title: Re: capture apl_exec stdout
Post by: toby on February 07, 2023, 01:20:03 am
with your code as :

i get following compile error

Compiling ulibapl.pas
cb.pas(15,47) Error: Identifier not found "TAPL_value"
cb.pas(15,64) Error: Identifier not found "cint"
cb.pas(15,71) Error: Identifier not found "cint"
cb.pas(19,9) Error: Identifier not found "print_value_to_string"
cb.pas(21,14) Error: Operator is not overloaded: "<erroneous type>" = "ShortInt"
cb.pas(30,1) Error: Identifier not found "res_callback"
cb.pas(33,1) Error: Identifier not found "res_callback"
cb.pas(44) Fatal: There were 7 errors compiling module, stopping

are you using a new ulibapl.pas that i do not have ??




Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. program cb;
  4.  
  5. {$linklib apl}
  6.  
  7. //uses libaplu;
  8. uses ulibapl;
  9.  
  10. var
  11. global_exec_callback_result : string;
  12.  
  13. err: Tlibapl_error;
  14. exec_result: string;
  15. const example = '2*8';
  16.  
  17. function exec_callback(const value: TAPL_value; committed: cint): cint; cdecl;
  18. var temp:pchar;
  19. begin
  20. global_exec_callback_result := '';
  21. temp := print_value_to_string(value);
  22. // if value has not been committed
  23. if committed = 0
  24. then setstring(global_exec_callback_result,temp,strlen(temp));
  25. // do not commit
  26. result := 0;
  27. end;
  28.  
  29. // wrapper function
  30. function apl_exec(apl_line: string; out ret_value: string): Tlibapl_error;
  31. begin
  32. res_callback:=@exec_callback;
  33. result:=ulibapl.apl_exec(apl_line);
  34. ret_value:=global_exec_callback_result;
  35. res_callback:=nil;
  36. end;
  37.  
  38. begin
  39. init_libapl(argv[0], 0);
  40. err := apl_exec(example, exec_result);
  41. if err = 0
  42. then writeln('.APL code "', example, '" was interpreted successful: answer = ', exec_result)
  43. else writeln('.APL code "', example, '" failed miserably');
  44. end.
  45.  
  46.  
Title: Re: capture apl_exec stdout
Post by: TRon on February 07, 2023, 05:18:33 am
i get following compile error
ah yes, craps. Sorry for that.

Quote
are you using a new ulibapl.pas that i do not have ??
Yes, but it is no big change (although I have now almost all functions declared) . Only requires one additional function for the example to work.

Try the following instead, with the ulibapl from post #17.

Code: Pascal  [Select][+][-]
  1. program test;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   ulibapl, ctypes;
  7.  
  8.   // additional function declaration that should be declared inside ulibapl.pas
  9.   // print value into a string
  10.   function  print_value_to_string(const value: TAPL_Value): pchar; cdecl; external apllib;
  11.  
  12.  
  13.   // example code follows:
  14.  
  15. var
  16.   global_exec_callback_result : string;
  17.  
  18.  
  19. function exec_callback(const value: TAPL_value; committed: cint): cint; cdecl;
  20. var temp:pchar;
  21. begin
  22.   global_exec_callback_result := '';
  23.  
  24.   temp := print_value_to_string(value);
  25.  
  26.   // if value has not been committed
  27.   if committed = 0
  28.     then setstring(global_exec_callback_result,temp,strlen(temp));
  29.  
  30.   // do not commit
  31.   result := 0;
  32. end;
  33.  
  34.  
  35. // wrapper function
  36. function apl_exec(apl_line: string; out ret_value: string): Tlibapl_error;
  37. begin
  38.   res_callback:=@exec_callback;
  39.   result:=ulibapl.apl_exec(apl_line);
  40.   ret_value:=global_exec_callback_result;
  41.   res_callback:=nil;
  42. end;
  43.  
  44. var
  45.   err: Tlibapl_error;
  46.   exec_result: string;
  47.  
  48. const
  49.   example = '2*8';
  50.  
  51. begin
  52.   init_libapl(argv[0], 0);
  53.   err := apl_exec(example, exec_result);
  54.   if err = 0
  55.    then writeln('.APL code "', example, '" was interpreted successful: answer = ', exec_result)
  56.    else writeln('.APL code "', example, '" failed miserably');
  57. end.
  58.  
Hopefully that does the trick.
Title: Re: capture apl_exec stdout
Post by: toby on February 07, 2023, 10:57:11 pm
Tron

absolutely fantastic!!

I'm doing some speed testing and large data result testing and will report back

Thanks for this complicated understandng of what was going on


Title: Re: capture apl_exec stdout
Post by: BobDog on February 08, 2023, 12:31:18 am

Windows console program.
catch stdout and stderr in an ansistring.
Code: Pascal  [Select][+][-]
  1.  
  2. {$APPTYPE CONSOLE}
  3. uses
  4. sysutils;
  5.  
  6. function  system(s:pchar):integer ; cdecl external 'msvcrt.dll' name 'system';
  7. function tempnam(n:pchar;f:pchar):pchar ;cdecl external 'msvcrt.dll' name '_tempnam';
  8.  
  9. function filelength(filename:ansistring):longword;
  10. Var F : File Of byte;
  11. var L:longword;
  12. begin
  13. Assign (F,filename);
  14.   Reset (F);
  15.   L:=FileSize(F);
  16.   Close (F);
  17.   exit(L);
  18. end;
  19.  
  20. procedure loadfile(var content: ansistring;filename:ansistring);
  21.    Var Fin : File;
  22.    Var x:longint;
  23.    begin
  24.    x:=filelength(filename);
  25.    setlength(content,x);
  26.    Assign (Fin,filename);
  27.    Reset (Fin,x);
  28.    BlockRead (Fin,content[1],1);
  29.    close(fin);
  30. end;
  31.  
  32.  
  33. function stream(command:ansistring):ansistring;
  34. var
  35. _file:ansistring;
  36. e:integer;
  37. s:ansistring;
  38. begin
  39. _file:=tempnam(nil,'catchstream');
  40. e:=system(pchar(command +' '+' >>'+_file +' 2>&1'));
  41. loadfile(s,_file);
  42. deletefile(_file);
  43. if (FileExists(_file)=true) then writeln('delete '+_file+' manually');
  44. exit(s);
  45. end;
  46.  
  47.  
  48. var
  49. s:ansistring;
  50.  
  51.  
  52. begin
  53. s:=stream('fpc nosuchfile.pas');
  54. writeln(s);
  55.  
  56. s:=stream('gcc -c nosuchfile.c');
  57. writeln(s);
  58. writeln('Press return to end . . .');
  59. readln;
  60. end.
  61.  
Title: Re: capture apl_exec stdout
Post by: toby on February 09, 2023, 10:13:56 pm
Tron

added to modified post
apl_exec_cb('rs1←⍳4', s); //  create rs1
//apl_exec_cb('⍕rs1', s); // good capture in s
//apl_exec_cb('⍞←rs1', s); // not captured in s ??
apl_exec_cb('⍞←⍕rs1', s); // hot captured in s ??
writeln('s : *', s, '*');

modified post info
the ⍕ character is the
Monadic format ⍕B A character representation of B U+2355 ⍕
which shows correctly on post createion using seaamonkey but as a 'bad unicode character' when i look at the post using dillo)

the callbacl method takes 96msec for a 30000 integer -> 89999 byte string test result (my largest current use exanple/need)
vs 26msec when writing the result to a file with apl and then reading it back into an tstringlist
(changing the call back apl_exec to apl_exec_cb did not effect anything time wises)

for reference the fred vs method of writing result to file and reading it back in also only takes 26msec

⎕pw ⍕ problem
this is not inlcuding the time that would be need to be taked to get rid of the #10 added to each ⎕pw defined line (i use ⎕pw←132) which i don't have to worry about with apl file writing method
⎕pw⊢80 gives 1250 lines vs ⎕pw←132 giving 715 lines
(i would have to use stringreplace( to remove the #10 chars
using ⍕ gives only 1 line - i am still playing with gettng the ⍕ to work with the call back but nothing so far.

(using ⍕ has no effect for some reason with the call back method as it doen with both file writing methods so there would be no ⎕pw line dependent #10 added

because of the 3x time increase i renamed your apl_exec call back function apl_exec(example, s) to apl_exec_cb(example. s)
and 'added' back in the original apl_exec for apl use and only use the apl_exec_cb callback for actualy capture of apl_exec output wnen needed.

also of major importance is that your apl_exec_cb code does not negatively impact using it with )copy command
i know i should use apl_command but i have automated things and testing every command in my command file for the one )copy command would be a waste of programming time

again thank you for you programming work here - you sure made it clean that the libapl coding needs rewriting but i fear the original libapl dev is long gone

this has been fun coding for me indeed
Title: Re: capture apl_exec stdout
Post by: TRon on February 10, 2023, 11:38:50 am
catch stdout and stderr in an ansistring.
Close but no cigar  :)

Thank you for your effort BobDog but the issue is about catching your own program's I/O. Or to be more precise: the I/O from a library that is used in your program. And it already works (for stdout) except that it seems impossible to do the same for stderr.
Title: Re: capture apl_exec stdout
Post by: TRon on February 10, 2023, 11:45:59 am
//apl_exec_cb('⍞←rs1', s); // not captured in s ??
apl_exec_cb('⍞←⍕rs1', s); // hot captured in s ??
Those are using an immediate input request. For that you need to use/install a callback with install_get_line_from_user_cb.

I've added functionality for that to reply #17 on jan 17th, see also example apl_test2b.pas

You are on your own with regards to the different input modes as I have no clue how the different modes should behave (I simply lack the time to investigate as documentation for apl for/on certain topics is very scarce).

Quote
the ⍕ character is the Monadic format ⍕B A character representation of B U+2355 ⍕

the callbacl method takes 96msec for a 30000 integer -> 89999 byte string test result (my largest current use exanple/need) vs 26msec when writing the result to a file with apl and then reading it back into an tstringlist
Can't comment on that a.t.m. other then If you need a different storing method for the result (f.e. stringlist) then you can create your own callback wrapper function.

I have no idea w.r.t. the time(penalty). You should be looking at direct library function calls, see also below.

Quote
⎕pw ⍕ problem
Are you referring to the plot window functionality ? (e.g. apl is non-case sensitive so that pw = PW ?)

Quote
this is not inlcuding the time that would be need to be taked to get rid of the #10 added
Although there is little documentation w.r.t. behaviour on certain functionality /this/ is at least a topic that is present in the docs  :)

Have a look at CR. It is advised to use that in combination with certain function(s)/(ality)

There is an example in the official documentation that removes them i.c.w. XML.

Quote
because of the 3x time increase i renamed your apl_exec call back function apl_exec(example, s) to apl_exec_cb(example. s) and 'added' back in the original apl_exec for apl use and only use the apl_exec_cb callback for actualy capture of apl_exec output wnen needed.
In that regards you can do whatever you please :) It is encouraged to write your own additions/correction. Only then things can improve

Just keep in mind that when you run into problems that you post the complete implementation that you used because how you implement things can influence behaviour (which we cannot replicate without using the same implementation).

I've named it the way I did for convenience because it is an overloaded function, which keeps things easy for the end-user.

Also note that I set and remove the callback for each call to apl_exec because you can install a different callback based on what functionality you require (and that can depend on which function you are actually executing).

So if you need other functionality then go ahead and add it to your own custom callback routine and name it whatever floats you boat  :)

Quote
also of major importance is that your apl_exec_cb code does not negatively impact using it with )copy command
You should have a better/closer look at the other functionality that the apl library exposes.

I simply lack the time to delve deeper into things but am able to tell you that you can call functions directly as well as access values using direct calls to functions.

I'm a bit stuck with regards to the loc parameter stuff. Seems highly undocumented so would need to delve into their source-code to see how that should be used correctly.

Also I am unsure about how custom functions affect the different workbook environments when using direct library calls.

If you are interested in apl then you should experiment with those as well. I have added them in my local copy of ulibapl.

My knowledge about the apl language/library is pretty limited so am more often not able to tell if the experienced behaviour is as expected.

regards,
Title: Re: capture apl_exec stdout
Post by: toby on February 11, 2023, 10:01:20 pm
Hi Tron

there really is no problem really with anything you did - i was just getting into the weeds

i have gotten the call back time decreased by doing more inline apl coding

so this is great - no problem with any apl functions or apl commands and all my code runs as expected

see Dr post where use of ⍕ for apl line submitted to the call back is needed - i have just included a ⍕ before the apl_line before submitted to the call back

---

i took care of my ⎕pw problem

the ⎕ and ⍞  apl input/output methods with ⍕ are use dependent
with ⎕pw they are like
writeln vs write with an extra writeln thrown in after the ⎕pw number of chars

var i, j : integer;

begin

for i := 1 to 9 do // ⎕pw := 9
  begin
  for j := 1 to 9 do write(i, j, ' ');
  writeln; //
  end;

end.

if you do do any additional programming to your ulibapl.pas could you please add it as a new reply to this forum/topic thread instead of as a modified post?  thanks

it is easier to check for new replies rather than if a post has been modified

for very very large captures the time taken tapers off - there must be a minimum time overhead to it ?

you never mentioned it but did you have any problems using the apl keyboard/fonts ?
i added the following to an    apl.xodmap file so with xmodmap all the apl key combo are on the right alt key and the everything programmed for fluxbox and dillo and .... etc are left alone on the left alt key
keycode 113 = Mode_switch ISO_Level3_Shift Multi_key
Title: Re: capture apl_exec stdout
Post by: BobDog on February 15, 2023, 12:08:14 am
catch stdout and stderr in an ansistring.
Close but no cigar  :)

Thank you for your effort BobDog but the issue is about catching your own program's I/O. Or to be more precise: the I/O from a library that is used in your program. And it already works (for stdout) except that it seems impossible to do the same for stderr.
I can catch output from a pascal file along with errors, hints, or any other information needed, both from stdout and stderr using my previous code.
I really could use a cigar but I have quit smoking, doctor's orders.
Example
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3.  
  4. {$APPTYPE CONSOLE}
  5. uses
  6. sysutils;
  7.  
  8. function  system(s:pchar):integer ; cdecl external 'msvcrt.dll' name 'system';
  9. function tempnam(n:pchar;f:pchar):pchar ;cdecl external 'msvcrt.dll' name '_tempnam';
  10.  
  11. function filelength(filename:ansistring):longword;
  12. Var F : File Of byte;
  13. var L:longword;
  14. begin
  15. Assign (F,filename);
  16.   Reset (F);
  17.   L:=FileSize(F);
  18.   Close (F);
  19.   exit(L);
  20. end;
  21.  
  22. procedure loadfile(var content: ansistring;filename:ansistring);
  23.    Var Fin : File;
  24.    Var x:longint;
  25.    begin
  26.    x:=filelength(filename);
  27.    setlength(content,x);
  28.    Assign (Fin,filename);
  29.    Reset (Fin,x);
  30.    BlockRead (Fin,content[1],1);
  31.    close(fin);
  32. end;
  33.  
  34. procedure savefile(s:ansistring ;filename:ansistring);
  35.     var
  36.     fout:file;
  37.     begin
  38.     Assign(fout,filename);
  39.     Rewrite(fout,length(s));
  40.     blockwrite(fout,s[1],1);
  41.     close(fout);
  42.   end;
  43.  
  44.  
  45. function stream(command:ansistring):ansistring;
  46. var
  47. _file:ansistring;
  48. e:integer;
  49. s:ansistring='';
  50. begin
  51. _file:=tempnam(nil,'catchstream');
  52. e:=system(pchar(command +' '+' >>'+_file +' 2>&1'));
  53. loadfile(s,_file);
  54. deletefile(_file);
  55. if (FileExists(_file)=true) then writeln('delete '+_file+' manually');
  56. exit(s);
  57. end;
  58.  
  59.  
  60. var
  61. s:ansistring;
  62. txt:ansistring;
  63.  
  64.  
  65.  
  66.  
  67. begin
  68. txt:='{$APPTYPE CONSOLE}'+#10;
  69. txt:=txt+'var'+#10;
  70. txt:=txt+'i,j:integer;'+#10;
  71. txt:=txt+'begin'+#10;
  72. txt:=txt+'for i:=1 to 20 do writeln(i);'+#10;
  73. txt:=txt+'end.'+#10;
  74.  
  75. savefile(txt+chr(10),'vtemp.pas');
  76.  
  77. s:=stream('type vtemp.pas');
  78. s:=s+stream('fpc -vwnhi vtemp.pas');
  79. s:=s+stream('vtemp.exe');
  80.  
  81. savefile(s,'results.txt');
  82.  
  83. system('notepad.exe results.txt');
  84. writeln('Press return to end . . .');
  85. readln;
  86. deletefile('vtemp.pas');
  87. deletefile('vtemp.exe');
  88. end.
  89.  
Title: Re: capture apl_exec stdout
Post by: TRon on February 15, 2023, 01:08:18 am
I can catch output from a pascal file along with errors, hints, or any other information needed, both from stdout and stderr using my previous code.
The code that you showed (thank you for the example) is executing an external program and catches that program's error/output, not its own error and output. You also combined out and err which is a no-go for using this 3th party library. TS also expressed not wanting to use external files.

All the above was already mentioned in this thread.

fwiw: Your code looks overly complicated to me for something that can be done in a cross-platform manner with runcommand (https://www.freepascal.org/docs-html/fcl/process/runcommand.html)

Quote
I really could use a cigar but I have quit smoking, doctor's orders.
Smart doctor, smart patient  :)
Title: Re: capture apl_exec stdout
Post by: TRon on February 15, 2023, 08:35:43 am
Just a quick note, i'll try respond to your other remarks later.

if you do do any additional programming to your ulibapl.pas could you please add it as a new reply to this forum/topic thread instead of as a modified post?  thanks
I have no problem posting my additions and will do so later. It is just that my code includes dynamic loading which isn't supported by the library so I have to remove that and i haven't had the time to do that yet.

Quote
it is easier to check for new replies rather than if a post has been modified
Actually I disagree on that matter. That way I would have to keep track of every post with a published zip in the thread to make sure old attachments are removed. As things progresses, so  will the length of the thread so that people have to read through the whole thread to be able to locate the latest attachment. imo keeping old attachments alive works counterproductive as people tend to download old versions then ask about issues that are already solved in a later version and keeping old attachments will needlessly waste forum space.

Thus I believe it is more productive/maintainable to post whenever a new update is available and point to the post that includes the (replaced and updated) attachment. Ideally that would be the first post of the thread but alas that ship has sailed  :D


Title: Re: capture apl_exec stdout
Post by: TRon on February 16, 2023, 04:05:38 pm
Update post #17 (https://forum.lazarus.freepascal.org/index.php/topic,61841.msg466957.html#msg466957) and attached version v3.

I have added a license which I deemed to be satisfactory (pay it forward) but since the package also contains work from others and/or is based on suggestion/examples made by others from this forum i have to take into consideration that someone would object to that.

In such case please let me know so I can rectify anything you do not agree with/to.

Mentioned users in package:
- Thaddy (for his contributions/ideas)
- FredVs (for his ideas/examples)
- Engkin (for his piping example code somewhere on this forum which inspired me how a solution for libapl could be implemented)
Title: Re: capture apl_exec stdout
Post by: toby on February 16, 2023, 08:18:31 pm
Tron

looks like you have no problem taking my original libapl work and appropriating it as your own

pretty sad
Title: Re: capture apl_exec stdout
Post by: TRon on February 16, 2023, 09:57:02 pm
Tron

looks like you have no problem taking my original libapl work and appropriating it as your own

pretty sad
Title: Re: capture apl_exec stdout
Post by: TRon on February 16, 2023, 09:59:56 pm
Quote
APL Library itself published under the GNU General Public License
Title: Re: capture apl_exec stdout
Post by: TRon on February 16, 2023, 10:01:35 pm
Quote
All Pascal related files in this archive are licensed as "Pay it forward"
Title: Re: capture apl_exec stdout
Post by: TRon on February 17, 2023, 01:34:05 pm
Tron

looks like you have no problem taking my original libapl work and appropriating it as your own

pretty sad
@toby:

Please accept my deepest apologies as I humbly ask you to forgive me for my rude manners in copy-pasting all your hard work and claim it as my own (wherever I supposedly did that, thus please enlighten me).

Would you be so kind and point me to the work that you designated as your "original libapl work" so that I am able to rectify my blameworthy behaviour ?

As a (demonstrable) reminder: All my work, with the exception of users mentioned in post #68 (https://forum.lazarus.freepascal.org/index.php/topic,61841.msg471271.html#msg471271) (as well as in the source-files) is based on publications from libapl and/or is actually my own.
Title: Re: capture apl_exec stdout
Post by: toby on February 17, 2023, 10:42:03 pm
you want to show me where a libaplu.pas was in any apl publications or online??
there was NO libaplu.pas until i wrote it and submitted it as reply #6 to this topic

NOTHING you did has increased the usable functionality of my libaplu.pas - NOTHING
nothing you did in your ulibapl.pas is needed to get the most complex apl/apl_exec statements running - NOTHING

in fact i have been able to bring the results of an apl_exec statement into an fpc ansistring and tstinglist for years

your call back increases the time to get the rsults into a tstiringlist and ansistring to unacceptable amounts

---

timing testing :  run on your system to verify - use your apl_exec callback     compare for relative times from my system to yours
also note the bug in the results your callback and fpdup methods produce

Date Friday 2/17/2023 Time 13:58.46.298         20 msec to create rs with 1 line of 900000 chars
rs←900000↑"ab "
⍴rs
900000
Date Friday 2/17/2023 Time 13:58.46.318

Date Friday 2/17/2023 Time 13:58.46.318
soa rs                      45 msec for my apl fns to bring rs into an fpc tstringlist and then ansistring s
astringlist.count : 1       correct nymber of lines of 900000 chars
s := astringlist.text;
length(s) : 900000          correct number of chars
Date Friday 2/17/2023 Time 13:58.46.363


Date Friday 2/17/2023 Time 13:58.46.363
so rs                      10 seconds 239 msec for fpdup2 method to bring rs into an foc tstringlist and then ansistring s
astringlist.count : 12163        it should be 1 line of chars not 12163 lines
length(s) : 985134               it should be 900000 chars
Date Friday 2/17/2023 Time 13:58.56.602


Date Friday 2/17/2023 Time 13:58.56.602
soc rs          11 seconds 862 msec fpr your call back code to bring rs into an fpc tsringslit and then ansistring s
astringlist.count : 12163        it should be 1 line of chars not 12163
length(s) : 985135               it should be 900000 chars
Date Friday 2/17/2023 Time 13:59.8.482

run bringing the rs apl variable into an fpc taringlist and assistring from the fpdup and call back methods on your system and post the timing results


your problem is even more pronouned with a simple
rs←⍕3 3000⍴⍳9
which should resukt in rs being a 3x3000 array     3 lines of 3000 '9'
your code and the fpdup code produce result of 243 lines and different lengths if s
in order to fix your code to give correct reaults the time was increased from and already unacceptable time




TinyPortal © 2005-2018