Recent

Author Topic: Problems with Crt and Dos.Exec  (Read 1061 times)

pleumann

  • Jr. Member
  • **
  • Posts: 94
Problems with Crt and Dos.Exec
« on: May 27, 2022, 09:52:34 am »
Hello,

I ran into a little problem yesterday. It's possible my expectations are wrong, so please bear with me.

I have a command line program that calls other command line programs via Dos.Exec and expects their output to "blend in" with its own. That worked fine until I added the Crt unit to my project because I needed ReadKey functionality. Now the output of the "subprograms" moves to the right on each new line (I am on Mac, so programs just send #10), i.e. it lacks the automatic carriage return. Basically like this:

Code: [Select]
stdout #0
         stdout #1
                  stdout #2
                           stderr #0
                                    stderr #1
                                             stderr #2

Is this an expected consequence of using Crt? I remember that one benefit of Crt on MS-DOS used to be direct screen access, resulting in improved speed. Is this still what it does in a modern Unix-like world or does it act mostly like a terminal wrapper? Can it be influenced somehow to get the non-Crt behavior?

As an alternative, is there a simple and portable way of doing ReadKey without Crt? I found a unit called Keyboard, but it seems to have the same effect.

Best regards
Joerg

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Problems with Crt and Dos.Exec
« Reply #1 on: May 27, 2022, 10:46:02 am »
The reason for this is that CRT switches your terminal to what I call "direct read" mode (on unix this is called raw mode and an windows it does not have a name), where chars are read directly and not linewise buffered (a prequesite for readkey). The problem with this is, that this changes the behavior of the terminal with respect to output. One of these changes is that now the terminal requires both control chars carriage return (#13) and linefeed (#10) to perform a newline operation (as now you need two steps, first move the cursor to the beginning of the line and second move it down one line).

If you need a readkey like functionality, there is not much way around this behavior, but you don't need the direct read mode to be enabled all the time. What you could do is to disable this mode for the execution of the other processes and reenable it when you need to read the key.

You can either do this directly via the respective APIS (SetConsoleMode in Windows and termio in Unix), but this might completely break your I/O with crt (crt does a lot of stuff in the background, and alters your terminal behavior completely), or you could use my LazTermUtils to do so:
Code: Pascal  [Select][+][-]
  1. Term.Input.DirectRead := True;
  2. // ...
  3. key := Term.Input.ReadKey;
  4. // do something with key
  5. // ...
  6. Term.Input.DirectRead := False;
  7. // Exec external program
  8. Term.Input.DirectRead := True;
  9. //...

pleumann

  • Jr. Member
  • **
  • Posts: 94
Re: Problems with Crt and Dos.Exec
« Reply #2 on: May 27, 2022, 11:06:53 am »
The reason for this is that CRT switches your terminal to what I call "direct read" mode (on unix this is called raw mode and an windows it does not have a name), where chars are read directly and not linewise buffered (a prequesite for readkey). The problem with this is, that this changes the behavior of the terminal with respect to output. One of these changes is that now the terminal requires both control chars carriage return (#13) and linefeed (#10) to perform a newline operation (as now you need two steps, first move the cursor to the beginning of the line and second move it down one line).

If you need a readkey like functionality, there is not much way around this behavior, but you don't need the direct read mode to be enabled all the time. What you could do is to disable this mode for the execution of the other processes and reenable it when you need to read the key.

You can either do this directly via the respective APIS (SetConsoleMode in Windows and termio in Unix), but this might completely break your I/O with crt (crt does a lot of stuff in the background, and alters your terminal behavior completely), or you could use my LazTermUtils to do so:
Code: Pascal  [Select][+][-]
  1. Term.Input.DirectRead := True;
  2. // ...
  3. key := Term.Input.ReadKey;
  4. // do something with key
  5. // ...
  6. Term.Input.DirectRead := False;
  7. // Exec external program
  8. Term.Input.DirectRead := True;
  9. //...

Thanks for the answer and the pointer to LazTermUtils! This would add some dependencies (such as SysUtils) that I would like to avoid at this point. But I understand the problem a bit better now. I wonder if for my case I could solve it the other way round: Enable raw mode only for ReadKey (or an equivalent), then disable it again. Are tcgetattr and tcsetattr available and could I apply them directly to Input? Would Read() be able to deal with that?

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Problems with Crt and Dos.Exec
« Reply #3 on: May 27, 2022, 11:11:57 am »
Thanks for the answer and the pointer to LazTermUtils! This would add some dependencies (such as SysUtils) that I would like to avoid at this point. But I understand the problem a bit better now. I wonder if for my case I could solve it the other way round: Enable raw mode only for ReadKey (or an equivalent), then disable it again. Are tcgetattr and tcsetattr available and could I apply them directly to Input? Would Read() be able to deal with that?

I'm reluctant to get involved here since OP has failed to give us any info at all on what OS he's running etc.: even IBM mainframes claim to be "unix-like" these days.

The problem is commonly referred to as staircasing, and can be fixed from the shell prompt using "stty sane" etc. There's examples of doing it under program control at the link below.

https://forum.lazarus.freepascal.org/index.php/topic,58709.msg438286.html#msg438286

Without MUCH more information including example code illustrating the problem etc. that's my final word.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Problems with Crt and Dos.Exec
« Reply #4 on: May 27, 2022, 11:29:14 am »
Thanks for the answer and the pointer to LazTermUtils! This would add some dependencies (such as SysUtils) that I would like to avoid at this point. But I understand the problem a bit better now. I wonder if for my case I could solve it the other way round: Enable raw mode only for ReadKey (or an equivalent), then disable it again. Are tcgetattr and tcsetattr available and could I apply them directly to Input? Would Read() be able to deal with that?
So TCSetAttr, CFMakeRaw and TCGetAttr are available in the termio unit. I don't know if read can directly handle them because Read does a lot of stuff behind the scenes and I try to avoid it. What defenitely works is to use the system api fpread(StdInputHandle, @c, 1); to (wait for and) read exactly 1 char from stdin.

That said, a keypress is not necessarily one char. Thanks to unicode, you can have multiple (8-bit)chars encoding one UTF-8 char, or you can also receive escape sequences like ctrl+char or ctrl+shift+char, which are reperesented using string sequences of multiple chars.
For sequences you can look at my readsequence function. UTF-8 is another thing I haven't implemented for readkey in LazTermUtils yet and so I can't help you there that much

pleumann

  • Jr. Member
  • **
  • Posts: 94
Re: Problems with Crt and Dos.Exec
« Reply #5 on: May 27, 2022, 11:30:49 am »
Thanks for the answer and the pointer to LazTermUtils! This would add some dependencies (such as SysUtils) that I would like to avoid at this point. But I understand the problem a bit better now. I wonder if for my case I could solve it the other way round: Enable raw mode only for ReadKey (or an equivalent), then disable it again. Are tcgetattr and tcsetattr available and could I apply them directly to Input? Would Read() be able to deal with that?

I'm reluctant to get involved here since OP has failed to give us any info at all on what OS he's running etc.: even IBM mainframes claim to be "unix-like" these days.

The problem is commonly referred to as staircasing, and can be fixed from the shell prompt using "stty sane" etc. There's examples of doing it under program control at the link below.

https://forum.lazarus.freepascal.org/index.php/topic,58709.msg438286.html#msg438286

Without MUCH more information including example code illustrating the problem etc. that's my final word.

MarkMLl

Sorry for not being specific enough. I'm working on MacOS most of the time, but would like to have a portable solution. This is the compiler project discussed a few days ago.

I think I was able to adapt the example from here

https://www.freepascal.org/docs-html/rtl/keyboard/getkeyevent.html

to my case. I now have this code and it seems to work reasonably well:

Code: Pascal  [Select][+][-]
  1. function GetKey: Char;
  2. var
  3.   C: Char;
  4.   K: TKeyEvent;
  5. begin
  6.   InitKeyboard;
  7.   Write('>');
  8.  
  9.   Repeat
  10.     K:=GetKeyEvent;
  11.     K:=TranslateKeyEvent(K);
  12.   until GetKeyEventFlags(K) = kbASCII;
  13.  
  14.   C := GetKeyEventChar(K);
  15.   WriteLn(C);
  16.   GetKey := C;
  17.   DoneKeyboard;
  18. end;
  19.  

Does anyone see a downside of doing that (especially the repeated InitKeyboard/DoneKeyboard)? I like that it isolates the functionality in a single function and doesn't add dependencies. It also seems to be portable. I will need only a few ASCII keys and am willing to deal with function keys or Unicode "manually" if that comes up at some point.
« Last Edit: May 27, 2022, 11:34:28 am by pleumann »

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Problems with Crt and Dos.Exec
« Reply #6 on: May 27, 2022, 11:48:41 am »
Sorry, forgot it was you who was doing the compiler (no harm in starting a new question with "following on from my discussion of writing a compiler... :-)

I think the major downside of using crt etc., or any sort of direct termios manipulation, is that you risk making input and output incompatible with redirection. It might not sound like much, but that will bite e.g. when paging --help output or when trying to filter maximum-verbosity debugging output.

If you do start trying clever stuff, note my comment in the thread I cited about the unix API's select() call varying between the BSD family- which I think includes Macs- and Linux.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

pleumann

  • Jr. Member
  • **
  • Posts: 94
Re: Problems with Crt and Dos.Exec
« Reply #7 on: May 27, 2022, 11:58:17 am »
Sorry, forgot it was you who was doing the compiler (no harm in starting a new question with "following on from my discussion of writing a compiler... :-)

You're right. Will keep that in mind for the next one. :)

Quote
I think the major downside of using crt etc., or any sort of direct termios manipulation, is that you risk making input and output incompatible with redirection. It might not sound like much, but that will bite e.g. when paging --help output or when trying to filter maximum-verbosity debugging output.

If you do start trying clever stuff, note my comment in the thread I cited about the unix API's select() call varying between the BSD family- which I think includes Macs- and Linux.

MarkMLl

Yes, I see that point. My compiler is primarily supposed to be a well-behaved citizen of the command line, i.e. play nice with grep/less/etc. I just added - purely for fun - an interactive mode that would give you a simple TP3-like UI. The only special requirement here is being able to read the keyboard. If I limit that to my GetKey function and that function is only ever be used in interactive mode (which should not be combined with grep/less/etc.) , would I be fine? Or could the InitKeyboard/DoneKeyboard still have consequences for the standard file handles that get me into trouble elsewhere (if that question makes sense)?

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Problems with Crt and Dos.Exec
« Reply #8 on: May 27, 2022, 12:15:45 pm »
Yes, I see that point. My compiler is primarily supposed to be a well-behaved citizen of the command line, i.e. play nice with grep/less/etc. I just added - purely for fun - an interactive mode that would give you a simple TP3-like UI. The only special requirement here is being able to read the keyboard. If I limit that to my GetKey function and that function is only ever be used in interactive mode (which should not be combined with grep/less/etc.) , would I be fine? Or could the InitKeyboard/DoneKeyboard still have consequences for the standard file handles that get me into trouble elsewhere (if that question makes sense)?

One possibility there would be to change what is currently the compiler's program unit into a function that can be called by a wrapper. Then that wrapper could either be trivial if it's a command-line program (it would be a good place to parse the commandline so that you could handle unix- and DOS-style parameter expectations) or somewhat chunkier if it incorporated an editor, with variants for GUI and TUI.

I've looked in the past at trying to have a program (a) decide whether it had redirected I/O and (b) "do the right thing" depending on whether there was pending input, and it got complicated. I've also got a live program which attempts the "CAD trick" of accepting typed commands and using them in parallel with GUI events (pane size changes, abort button clicks and so on) and that got particularly painful.

I'm moderately confident that the approach I used in the link I gave earlier is fairly benign since it only uses the preallocated stdin handle and if that's a redirected file the termios calls will fail harmlessly (they're trying to fix a problem that can't occur). The potential nasty is that fpSelect() if cumulative timeouts are attempted, which I can't test on a Mac.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Problems with Crt and Dos.Exec
« Reply #9 on: May 27, 2022, 12:26:48 pm »
About the piping thing, for this the libc (wrapped in fpcs termio unit) provides the function isatty, which lets you check for any file handle if it is connected to a terminal, or not (i.e. if it is a file or a pipe).

So you could do either restrict interactive mode just for ttys (or at least show some warning if trying to use interactive mode without a tty like vim does), or in case it is not a tty, then you can use read to read single chars without having any setup, because then it's the piping program that is controlling the buffering and not the terminal.
It's actually one of the things I was designing my LazTermUtils library around, to also work with files (even directly, you can say in the constructor to just use files as I/O target)

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Problems with Crt and Dos.Exec
« Reply #10 on: May 27, 2022, 12:53:04 pm »
About the piping thing, for this the libc (wrapped in fpcs termio unit) provides the function isatty, which lets you check for any file handle if it is connected to a terminal, or not (i.e. if it is a file or a pipe).

I'm afraid that I can't remember the detail, but I've definitely found cases where that wasn't an adequate check.

Looking very briefly at some old code, these were the cases I was trying to handle:

Code: Pascal  [Select][+][-]
  1. (* There are four possibilities here, handled in this order:            *)
  2. (*                                                                      *)
  3. (* 1)   No input: adopt quiescent state.                                *)
  4. (* 2)   Piped input: read to EOF.                                       *)
  5. (* 3)   File input: read file, (4) on error.                            *)
  6. (* 4)   Directory input: display file select dialogue, exit on error.   *)
  7. (*                                                                      *)
  8. (* Of these (4) changes the current directory if the parameter is valid *)
  9. (* while the remainder stay in the original working directory. This is  *)
  10. (* the "right thing" since if the directory isn't changed a list- or    *)
  11. (* execute-selected command applied to an unqualified filename won't    *)
  12. (* work as expected.                                                    *)
  13.  

This was a file-display program rather than an editor, and in any of those cases it was expected to behave like a standard GUI program after it had handled initial input.

I've also got stuff somewhere which handles a combination of (syntax definition) files from the command line, then any sequence of source files and redirected input. And recently I've been comparing the behaviour of pipes, FIFOs and unix-domain sockets.

There's really no alternative to enumerating all possible cases and then testing them... :-/

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Fred vS

  • Hero Member
  • *****
  • Posts: 3158
    • StrumPract is the musicians best friend
Re: Problems with Crt and Dos.Exec
« Reply #11 on: May 27, 2022, 01:10:16 pm »
Hello.

I did also have problems with crt unit.
The solution was to use keyboard unit instead of crt.

See here (and previous post): https://forum.lazarus.freepascal.org/index.php/topic,57284.msg426272.html#msg426272

Fre;D
I use Lazarus 2.2.0 32/64 and FPC 3.2.2 32/64 on Debian 11 64 bit, Windows 10, Windows 7 32/64, Windows XP 32,  FreeBSD 64.
Widgetset: fpGUI, MSEgui, Win32, GTK2, Qt.

https://github.com/fredvs
https://gitlab.com/fredvs
https://codeberg.org/fredvs

pleumann

  • Jr. Member
  • **
  • Posts: 94
Re: Problems with Crt and Dos.Exec
« Reply #12 on: May 27, 2022, 01:51:02 pm »
Hello.

I did also have problems with crt unit.
The solution was to use keyboard unit instead of crt.

See here (and previous post): https://forum.lazarus.freepascal.org/index.php/topic,57284.msg426272.html#msg426272

Fre;D

That looks pretty much like the solution I came up with (InitKeyboard/DoneKeyboard on the fly). Thanks for confirming this works!

Thanks also to MarkMLI and Warfley. I'll keep the current solution until I run into further trouble (and look into checking IsATTTY and related things in parallel).

Best regards
Joerg
« Last Edit: May 27, 2022, 05:20:29 pm by pleumann »

 

TinyPortal © 2005-2018