Lazarus

Free Pascal => General => Topic started by: big_M on October 27, 2021, 12:21:26 pm

Title: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 12:21:26 pm
Hi. Not sure if this is a bug of if I am missing something: To show the status of a calculation I want to repeatedly write over the same line in the terminal. To to that I use crt with the GotoXy function, however the result is buggy. Sometimes it works as expected, sometimes it doesn't. Here a test program:

Code: Pascal  [Select][+][-]
  1. program Repeat_WriteLn;
  2.  
  3. uses SysUtils, crt;
  4.  
  5. var i:integer;
  6.  
  7. begin
  8.  
  9.   for i:=1 to 1000 do
  10.   begin
  11.     writeLn('Line'+inttostr(i));
  12.   end;
  13.  
  14.   for i:=1 to 10000 do
  15.   begin
  16.     //GoToXy(lo(WindMin)+1,(hi(WindMax)));
  17.     GoToXy(WhereX,WhereY-1);
  18.     DelLine;
  19.     writeLn(inttostr(i));
  20.     sleep(100);
  21.   end;
  22. end.      

When running like this I expect it to go to line 1000 and overwrite it. However it goes somewhere to around the middle of the currently visible lines (depending on the size of the actual window of the terminal), or it creates new lines instead when the window is very small. So the result looks very erratic and (partly?) depends on the size of the terminal window. I tried this with xTerm and with Konsole with the same result. Sometimes however it also just works as expected (not with this test program so far, but in my actual program where I use this)

Any idea?
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 12:39:54 pm
I don't think you can safely make any assumptions about positions larger than the konsole (etc.) window, and in practice you might find that resizing the window while it's running will give problems. (There's a unix-only signal when resizes happen, I don't believe that there's anything comparable for other OSes running text-mode programs).

The best workaround would probably be to move your program over to Lazarus and to output to a memo or grid... there are a few gotchas involved with keeping what you're looking at visible if extra lines are added etc., but by and large it works. You might find that you have to change things so that your computation starts on a button press or similar rather than from the program's main block, and you'll need to insert the /occasional/ Application.ProcessMessages in tight loops to make sure the GUI stays responsive.

The main reason that I'm posting this early with a bit of handwaving is to warn you that this is definitely an either/or situation: you either tinker with the Crt unit (which doesn't get much love these days) or you move to a GUI... don't try to mix them.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 01:18:02 pm
Ah, too bad. Thanks. You see a lot of linux programs that do this so I thought it should be easily doable. I guess that is a shortcoming then of pascal traditionally being very Windows oriented.

No, I can't move it to a GUI because the program should also be able to be executed by a server. For that purpose the status update would not be important anyway, but it would be nice to have when run manually. I think I will just leave it out then, unless someone knows an easy workaround.
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: ojz0r on October 27, 2021, 01:30:08 pm
Maybe try something like this (just a speculation):

Code: Pascal  [Select][+][-]
  1. for i:=1 to 1000 do
  2. begin
  3.         GotoXY(1, i);
  4.         write('Line'+inttostr(i));
  5. end;
  6.  
  7. for i:=1 to 10000 do
  8. begin
  9.         GoToXy(1, WhereY);
  10.         write('                               '); // blank string to clear previous entry
  11.         GoToXy(1, WhereY);
  12.         write(inttostr(i));
  13.         sleep(100);
  14. end;
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: trev on October 27, 2021, 01:30:42 pm
Any idea?

Clear Screen each time?
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 01:48:51 pm
Ah, too bad. Thanks. You see a lot of linux programs that do this so I thought it should be easily doable. I guess that is a shortcoming then of pascal traditionally being very Windows oriented.

Speaking as a long-term Linux user, I can't immediately think of something that will update (say) line 1000 live unless the output is being handled with some sort of background paging buffer. I'd point in particular at top, which either redraws its entire output or updates the top one or two lines. I have to admit that I've never explored whether the escape sequences that control this sort of thing have a documented numeric limit... traditional terminals hence traditional escape sequences (in termcap/terminfo etc.) went up to around 25 lines plus sometimes an extra non-scrolling status line.

As for Pascal being Windows-oriented: that applies to the Delphi lineage obviously, but the FPC developers lean towards unix (predominantly Linux these days).

Quote
No, I can't move it to a GUI because the program should also be able to be executed by a server. For that purpose the status update would not be important anyway, but it would be nice to have when run manually. I think I will just leave it out then, unless someone knows an easy workaround.

Depends. You can attach to a background session using something like VLC, or you can run it using ssh with the graphical output tunelled. But for something like this you might be better off sending your status output to a socket, or in any event not updating your output until e.g. USR1 is sent to the process.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: ojz0r on October 27, 2021, 02:06:26 pm
Okay i just tried my original code and that was a bit buggy..
However i tried it like this and it seems to work in my terminal atleast:

Code: Pascal  [Select][+][-]
  1. program main;
  2.  
  3. uses
  4.         crt,
  5.         sysutils;
  6.  
  7. var
  8.         i: integer;
  9.  
  10. begin
  11.         for i:=1 to 1000 do
  12.                 begin
  13.                 GotoXY(1, i);
  14.                 writeln('Line'+inttostr(i));
  15.         end;
  16.  
  17.         GoToXy(1, WhereY-1);
  18.        
  19.         for i:=1 to 1000 do
  20.         begin
  21.                 GoToXy(1, WhereY);
  22.                 write('                               '); // blank string to clear previous entry
  23.                 GoToXy(1, WhereY);
  24.                 write(inttostr(i));
  25.                 sleep(10);
  26.         end;
  27.         clrscr;
  28. end.

Ps. I changed the sleep and second for loop max value to limit the run time for testing.
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 02:43:10 pm
Okay i just tried my original code and that was a bit buggy..
However i tried it like this and it seems to work in my terminal atleast:
Oh, thanks a lot ojz0r, this runs perfectly! (not really sure why it runs differently at all though)

Speaking as a long-term Linux user, I can't immediately think of something that will update (say) line 1000 live unless the output is being handled with some sort of background paging buffer. I'd point in particular at top, which either redraws its entire output or updates the top one or two lines. I have to admit that I've never explored whether the escape sequences that control this sort of thing have a documented numeric limit... traditional terminals hence traditional escape sequences (in termcap/terminfo etc.) went up to around 25 lines plus sometimes an extra non-scrolling status line.
Hm, seeing this often was maybe an overstatement, but what about apt-get? That updates the last line giving the download percentage etc. And I'm sure I came across other programs that did similar things, even with multiple lines.

Quote
Depends. You can attach to a background session using something like VLC, or you can run it using ssh with the graphical output tunelled. But for something like this you might be better off sending your status output to a socket, or in any event not updating your output until e.g. USR1 is sent to the process.
Hm, yeah, so far I just write every message in a log file. Other methods are beyond me tbh  :-[

Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: Warfley on October 27, 2021, 02:53:37 pm
If you just want to continously update one line, use a CR Char
Code: Pascal  [Select][+][-]
  1. write('line 1', #13);
  2. Sleep(1000);
  3. write('line 2', #13);
  4. Sleep(1000);
  5. writeln('final line');

This is exactly what CR is made for, no external Unit needed

Besides this, I can always recommend laztermutils when writing console applications. It has some advantages over CRT and can do more stuff (but is restricted to stern escape sequences so only usable on current windows versions and modern Unix terminals)
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 03:27:27 pm
Hm, seeing this often was maybe an overstatement, but what about apt-get? That updates the last line giving the download percentage etc. And I'm sure I came across other programs that did similar things, even with multiple lines.

Yes, and as I said already: WITHIN THE BOUNDS OF THE TERMINAL.

Now again, as I think I've said, on unix (but not Windows) you can hook the WINCH signal so can detect when the size of the Window changes, I can't remember without checking what is made available about window size and scroll position (see e.g. the Lazarus IDE's debugger/test/watchconsolesize.pas file). And you'll find plenty of text programs which get confused when the window size changes, since in many cases they look for the window size in shell variables at startup and generally speaking there's no mechanism for these to be updated during execution.

Later: from the file I referred to above

Code: Pascal  [Select][+][-]
  1. procedure reportSize;
  2.  
  3. var
  4.   winSize: TWinSize;
  5.  
  6. begin
  7.   Write(signalCount, ': ');
  8.   FillChar(winSize, sizeof(winSize), 0);
  9.   if IsaTty(StdInputHandle) = 1 then
  10.     if fpioctl(StdInputHandle, TIOCGWINSZ, @winSize) >= 0 then
  11.       Write(winSize.ws_row, ' x ', winSize.ws_col);
  12.   WriteLn;
  13.   signalCount += 1
  14. end { reportSize } ;
  15.  
  16.  
  17. procedure winchHandler(sig: longint; {%H-}info: PSigInfo; {%H-}context: PSigContext); cdecl;
  18.  
  19. begin
  20.   case sig of
  21.     SIGWINCH: reportSize
  22.   otherwise
  23.   end
  24. end { winchHandler } ;
  25.  

...but if there were a corresponding ioctl for scroll position I'd probably have used it for demonstration purposes.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 03:57:19 pm
Sorry, no idea what you are on about...

All I said is that there are programs that can update a specific line. Just tested zypper. Updates last line properly regardless of the window/terminal size, resizing, amount of previous lines, or whether the last line is visible or not. That's all I want, and apparently that is possible. And so far ojz0r example works good enough for me. So please, no need to use your CAPSLOCK. Thank you
 
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 04:14:03 pm
Sorry, no idea what you are on about...

All I said is that there are programs that can update a specific line. Just tested zypper. Updates last line properly regardless of the window/terminal size, resizing, amount of previous lines, or whether the last line is visible or not. That's all I want, and apparently that is possible. And so far ojz0r example works good enough for me. So please, no need to use your CAPSLOCK. Thank you

Well finding the relevant bit of documentation is your problem then, since despite admitting that you don't understand what I'm trying to explain to you you're clearly under the impression that you know more about this stuff than I do.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 04:23:57 pm
Sorry, no idea what you are on about...

All I said is that there are programs that can update a specific line. Just tested zypper. Updates last line properly regardless of the window/terminal size, resizing, amount of previous lines, or whether the last line is visible or not. That's all I want, and apparently that is possible. And so far ojz0r example works good enough for me. So please, no need to use your CAPSLOCK. Thank you

Well finding the relevant bit of documentation is your problem then, since despite admitting that you don't understand what I'm trying to explain to you you're clearly under the impression that you know more about this stuff than I do.

MarkMLl

Sorry, but I really have no idea what gave you that impression. Clearly you have more knowlegde of this then me. I'm not sure why you all of a sudden have a very lecturing attitute
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 04:57:08 pm
OK, let's both try again. I think you've demonstrated that GotoXY() or whatever works for comparatively small cursor positions, i.e. within the bounds of something that approximated a physical terminal. (25 rows x 40 cols etc.).

I'm typing while trying to monitor a machine tool on a video feed...

Leaving aside for the moment that there might possibly be an ANSI sequence for the status line (line 26) at the bottom of the screen, and leaving aside that writing to a line which is /just/ off the bottom of the screen might have the expected result if the window is then resized.

Note what I said about there being the possibility of a backing store (buffer), and note what I said about the WINCH signal (normally written in caps since that's how it's defined in case-sensitive C). That leaves the possibility that if something is to be written /way/ offscreen (e.g. your line 1000) it will initially go to the backing store, and then will be drawn onto the screen when appropriately scrolled (note that for e.g. Konsole you can explicitly set the size of the backing store) or if the window is resized to be really big (or use an illegibly-tiny typeface).

When I was checking the terminal handling code earlier I found a note that the window size is held per-TTY by the kernel, but the kernel itself makes no use of that. There isn't anything equivalent for the scroll position, so it's down to Konsole etc. to make sense of that and pull stuff from the backing store when needed... an application program can't just ask the kernel for the window position (only for the size).

Generally speaking,  /something/ will make initial sizes available as shell variables:


COLORFGBG='0;15'
COLORTERM=truecolor
COLUMNS=187
...
LINES=63


but those are inherited when a program starts running, and there's no mechanism for something outside the program to update them. And that's why some things can end up in a mess if the geometry changes and they've not hooked WINCH... and this is a problem which affects plenty of programs.

And from what you've said it also affects the crt unit, but you're asking things from it that it was never intended to deliver.

Hope that clears up some of the ambiguities.

MarkMLl





Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 06:32:52 pm
Okay, thanks for bearing with me MarkMLI :)

I can't say I fully understand the problem, but at least I see a bit of the mechanics behind this. But if I understand this correctly, the program sends - let's say a line - to the terminal at the current cursor position, but the cursor position is only just defined within a limited bound (x,y number of cols/rows), so the terminal emulator then has to see where this line fits within it's buffer. And this can get messy, especially when the window gets resized, and therefor the number of cols and rows changes?

Anyway, it runs good enough now for me and I can live with a bit of quirkiness. The code that I posted just didn't work at all, and I was just wondering if this was supposed to work at all, or not. I didn't want to imply that it had to work with crt or anything.
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 06:52:24 pm
I've not looked at the code but it's reasonable to assume that crt doesn't really have a buffer... or if it does it's limited at say 50 rows and 132 cols (those being about the largest I remember on TP-era screens, and 132 also being significant for printer paper).

So if you wanted to implement a larger buffer you'd need to do it yourself, hook the various output routines to put text into that buffer rather than simply sending it to the screen at the current cursor position, and then output that part of the buffer which was determined by (a) the physical window size and (b) the scroll position.

You'd know if the window size changed (from hooking WINCH) but knowing when the terminal was scrolling would be more of a problem... these days it might be handled by DBus etc. but I don't know the "legacy" way of doing it (or if the problem was left entirely with Konsole etc.).

Two final thoughts. First, cursor addressing has an inherent compatibility problem with scrolling... you have to consider what happens if the top line gets deleted. Second, you might find GNU Screen worth looking at since it reimplements cursor etc. handling and has its own buffering.

Somewhat later: there is a 1024x1024 backing store in the crt unit, with an assumed display window of 80x25. I don't think it does anything particularly clever to detect window sizes etc. imposed from external to the program, so most of my points above still apply.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: Warfley on October 27, 2021, 09:11:59 pm
Some time ago I had a similar question to you, how all these unix programs can make so fancy stuff with the terminal and how I could use this myself, and like you the first thing I've found was the crt unit.
Well I am not a big fan of the crt unit and I will explain later why, but because of that I went out to try and figure out how this stuff works.

Generally speaking, every operating system and every terminal emulator does this differently. On DOS you basically get a buffer that is what is represented in the console, if you update the buffer the screen updates at the positions accordingly. On windows this is replaced with more traditional api calls, but because windows is backwards compatible to the stone ages, it also supports the dos style. But most modern terminals use so called escape sequences, basically if you write special commands to the output the terminal will interpret this and make things happen.
Each of these different approaches have different capabilities and this is one of the reason why I don't like crt, crt is designed to work on pretty much every system supported by the FPC, therefore it only supports the subset of functions that is available on all platforms.
The second problem is that it tries to simulate some of the functionality not available on the plattform (e.g. the WhereX on Linux by simply counting the characters printed and lines and stuff), for doing this crt basically intercepts every read and write operation, which has the side effect that simply by including crt in the uses clausel, the behavior of your program will change (significantly in some cases) even if you don't use any functionality of crt, which is something I really don't like when using a library of which I only need a few functions.

So whats the alternative? Well there are a few things that work on any terminal. The CR char (#13) will move  the cursor the the beginning of the current line and the backspace char (#8) moves the cursor one char backwards.
When you write, you write at the current position of your cursor and override anything that is at that position. So if you move your cursor to the start of the line and write something you override the old line.

So if your goal is to only rewrite the last line, you don't need any special libraries or escape sequences, just don't write a newline at the end of the line and when overriding start your string with #13:
Code: Pascal  [Select][+][-]
  1. Write('This line will be replaced');
  2. Write(#13, 'With this line!');

But let's say you wanted to do a more complex application, like for example wget which has like 4 lines to display the current state of the download, you must use something more advanced.
You could do so by using crt, but as I said it's not the best library, but if you don't have a problem with not supporting systems like 16 bit DOS or amiga, you could simply go by using escape sequences, as these are currently supportet on all major operating systems (windows, linux, macos, bsd). Also by doing this you have much more possibilties than with the crt unit (e.g. full 24bit true color support)

The great thing about escape sequences is, they are very easy to implement even without a specialized library, simply write a given string into the console and the magic happens
E.g. to override the last 4 lines:
Code: Pascal  [Select][+][-]
  1. WriteLn('Line 1');
  2. WriteLn('Line 2');
  3. WriteLn('Line 3');
  4. Write('Line 4'); // no newline so we have to go back 1 line less
  5. Write(#27'[3D'); // #27 + '[' is escape sequence, 4D is 3 lines up
  6. WriteLn('Line 1 New');
  7. WriteLn('Line 2 New');
  8. WriteLn('Line 3 New');

To move in any direction us #27'[XY' where X is the number of cells you want to move and Y is A to move to the left, B to move to the right, C to move down and D to move up, so 3D means move 3 up. It will automatically stop at the first visible line (i.e. the first line in the window) so you can't override older lines.
After moving your cursor where you desire, simply rewrite the line with the new content and you are done.

This is what I would recommend if you want to do very simple stuff, no libraries needed, just a few writes. If you want to learn more about escape sequences, the Wikipedia article is quite nice: Link (https://www.wikiwand.com/en/ANSI_escape_code)

So what if you want to do more complex stuff, like really interactive console applications such as vim or emacs. Then you could use crt, but again here due to the fact that crt only implements the subset of functionality available on all platforms this also limits your possibilities.
Thats why I a few years ago build my own library that specializes on Escape sequences, it can basically everything crt can, but better because it doesn't need to worry about systems I won't ever use. If you want to take a look at it, check it out: Link (https://github.com/Warfley/LazTermUtils)
There is one exception to that statement, it can't track the cursor position (WhereX, WhereY of CRT) the reason for this is, that this is actually really hard to do and the main reason why crt has to do all these naughty things with the input and output files changing program behavior. Basically because I wanted to keep my library seperate from the default readln writeln functionality as good as possible there is no way for it to track the current cursor position. If you want to do this you need to implement that functionality for yourself.

But it is pretty powerful, for example the colortest example shows that you can easiely repaint the whole screen at 30-60 fps (60 on linux, 30 on windows because windows console is very slow) so you can easiely use this to print animations or play games (with a very poor resolution, but it works)

There is just one thing you always need to keep in mind when working with escape sequences, the terminal scrolls when your cursor hits the last line. So having one WriteLn instead of a write at the end, might scroll past the first line, resulting in that line not being accessible anymore (as i said you can only move to the window border)
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 09:31:17 pm
Broadly agreed with @Warfley, and I'd note the significance of the term "ansi.sys" in the DOS era. However this does rather sweep scrolling and window size changes under the carpet: it relies entirely on the window manager etc. to get it right.

I'd comment that IBM's TopView attempted to regiment screen-mapped I/O a bit better, and that some of the same approaches were later implemented (with minimal documentation) by the much more popular DesqView for compatibility.

As a fairly early adopter of the OS/2, Windows etc. OSes but a late adopter of their development tools (in part because I was developing for embedded systems in parallel with anything I did on the desktop) I've done /rather/ a lot of investigation of this over the years.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: Warfley on October 27, 2021, 09:36:39 pm
However this does rather sweep scrolling and window size changes under the carpet: it relies entirely on the window manager etc. to get it right.
With regards to this, I found the easiest solution is to just grab the current terminal size before doing any writing, and compute how you need to do your writing (e.g. a progressbar how many colums are filled and how many are empty). This is the approach most of the linux applications (like wget) use. They don't care if lines are broken once they are "finished", they just redraw the currently active lines according to the current screen size which will overdraw the old version that might have been broken through resizing.

If it is more complex I then go to the other extreme and simply redraw the whole frame whenever a resize occured. Linux terminals like XTerm, Gnome Terminal or Konsole, as well as MacOS Terminal can easiely redraw at 60 frames per second and even the painfully slow windows console can redraw at 30. (Of course dependent on window size and font size, this was for my 4k monitor fullscreen with font size 14)
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: MarkMLl on October 27, 2021, 10:00:56 pm
One of the reasons I slipped WINCH monitoring into the Lazarus IDE's console window was that I wanted to test that it did in fact work in the context of something which hadn't been explicitly written as a shell.

However I agree with you: most programs quite simply ignore it unless they're actually using something like Curses.

Handling scrolling is of course a separate issue, and risks getting bogged down in the fundamental difference between Windows and X11 window handling (on Windows, the "furniture" is part of the program, while on X11 it's not).

The Linux kernel's input event handling doesn't get scroll events, so I suspect that anything interested would have to interact with X11... which could be a problem for a non-GUI program.

MarkMLl
Title: Re: GotoXy in terminal (crt) (Linux, FPC 3.2)
Post by: big_M on October 27, 2021, 10:59:08 pm
Oh, thanks a lot @Warfley for this great explanation, this makes this topic much clearer to me now. I will definitely try both the escape sequences as well as the CR chars. This was really helpful, thank you. I will have to play around a bit with this to get a sense of the possibilities and limitations, but I'm really not after something complex.

Also thanks again @MarkMLl

TinyPortal © 2005-2018