Console In/Output slow

[jollytall]

I don't know who uses the Lazarus built-in Console In/Output (Ctrl+Alt+O). I write a lot of command line programs and typically I use this feature (can debug and do not need to go out from Lazarus to the character terminal).
When I have a lot to write to the screen apparently there is some threading like bug, I guess. Lazarus does the write (using simply writeln('...') somehow in the background, but it is slower than the program itself. When the program completes, apparently it also stops the thread (or what other mechanism used) to write out the output. So, I loose the last lines of the outputs.
Of course there is a solution, to add something like Sleep(10000) before the last end., but I find it not too elegant.
Has anyone seen this before?

I am using Linux (Debian, gtk3) and Lazarus 2.2.6.

[cdbc]

Hi
I've seen that too... It doesn't particularly like escape-sequences 
Regards Benny

[Thaddy]

Compile with {$I-} will speed things up considerably.
You can also call Flush(Output); or Flush(stdOut) before terminating to make sure the Output buffer is completely processed.
You may also want to call Flush(ErrOutput) or Flush(StdErr);
The handles are simply aliases.
In {$I-} mode you can still do old school error checking.

[MarkMLl]

I've seen that too... It doesn't particularly like escape-sequences 
Regards Benny

My recollection is that it's generally... messy code, and I say that without intending disrespect to the (other) people who've worked on it.

The problem is that closely-entangled code is trying to filter everything that's come through the debugger and from the program itself, and it can be very difficult to parse related characters: \r\n and escape sequences as one example, and UTF-8 sequences as another.

In addition everybody agrees that "smart" terminal output isn't something that the LCL is particularly good at: no real attempts at standardised screen and keyboard handlers, and the community would never agree on what was needed if such a thing was proposed. There's also the fundamental issue that Windows console programs don't have an equivalent of the unix WINCH signal.

If you really do need decent escape etc. handling, I suggest starting your program from a shell (running in an appropriate terminal session) and then attaching the debugger to it using Lazarus.

MarkMLl

[Thaddy]

Benny is referring to ^<ansiChar> escapes as are use by crt for the console.

[MarkMLl]

Benny is referring to ^<ansiChar> escapes as are use by crt for the console.

He said escape sequences, not control characters.

MarkMLl

[cdbc]

Hi
@Thaddy understood my writing correctly.
Sorry if it was bad wording on my part...
Regards Benny

[Kays]

[…] Of course there is a solution, to add something like Sleep(10000) before the last end., but I find it not too elegant. […]

The better solution is of course: Don’t produce so much output. Most humans can’t read that fast. A common default bandwidth for a non‑physical TTY device is 9600 baud; that’s 9600 symbols per second.

If you’re doing semi‑graphics on the terminal requiring high throughput, e. g. rendering a color video with, like, 132 × 40 printable characters @25 FPS, the first thing to do is to crank up the baud rate, see cfsetospeed, tcsetattr POSIX™ functions.

I suppose the Lazarus console presents itself as a pipe, though. I don’t know, I don’t use Lazarus. One thing you could do is increase the pipe’s capacity (fcntl(F_SETPIPE_SZ)); it might affect the observed speed. Maybe Lazarus’ rendering is indeed just too slow.

[Thaddy]

Hi
@Thaddy understood my writing correctly.
Sorry if it was bad wording on my part...
Regards Benny

Things like writeln(^M^J^I'Hello, world');
Same as #13#10#9'Hello, world' but shorter and it isn't bad wording, ctrl characters belong to the escape family.
Mark is probably not even aware of this pretty ancient syntax.
But still part of the language and supported in all modes.

Anyway, to speed up the console follow my first advice, {$I-} and Flush()
Speed will be much, much faster.

[Martin_fr]

When the program completes, apparently it also stops the thread (or what other mechanism used) to write out the output. So, I loose the last lines of the outputs.
Of course there is a solution, to add something like Sleep(10000) before the last end., but I find it not too elegant.

You should report this part as a bug, please.

That Window is known to need some work... But, alas, so is a lot of other stuff. Anyway, the loss of output is a bug.

I assume you use FpDebug?
(Because that console works afaik with gdb and fpdebug, but the code handling it is in different places....)

As for the speed, it sets up a virtual pty and then it does regular "read" from that. Not sure what can be done.

When you say "speed" do you mean throughput or lag?

I.e. Does the data get "throttled", or is it just a certain time behind, but that time does not increase over a certain value?

[Thaddy]

The loss of output is only a "bug" if you do not call flush()....
To solve that in the compiler is not trivial and not necessary.
Maybe just against documentation. Especially with multiple threads it should be documented that you need to call Flush() at thread finish().
I thought that was common knowledge, but given this thread probably not.

BTW that has nothing to do with the speed, but {$I+/-} does..

[Martin_fr]

The loss of output is only a "bug" if you do not call flush()....

Well, ... It is a bug, if the app has written it (for whatever reason), but the IDE does not read it.  From a quick glance, that may be possible. So a report is welcome.

[marcov]

Runcommandloop polls the pipes again after the process is terminated to make sure.

[Thaddy]

That is true and will finish, but runcommandloop is not mainstream yet for most users... So if you use just TThread.Execute the programmer is responsible to call flush(). I don't consider that a bug since the compiler can not analyse such a program flow.
With the provision that it can be better documented.

[jollytall]

Thanks for the replies. Some answers:
- No, I do not use any special (escape, UTF, etc.) characters.
- I do a lot of calculation and then dump the result, what I "Select All", "Copy" and Paste to a spreadsheet. So, I do not try to read numbers very fast.
-- Yes, I know I could write it to a file, but then I need to open the file (e.g. a .csv) in the spreadsheet and then there copy/paste it to a sheet, where I have all the charting, etc.
-- Yes, I could also make the whole spreadsheet in fpSpreadsheet, but it has its own challenges.
- I think it is a bug, if the print out to a black and white terminal works OK, but the Lazarus built-in looses some.
- I tried to switch off the error checking, but it made no difference.
- I tried to add flush(Output), but it made no difference.

Here is a very small code:

Code: Pascal  [Select][+][-]

1. var i : integer;
2. begin
3. {$I-}
4. for i:=1 to 1000 do
5.   writeln('My number is ', i);
6. flush(Output);
7. end.

With/without {$I-} and with/without flush I get 461 lines. If I add Sleep(100) before the end., then I get 688 lines, and with Sleep(200) I get all the lines. It seems there are some fixed steps, so, I can never get e.g. 500 lines.
I also tried to leave out the text, and used only writeln(i), in which case I got 9057 lines (with a 10 000 loop of course).