Console In/Output slow

[Martin_fr]

I understood you, that your patch is included in *Lazarus* trunk, so I installed Lazarus trunk = main_3_99-1434-g76da2aad75 with FPC 3.2.2-r0d122c49 (release).

With FPDebug I see only 461 lines. If I switch to GDB, I see 1000 lines. So this patch does not work here.

As said, I'm on Linux Ubuntu 22.04 64-bit with GTK2.

Yes you only need Lazarus trunk/main.

Strange, since I could reproduce without the patch, but no longer with it (Fedora).

Do you have multiple Lazarus installed? Just to be sure there was no mixup? Also could you check if the file content matches the link I gave to gitlab?
Sorry for the extra inconvenience.

main_3_99-1434-g76da2aad75 is definitely correct

[zeljko]

Tested Lazarus 3.99 (rev main_3_99-1412-gd6b7ffd24c) FPC 3.2.2 x86_64-linux-qt5 LinuxMint 20.02 - everything works fine, even 50k writelns are written to the console in/out (with fpdebug and with gdb).

[Hartmut]

Do you have multiple Lazarus installed? Just to be sure there was no mixup?

I have multiple Lazarus installed. The new installation was made via fpcupdeluxe version 2.4.0c into a new, empty folder. My test was with all IDE-options as Defaults.

Also could you check if the file content matches the link I gave to gitlab?

I checked your patch = https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/47c34d2e7ea0356f0b101becc5df8cc311e8c4cd (from reply #31) manually and what I see, the patch is there. This is my file <installdir>/lazarus/components/fpdebug/fpdbglinuxclasses.pas from line 1434 on:

Code: Pascal  [Select][+][-]

1. function TDbgLinuxProcess.GetConsoleOutput: string;
2. var
3.   ABytesAvailable: DWord;
4.   ABytesRead: cint;
5.   s: string;
6. begin
7.   if fpioctl(FMasterPtyFd, FIONREAD, @ABytesAvailable)<0 then
8.     ABytesAvailable := 0;
9.  
10.   result := '';
11.   while ABytesAvailable>0 do
12.   begin
13.     setlength(s, ABytesAvailable);
14.     ABytesRead := fpRead(FMasterPtyFd, s[1], ABytesAvailable);
15.     SetLength(s, ABytesRead);
16.     Result := Result + s;
17.  
18.     if fpioctl(FMasterPtyFd, FIONREAD, @ABytesAvailable)<0 then
19.       ABytesAvailable := 0;
20.   end;
21. end;

As an additional test I set up a new project within Lazarus 3.99 to have virgin project options. A question regarding the debugger info format (see screenshot) I answered with the default (Dwarf 3).
I attached this project, maybe this helps you.

[jcmontherock]

Did you try the "Keyboard" package ?

[Hartmut]

Did you try the "Keyboard" package ?

No, I did not install any packages. Should I? I tested with a virgin installation made by fpcupdeluxe.

[Martin_fr]

Did you try the "Keyboard" package ?

How is that related?????????????????????

[jcmontherock]

... To the first question of the topic !

[jollytall]

Which first question do you mean?
As I mentioned, I did not add any special package, anyway.
Without adding any package, of course I can simply add a readln; also instead of a sleep(), so the problem is not that I cannot get the whole printout, the problem is that it should not work like that (as the long thread above details all aspects of it).

[Martin_fr]

Did you try the "Keyboard" package ?

No, I did not install any packages. Should I? I tested with a virgin installation made by fpcupdeluxe.

With regards to the issues of the "console output" not showing all info => no. no need to install any package.

[Martin_fr]

Ok, one of my other VM also kept being incomplete after the patch.

It seems it is a timing/waiting issue.

With GDB, GDB sends the message that the target app stopped to the IDE via a pipe. However fast pipes may be, it takes a moment (and the same moment may be needed by the StdOut of the target app to pass through its pipe).
So the content is read, even if a short delay is needed.

FpDebug is too fast. (really that shouldn't be a bad thing...)

So I added a "sleep"
https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/972e988e00b4ee0c361d882ec5f19abf9e246e33

For me, even a sleep(20) was enough. But that can depend on a ton of things. I added sleep(50) in the hope it is enough.
If it is not, please try to increase it in steps of 100, until you get all the content.

I have merged this to fixes branch now. Since it definitely improves the situation on some platforms.

I may look at moving the code, so that the debugger can (in OnIdle) keep reading for another few seconds. Or maybe it can detect if the pipe got closed.

But for now, I wanted something that did not change a lot of code, so it could easily be merged to the fixes branch. (If needed the sleep can still be increased a bit).

[Hartmut]

I installed a new trunk version and tested but unfortunately it does not work:

Lazarus = main_3_99-1455-g760247b8d1
FPC = 3.2.2-r0d122c49 (release)
OS = Linux Ubuntu 22.04 64-bit with GTK2

On the 1st test I kept the loop 'for i:=1 to 1000 do' and I got only 233 and a half lines (see screenshot1) and the IDE hung (was not longer operable and must be restarted).

Several tests with loop values up to 230 worked perfectly (see screenshot2).

All loop values from 235 and greater failed:

Several tests with loop values from 235 to 1000 showed NO Output (empty output window) and the IDE always hang.

During 4 tests with loop value=2000 I 3x saw 1136 lines (see screenshot3) and the IDE hang. 1x I saw 1351 lines and the IDE hung.

All tests have been with all IDE-options as Defaults after the new installation (see screenshot4).

I checked your patch = https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/972e988e00b4ee0c361d882ec5f19abf9e246e33 (from reply #54) manually and what I see, the patch is there. This is my file <installdir>/lazarus/components/lazdebuggers/lazdebuggerfp/fpdebugdebugger.pas from line 3474 on:

Code: Pascal  [Select][+][-]

1. procedure TFpDebugDebugger.FDbgControllerProcessExitEvent(AExitCode: DWord);
2. var
3.   AThread: TFpWaitForConsoleOutputThread;
4. begin
5.   if assigned(FConsoleOutputThread) then
6.     begin
7.     AThread := TFpWaitForConsoleOutputThread(FConsoleOutputThread);
8.     FConsoleOutputThread := nil;
9.     AThread.Terminate;
10.     AThread.WaitFor;
11.     sleep(50);
12.     AThread.DoHasConsoleOutput(0);
13.     AThread.Free;
14.     end;
15.  
16.   SetExitCode(Integer(AExitCode));
17.   {$PUSH}{$R-}
18.   DoDbgEvent(ecProcess, etProcessExit, Format('Process exited with exit-code %u',[AExitCode]));
19.   {$POP}
20.   LockRelease;
21.   try
22.     SetState(dsStop);
23.     StopAllWorkers;
24.     FreeDebugThread;
25.   finally
26.     UnlockRelease;
27.   end;
28. end;

[Thaddy]

i am missing a Output.Flush() in your code, or on the outputstream itself

[Hartmut]

i am missing a Output.Flush() in your code, or on the outputstream itself

You can see 'flush(Output)' in screenshots 2 and 3 in above post.

[Martin_fr]

All loop values from 235 and greater failed:

Several tests with loop values from 235 to 1000 showed NO Output (empty output window) and the IDE always hang.

During 4 tests with loop value=2000 I 3x saw 1136 lines (see screenshot3) and the IDE hang. 1x I saw 1351 lines and the IDE hung.

For the test with cut off output (some output, but not all): Can you try with "sleep(150)" or "sleep(250)" .... "sleep(1550)" ?


The IDE hanging is odd....

If that is related to the patch, it would have to be

Code: Pascal  [Select][+][-]

1. function TDbgLinuxProcess.GetConsoleOutput: string;
2. var
3.   ABytesAvailable: DWord;
4.   ABytesRead: cint;
5.   s: string;
6. begin
7.   if fpioctl(FMasterPtyFd, FIONREAD, @ABytesAvailable)<0 then
8.     ABytesAvailable := 0;
9.  
10.   result := '';
11.   while ABytesAvailable>0 do
12.   begin
13.     setlength(s, ABytesAvailable);
14.     ABytesRead := fpRead(FMasterPtyFd, s[1], ABytesAvailable);
15.     SetLength(s, ABytesRead);
16.     Result := Result + s;
17.  
18.     if fpioctl(FMasterPtyFd, FIONREAD, @ABytesAvailable)<0 then
19.       ABytesAvailable := 0;
20.   end;
21. end;
22.  

fpioctl either hangs.... In which case it probably is a race condition that existed before the change too.

Or "ABytesAvailable" never becoming 0. 


Maybe your distro has an ioctl implementation that differs... (strange though, since it works for other amounts of data).

Could you please do a test, please.

Add a counter to the above loop, and abort it at 100 iterations.

Code: Pascal  [Select][+][-]

1. TESTCNT := 0; ///// <<<<
2.   while (ABytesAvailable>0)    AND (TESTCNT < 100)    do
3.   begin
4.     inc(TESTCNT); // <<<<<<<<<<<<<<<<
5.     setlength(s, ABytesAvailable);
6.  

Then also check, if the output has any unexpected data....

If that still hangs, the fpIoctl probably hangs. Assuming it is not something entirely different.

If you can run the IDE itself in the debugger please.

[Hartmut]

I verified that your patch in <installdir>/lazarus/components/fpdebug/fpdbglinuxclasses.pas is in my newly installation exactly as you posted in reply #58.

I will do the requested tests for you. Do I see this right, that I have to recompile the IDE after changing the sleep value and after inserting counter 'TESTCNT' (I'm not familiar with that)?

If yes, to recompile the IDE I use menu "Tools / Build Lazarus with Profile: Normal IDE" and that's all?