Console In/Output slow

[Martin_fr]

Yes, after changing the files recompile the IDE from the tools menu.

If your lazarus executable is in a folder to which you have no write permissions (e.g. if installed from deb/rpm) then the new lazarus will be in your home folder ~/.lazarus or wherever you configured your PCP).

However, usually with fpcupdeluxe, you have the entire install in your home dir, and it is writeable to you. So then the lazarus exe will replaced.
- You may want to backup the original first
- In rare cases (if the file is locked) a lazarus.new exe is created) => rename when the lock is gone.

[Hartmut]

I made the requested tests but with no success:

A) First I tested without adding counter 'TESTCNT':
For sleep I tested values 150, 250, 650 and 1550:
 - with loop value = 100 everything is perfect (as before)
 - with loop values 250, 500 and 1000 there is NO output at all and the IDE hangs
 - with loop value = 2000 I get always 1136 or 1351 lines and the IDE hangs.
With sleep = 1550 it is obvious visible, that the output starts delayed.

B) Then I tested with adding counter 'TESTCNT':
For sleep I tested values 150, 650 and 1550:
I repeated above 5 loop values with identically results (and the IDE always hung). So no difference to before.

Here are my 2 patches:

Code: Pascal  [Select][+][-]

1. procedure TFpDebugDebugger.FDbgControllerProcessExitEvent(AExitCode: DWord);
2. var
3.   AThread: TFpWaitForConsoleOutputThread;
4. begin
5.   if assigned(FConsoleOutputThread) then
6.     begin
7.     AThread := TFpWaitForConsoleOutputThread(FConsoleOutputThread);
8.     FConsoleOutputThread := nil;
9.     AThread.Terminate;
10.     AThread.WaitFor;
11.     sleep(1550); /// changed HG 14.2.24
12.     AThread.DoHasConsoleOutput(0);
13.     AThread.Free;
14.     end;
15.  
16.   SetExitCode(Integer(AExitCode));
17.   {$PUSH}{$R-}
18.   DoDbgEvent(ecProcess, etProcessExit, Format('Process exited with exit-code %u',[AExitCode]));
19.   {$POP}
20.   LockRelease;
21.   try
22.     SetState(dsStop);
23.     StopAllWorkers;
24.     FreeDebugThread;
25.   finally
26.     UnlockRelease;
27.   end;
28. end;
29.  
30. function TDbgLinuxProcess.GetConsoleOutput: string;
31. var
32.   ABytesAvailable: DWord;
33.   ABytesRead: cint;
34.   s: string;
35.   TESTCNT: integer; /// HG 14.2.24
36. begin
37.   if fpioctl(FMasterPtyFd, FIONREAD, @ABytesAvailable)<0 then
38.     ABytesAvailable := 0;
39.  
40.   result := '';
41.   TESTCNT := 0; /// HG 14.2.24
42.   while (ABytesAvailable>0) AND (TESTCNT < 100) do  /// HG 14.2.24
43.   begin
44.     inc(TESTCNT); /// HG 14.2.24
45.     setlength(s, ABytesAvailable);
46.     ABytesRead := fpRead(FMasterPtyFd, s[1], ABytesAvailable);
47.     SetLength(s, ABytesRead);
48.     Result := Result + s;
49.  
50.     if fpioctl(FMasterPtyFd, FIONREAD, @ABytesAvailable)<0 then
51.       ABytesAvailable := 0;
52.   end;
53. end;

[Martin_fr]

Thanks for the feedback.

Found the hang. After I found - or I think I found - a potential crash.
(strange that I didn't get the hang, I should have...)

Both fixed (I think) https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/b1a91a2333b2d1a6fbdcf90c9eff7b128725a25a
(also in fixes branch)

1) The hang: I moved the DoHasConsoleOutput down, but then the thread never gets its event, and can't exit.
2) In some cases an async call could still be pending and hit the destroyed object.

I haven't changed the sleep. Since the lines you didn't get where (potentially) hidden by the endless wait before.

So, if with that some lines are missing (but hopefully nothing hangs) then try "sleep(100)" again.

[Martin_fr]

2) In some cases an async call could still be pending and hit the destroyed object.

Forget that, I found the existing RemoveAsyncCall.

[Thaddy]

RemoveAsyncCalls

[Hartmut]

Great Martin_fr that you found the hang. I will test the new lazarus trunk version tomorrow and report then.

[Hartmut]

I made new tests and the good news is, that the hang is fixed, but the bad news is, that the sleep does not help.

Lazarus = main_3_99-1463-g5fa267ce0a
FPC = 3.2.2-r0d122c49 (release)
OS = Linux Ubuntu 22.04 64-bit with GTK2

First I tested with the original sleep of 50:
 - with loop value = 500 everything is perfect
 - with loop value = 1000 I got always 688 lines
 - with loop value = 2000 I got always 1782 or 1998 lines
 - with loop value = 10000 I got always 9757 or 9972 lines

Then I changed the sleep to 150, 250, 650, 1550 and 3000 and repeated above loop values. I got the same results with 2 exceptions:
 - with sleep=1550 and loop value=10000 I got 1 time all 10000 lines (in 10 tests)
 - with sleep=3000 and loop value=2000 I got 1 time all 2000 lines (in 10 tests)
So these 2 exceptions look like random results and were only 1 time in 10 tests.

Here is my <install_path>/lazarus/components/lazdebuggers/lazdebuggerfp/fpdebugdebugger.pas from line 3474 on where I patched the sleep:

Code: Pascal  [Select][+][-]

1. procedure TFpDebugDebugger.FDbgControllerProcessExitEvent(AExitCode: DWord);
2. var
3.   AThread: TFpWaitForConsoleOutputThread;
4. begin
5.   if assigned(FConsoleOutputThread) then
6.     begin
7.     AThread := TFpWaitForConsoleOutputThread(FConsoleOutputThread);
8.     FConsoleOutputThread := nil;
9.     AThread.Terminate;
10.     AThread.DoHasConsoleOutput(0);
11.     AThread.WaitFor;
12.     sleep(3000); /// HG changed 15.2.24
13.     AThread.DoHasConsoleOutput(0);
14.     AThread.Free;
15.     end;
16.  
17.   SetExitCode(Integer(AExitCode));
18.   {$PUSH}{$R-}
19.   DoDbgEvent(ecProcess, etProcessExit, Format('Process exited with exit-code %u',[AExitCode]));
20.   {$POP}
21.   LockRelease;
22.   try
23.     SetState(dsStop);
24.     StopAllWorkers;
25.     FreeDebugThread;
26.   finally
27.     UnlockRelease;
28.   end;
29. end;

I saw, that 1 inserted line of your patch = https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/b1a91a2333b2d1a6fbdcf90c9eff7b128725a25a (from reply #62) is missing there:
   Application.RemoveAsyncCalls(AThread);
Hope that is correct.

[Martin_fr]

Ok, thanks for the test.
So there must be different issues that prevent the text from being completely read....

I'll have to get back to this. But for now I have other stuff to do first. (best if an issue is reported on the bug tracker).

At least, the sleep should improve it on some platforms.

[MarkMLl]

Thanks for that. I'll try to find time to see what happens if it's un-greyed, at least on v2.

I admit that I've not got to v3 yet, mostly because I missed the announcement due to the amount of wibble in the "Announcements" part of the forum.

The issue is that it's disabled in debugger/pseudoterminaldlg.lfm. Enabling it results in an error when the IDE starts: 'Unable to find the component class "TPseudoConsoleDlg". It is not registered via RegisterClass and no lfm was found. It is needed by unit: pseudoterminadlg.frm'.

I'm stopping at that point since I've not worked through the documentation to a sufficient extent that I understand how this stuff hangs together, and I don't know how many people might have a use for this particular facility. Also I suspect that any tinkering might impact on things like the missing lines issue that Martin is investigating elsewhere.

MarkMLl

[Martin_fr]

The issue is that it's disabled in debugger/pseudoterminaldlg.lfm. Enabling it results in an error when the IDE starts: 'Unable to find the component class "TPseudoConsoleDlg". It is not registered via RegisterClass and no lfm was found. It is needed by unit: pseudoterminadlg.frm'.

Mind that in 3.0 this form (like most other debugger stuff) has moved to ide/packages/idedebugger/
In my 3.99 shows that form, with the output of the debugger in it. (The one in the attached image)

I am not sure what you mean by "enabling it"? The unit is used. It's a TForm.

I am trying to make sense of that error. Descendants of TForm aren't "registered", they don't need to be found while loading a form. The are the form, they get created in code (like TForm in any GUI app).

In this case in
  ide\debugmanager.pas

Code: Pascal  [Select][+][-]

1. procedure TDebugManager.ViewDebugDialog(const ADialogType: TDebugDialogType;
2.   BringToFront: Boolean; Show: Boolean; DoDisableAutoSizing: boolean;
3.   InitFromSourceEdit: boolean);
4. const
5.   DEBUGDIALOGCLASS: array[TDebugDialogType] of TDebuggerDlgClass = (
6.     TDbgOutputForm, TDbgEventsForm, TBreakPointsDlg, TWatchesDlg, TLocalsDlg,
7.     TCallStackDlg, TEvaluateDlg, TRegistersDlg, TAssemblerDlg, TIDEInspectDlg,
8.     TPseudoConsoleDlg, TThreadsDlg, THistoryDialog
9.   );
10. var
11.   CurDialog: TDebuggerDlg;
12. begin
13. ...
14.   if FDialogs[ADialogType] = nil
15.   then begin
16.     CurDialog := TDebuggerDlg(DEBUGDIALOGCLASS[ADialogType].NewInstance);
17.  

The only guess I have is, that maybe you have some outdated ppu or resource files? Or an old file in an outdated location (duplicate file).

So, if it works for you when using gdb, then that is odd. Because the form does not depend on the backend used. The backend only provides the text/data as string.


FpDebug for some reason does not use DbgIntfPseudoTerminal. It sets up it's own way to read the StdOut of the debug-target. But it read the data, and sends it to the frontend.

[MarkMLl]

I am not sure what you mean by "enabling it"? The unit is used. It's a TForm.

The checkgroup inside the form is disabled.

So, if it works for you when using gdb, then that is odd. Because the form does not depend on the backend used. The backend only provides the text/data as string.

No, I said that it worked when I submitted the patch (which predated fpdebug).

I'll try to take another look, possibly with v3, but I've got a lot of other stuff on my plate. Plus, as I've said, I don't know how useful this would be to other people and there might be related issues parsing debugger output.

MarkMLl

[Martin_fr]

Strange, I can't find where this checkbox has ever been enabled. It was disabled on its very first commit
https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/4358281aed386de7d127e77d660b4e8abd1262b1#05189c72f4e7e3f113939623d830d8e7fd913ce8_23_139

Anyway, I had no issues to enable it, and rebuild the IDE (and test it / well, I only printed printable, so only checked line numbers)

[MarkMLl]

Strange, I can't find where this checkbox has ever been enabled. It was disabled on its very first commit
https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/4358281aed386de7d127e77d660b4e8abd1262b1#05189c72f4e7e3f113939623d830d8e7fd913ce8_23_139

Anyway, I had no issues to enable it, and rebuild the IDE (and test it / well, I only printed printable, so only checked line numbers)

Thanks, saved me a lot of effort: almost certainly a mistake on my part when I did the patch. The line numbers might possibly be useful to somebody, although that's at the mercy of how the debugger breaks things up.

MarkMLl