Recent

Author Topic: [SOLVED]error when modifying main form from indy thread  (Read 14522 times)

taazz

  • Hero Member
  • *****
  • Posts: 5363
Re: [SOLVED]error when modifying main form from indy thread
« Reply #30 on: October 21, 2015, 06:28:46 am »
Code: Pascal  [Select]
  1. procedure TApplication.ProcessMessages;
  2. begin
  3.   if Self=nil then begin
  4.     // when the programmer did a mistake, avoid getting strange errors
  5.     raise Exception.Create('Application=nil');
  6.   end;
  7.   WidgetSet.AppProcessMessages;
  8.   ProcessAsyncCallQueue;
  9. end;

It calls widgeset specific message handling procedure and procedure which handles asynchronous call queues periodically.  Widgeset specific message handling procedure (in case win32 widgeset) calls CheckSynchronize and CheckPipeEvents procedures.

CheckSynchronize procedure is important one and related to handling thread execution succesfully. If you are using threads, you need to call CheckSynchronize periodically in your main program loop.
good good, Can you point me to the main lcl application loop too? What it does?
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Cyrax

  • Hero Member
  • *****
  • Posts: 758
Re: [SOLVED]error when modifying main form from indy thread
« Reply #31 on: October 21, 2015, 06:36:31 am »
Code: Pascal  [Select]
  1. procedure TApplication.ProcessMessages;
  2. begin
  3.   if Self=nil then begin
  4.     // when the programmer did a mistake, avoid getting strange errors
  5.     raise Exception.Create('Application=nil');
  6.   end;
  7.   WidgetSet.AppProcessMessages;
  8.   ProcessAsyncCallQueue;
  9. end;

It calls widgeset specific message handling procedure and procedure which handles asynchronous call queues periodically.  Widgeset specific message handling procedure (in case win32 widgeset) calls CheckSynchronize and CheckPipeEvents procedures.

CheckSynchronize procedure is important one and related to handling thread execution succesfully. If you are using threads, you need to call CheckSynchronize periodically in your main program loop.
good good, Can you point me to the main lcl application loop too? What it does?

Code: Pascal  [Select]
  1. {------------------------------------------------------------------------------
  2.   TApplication Run
  3.   MainForm is loaded and control is passed to event processor.
  4. ------------------------------------------------------------------------------}
  5. procedure TApplication.Run;
  6. begin
  7.   if (FMainForm <> nil) and FShowMainForm then FMainForm.Show;
  8.   WidgetSet.AppRun(@RunLoop);
  9. end;
  10.  
  11. {------------------------------------------------------------------------------
  12.   TApplication RunLoop
  13.   control is passed to event processor.
  14. ------------------------------------------------------------------------------}
  15. procedure TApplication.RunLoop;
  16. begin
  17.   repeat
  18.     if CaptureExceptions then
  19.       try // run with try..except
  20.         HandleMessage;
  21.       except
  22.         HandleException(Self);
  23.       end
  24.     else
  25.       HandleMessage; // run without try..except
  26.   until Terminated;
  27. end;
  28.  
  29. {------------------------------------------------------------------------------
  30.   Method: TApplication.HandleMessage
  31.   Params: None
  32.   Returns:  Nothing
  33.  
  34.   Handles all messages first then the Idle
  35.  ------------------------------------------------------------------------------}
  36. procedure TApplication.HandleMessage;
  37. begin
  38.   WidgetSet.AppProcessMessages; // process all events
  39.   if not Terminated then Idle(true);
  40. end;
  41.  
  42. procedure TWidgetSet.AppRun(const ALoop: TApplicationMainLoop);
  43. begin
  44.   if Assigned(ALoop) then ALoop;
  45. end;
  46.  
  47.  

It seems that it calls widgeset specific main loop. In win32, AppRun method isn't overloaded.

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: [SOLVED]error when modifying main form from indy thread
« Reply #32 on: October 21, 2015, 09:34:52 am »
2) never ever call processmessages its the worst think you can do. although lcl makes it impossible you should realy try to avoid it at all costs.
Yeah... lcl makes it impossible but I should avoid it at all costs :) Nice. And how would I avoid it in this example?
1) what is processmessages? what it does and when is used?
In this case you simple don't need to use it at all. the processing is in an other thread the messages will be processed in due time from the main thread stop disrupting the normal flow of things for no reason.
O wow, for Windows you can indeed remove the ProcessMessages in my simple_postmessage.zip. BUT... on Linux you DO need the ProcessMessages. Otherwise the lines in memo1 don't get displayed until ALL messages are handled !!! My simple_postmessage is a simple program without threads but apparently Windows sees the chance to update the memo1 but Linux does not. So the messaging handling in Linux is really really different :)

I think I understand now why, on Linux, I get the error after 13.000 messages processed. I thought ProcessMessages shouldn't be called in events because you could click a button again but it should also not be called in a message-procedure. Here is why:
Here is the essence of my simple_messages:
Code: Pascal  [Select]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   p: integer;
  4. begin
  5.   Memo1.Clear; // BUG on Linux - Needs to be clear otherwise it doesn't scrol
  6.   for p := 1 to 20000 do
  7.   begin
  8.     PostMessage(Self.Handle, LM_POST_MESSAGE, p, p);
  9.     Button1.Caption := IntToStr(p);
  10.   end;
  11. end;
  12.  
  13. procedure TForm1.post_message(var Msg: TLMessage);
  14. begin
  15.   Memo1.Lines.add('p = ' + IntToStr(Msg.wParam) + ' ' + IntToHex(Integer(sPtr),4));
  16.   Application.ProcessMessages;                // <------- danger of recursion here if there are more LM_POST_MESSAGE on queue
  17. end;

When all the PostMessage are done the handling/processing begins. It runs post_message() and in that I call Application.ProcessMessages (note: on Linux this is necessary to update the memo!). What happens in Linux, I think, is that the App.ProcessMessages calls post_message() again for the second message. And because of the App.ProcessMessages it calls it again for the third. So we have recursion !! After 13.000 iterations/recursion it crashes with a SIGSEV.

My question is twofold:
1) How can I update the layout/form without calling ProcessMessages (because that recurses into all the other messages) ?
2) Shouldn't ProcessMessages (under Linux) have a mechanisme to prevent such recursion ??


Edit: You can see the recursion happen on Linux when I print out the sPtr (stack pointer) (nibbling 640 bytes from the stack each time). Without the App.ProcessMessages that number stays the same for all calls and with it it changes (until it crashes). On Windows it always stays the same regardless of the App.ProcessMessages. So, somehow Windows avoids the recursion (while the messaging on Linux does not).
« Last Edit: October 21, 2015, 10:32:37 am by rvk »

Cyrax

  • Hero Member
  • *****
  • Posts: 758
Re: [SOLVED]error when modifying main form from indy thread
« Reply #33 on: October 21, 2015, 11:34:41 am »
...
Code: Pascal  [Select]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   p: integer;
  4. begin
  5.   Memo1.Clear; // BUG on Linux - Needs to be clear otherwise it doesn't scrol
  6.   for p := 1 to 20000 do
  7.   begin
  8.     PostMessage(Self.Handle, LM_POST_MESSAGE, p, p);
  9.     Button1.Caption := IntToStr(p);
  10.   end;
  11. end;
  12.  
  13. procedure TForm1.post_message(var Msg: TLMessage);
  14. begin
  15.   Memo1.Lines.add('p = ' + IntToStr(Msg.wParam) + ' ' + IntToHex(Integer(sPtr),4));
  16.   Application.ProcessMessages;                // <------- danger of recursion here if there are more LM_POST_MESSAGE on queue
  17. end;
...

Move Application.ProcessMessages out of your message handling method, put it in Button1Click method and call it only in certain periods.

Like this :
Code: Pascal  [Select]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   p: integer;
  4. begin
  5.   Memo1.Clear; // BUG on Linux - Needs to be clear otherwise it doesn't scrol
  6.   for p := 1 to 20000 do
  7.   begin
  8.     PostMessage(Self.Handle, LM_POST_MESSAGE, p, p);
  9.     Button1.Caption := IntToStr(p);
  10.     if p mod 100 = 0  then
  11.        Application.ProcessMessages;
  12.   end;
  13. end;
  14. ...





rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: [SOLVED]error when modifying main form from indy thread
« Reply #34 on: October 21, 2015, 11:59:43 am »
Move Application.ProcessMessages out of your message handling method, put it in Button1Click method and call it only in certain periods.

Like this :
Code: Pascal  [Select]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   p: integer;
  4. begin
  5.   Memo1.Clear; // BUG on Linux - Needs to be clear otherwise it doesn't scroll
  6.   for p := 1 to 20000 do
  7.   begin
  8.     PostMessage(Self.Handle, LM_POST_MESSAGE, p, p);
  9.     Button1.Caption := IntToStr(p);
  10.     if p mod 100 = 0  then
  11.        Application.ProcessMessages;
  12.   end;
  13. end;
  14. ...
That doesn't quite work. The PostMessage returns immediately (after putting the message in the queue). It's not like a SendMessage, which waits for the message to be processed. So using App.ProcessMessages inside the Button1Click doesn't work because the messages are still being processed long after Button1Click is finished. And the ProcessMessages was needed in post_message() for the update of the memo1 (on Linux).

Executing App.ProcessMessages every 10 times in the post_message() should work to alleviate the problem somewhat, but the fact remains that recursion occurs on the Linux-platform (nibbling that 640 bytes from the stack each time).

For me it's not a problem (I'm not actively developing for Linux) but for those who do, it should be noted that multiple PostMessage in combination with Application.ProcessMessages might be a cause for problems because of the recursion.
« Last Edit: October 21, 2015, 12:02:34 pm by rvk »

Michael Collier

  • Sr. Member
  • ****
  • Posts: 253
Re: [SOLVED]error when modifying main form from indy thread
« Reply #35 on: October 22, 2015, 05:58:55 pm »
For me it's not a problem (I'm not actively developing for Linux) but for those who do, it should be noted that multiple PostMessage in combination with Application.ProcessMessages might be a cause for problems because of the recursion.

For multithreading:

Also for linux it should be noted that even without using processmessages() or exceeding the thread limit (of 11,000 ish), both postmessage() and sendmessage() cause the application to crash (even when using try/except I believe)

And for Mac developers (PowerPC Darwin), sendmessage() crashes, even without processmessages() or exceeding thread limit.

I have attached thread_sendmessage3.zip , it has a few more features for selecting processmessages() and for enabling a background timer. I also find it helpful to write to the first line of the output memo, rather than adding lines, so that scrolling isn't required.



rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: [SOLVED]error when modifying main form from indy thread
« Reply #36 on: October 22, 2015, 06:23:59 pm »
Yes, for me under Linux the PostMessage stops at around 1500. It slows down and stops (sometimes with a SIGSEGV). I didn't test your latest project yet.

Regarding the SendMessage. I think that will never work because the message is somehow executed in the same thread as where SendMessage is called from (on Linux). I tested this and as far as I could see this is still the case. So SendMessage should never be used if you're going to update something from the main thread. That is probably also the problem with SendMessage on the Mac.

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: [SOLVED]error when modifying main form from indy thread
« Reply #37 on: October 22, 2015, 06:46:28 pm »
There is something I don't understand about the new sendmessage3 example.

When I do 10.000 postmessages the thread should end relative quickly and the messages should still be processed. Could you make it visible when the thread is really finished? At this moment I only get the "Max reached" after all the postmessage-messages but I don't think that is correct. Even after the Max is reached there should still be messages being processed.

With update=none I can't get the example to crash anymore :( (or :))
« Last Edit: October 22, 2015, 06:52:21 pm by rvk »

Michael Collier

  • Sr. Member
  • ****
  • Posts: 253
Re: [SOLVED]error when modifying main form from indy thread
« Reply #38 on: October 22, 2015, 08:39:29 pm »
When I do 10.000 postmessages the thread should end relative quickly and the messages should still be processed. Could you make it visible when the thread is really finished? At this moment I only get the "Max reached" after all the postmessage-messages but I don't think that is correct. Even after the Max is reached there should still be messages being processed.

It's just an optical illusion of the list I think. I have added gettickcount() to the output, here is what I get:
postmessage :10 @291811315
OK. Thread reached max. @291811303

attached is thread-Sendmessage4.zip


Michael Collier

  • Sr. Member
  • ****
  • Posts: 253
Re: [SOLVED]error when modifying main form from indy thread
« Reply #39 on: October 22, 2015, 09:16:14 pm »
..I think I understand a bit better what you meant by the max should be displayed long before messages have been processed.

On my slow Mac the same program does as you describe, it shows the max has been reached, then continues to count up the postmessages().

Can't seem to get it to do same on linux..?

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: [SOLVED]error when modifying main form from indy thread
« Reply #40 on: October 22, 2015, 11:07:18 pm »
Yes, exactly.

And when you cancel the thread by clicking the free button, the thread is terminated but there should still be lots of messages (from PostMessage) in the queue. So you should see that these are still being processed.

I did manage it, a few days ago, to change the button to "ended" and then I could see the end and the messages were still being processed but I can't seem to recreate this with your last example. I'll try again tomorrow.

Does anyone know what queuing mechanism is used on the Mac? As I understand it, on Windows the messaging-queue from Windows is used and on Linux there is a self-made list-queue mechanism used. (Am I right about that?) What about Mac?

Michael Collier

  • Sr. Member
  • ****
  • Posts: 253
Re: [SOLVED]error when modifying main form from indy thread
« Reply #41 on: October 23, 2015, 12:42:27 am »
If you change the post_message routine so that it goes back to adding lines instead of modifying line[0]  then you get the output you expect:

Code: Pascal  [Select]
  1.     {
  2.     Memo1.Lines[0] := 'postmessage :'
  3.                     + IntToStr ( Message.lParam )
  4.                     + ' @'
  5.                     + IntToStr ( GetTickCount   )
  6.                     ;
  7.     }
  8.   add_line_to_memo( 'postmessage :'
  9.                    + IntToStr( Message.lParam )
  10.                    )                                    ;
  11.  

<output is placed on this line>
OK. Thread reached max. @306623698
postmessage :1 @306623705
postmessage :2 @306623705
postmessage :3 @306623705


Not sure why the memo couldn't sequence the output such that max is added to the list, whilst updates are made on line[0], but thought I'd let you know

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: [SOLVED]error when modifying main form from indy thread
« Reply #42 on: October 23, 2015, 03:14:03 pm »
In the first image you see that @gettick for the postmessage is older than the end-thread.
But... that should be logical. You did the GetTick in the process_message. You should save the GetTick at the moment of the PostMessage. I changed it here to show them both.

I did it like this:
Code: Pascal  [Select]
  1.         2 : begin  // PostMessage
  2.           PostMessage( form_thread_message.Handle
  3.                      , LM_POST_MESSAGE_TEST
  4.                      , GetTickCount
  5.                      , FCount+1
  6.                      )                                      ;
and
Code: Pascal  [Select]
  1.     Memo1.Lines.Add( 'postmessage :'
  2.                    + IntToStr( Message.lParam )
  3.                    + ' @'
  4.                    + IntToStr ( GetTickCount   )
  5.                    + ' posted @' + inttostr(Message.wParam)
  6.                    + ' diff = ' + inttostr(GetTickCount - Message.wParam)
  7.                    )                                    ;
In that case I get my second image.
Now you can see the PostMessage is really done before the real processing.

Somewhere around 5000 or 6000 the process slows down (or stops) and then, at once, all lines appear. But according to the log they where all added gradually. But at least it runs correctly for me. I couldn't get it to crash with the setting 1ms. When using 0ms it crashes at 1000 with a SIGABRT. When using 2ms it crashes at 7000. With what message does it crash for you?

I still have the feeling that the message-queue under Linux isn't a 100% thread-safe. (But I lack the skills to check this)
Sometimes it seems to hang and then after a few seconds all the lines appear immediately. Other times you get the SIGABRT crash.

F.Y.I. I trace the SIGABRT back to TGtkMessageQueue.Lock which is called from TGtk2WidgetSet.AppProcessMessages:
Code: Pascal  [Select]
  1. ... in Gtk2MsgQueue
  2. procedure TGtkMessageQueue.Lock;
  3. begin
  4.   inc(fLock);
  5.   if fLock=1 then
  6.     {$IFDEF USE_GTK_MAIN_OLD_ITERATION}
  7.     EnterCriticalsection(FCritSec);
  8.     {$ELSE}
  9.     g_main_context_acquire(FMainContext);  // <-- no check is lock succeeds ??
  10.     {$ENDIF}
  11. end;
  12.  
  13. ... in gtk2widgetset.inc
  14.    // then handle our own messages
  15.     while not Application.Terminated do begin
  16.       fMessageQueue.Lock;      // <--------- SIGABRT
« Last Edit: October 23, 2015, 03:56:27 pm by rvk »

Michael Collier

  • Sr. Member
  • ****
  • Posts: 253
Re: [SOLVED]error when modifying main form from indy thread
« Reply #43 on: October 27, 2015, 01:51:04 pm »
Thanks for the help on this, I have opted to use synchronize for my own application and it is able to run 24+ hours without error.

The lazarus wiki multithreading page mentions/promotes postmessage() & sendmessage()

Quote
one good option being the usage of SendMessage or PostMessage

http://wiki.freepascal.org/Multithreaded_Application_Tutorial#Using_SendMessage.2FPostMessage_to_communicate_between_threads

Maybe a caveat needs adding for linux/mac users? It would have helped me when I first encountered this (bug?)

taazz

  • Hero Member
  • *****
  • Posts: 5363
Re: [SOLVED]error when modifying main form from indy thread
« Reply #44 on: October 27, 2015, 02:24:39 pm »
O wow, for Windows you can indeed remove the ProcessMessages in my simple_postmessage.zip. BUT... on Linux you DO need the ProcessMessages. Otherwise the lines in memo1 don't get displayed until ALL messages are handled !!!
erm that explanation doesn't make sense to me. Either the messages are not processed or they are processed. IF they are processed but not showing on screen try adding a call to invalidate in the process procedure instead of calling the processMessages and see if that helps at all. If it does help then the problem can be in any part of the framework. If it doesn't help and still you get all the lines at the end then there is probably a deadlock somewhere, not so easy to debug but far smaller area to check.
 
My simple_postmessage is a simple program without threads but apparently Windows sees the chance to update the memo1 but Linux does not. So the messaging handling in Linux is really really different :)
Is it linux or gtk? If compiled as a QT application does it have the same problem? I'm asking because I think that QT hsa its own message queue implementation that is similar with the windows message queue.

Oh wait I have a 32bit linux now I can test my self!! Let me do that and get back to you.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64