Recent

Author Topic: [SOLVED] How to cleanup ReadDirectoryChangesW operations properly?  (Read 16776 times)

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
At the end of my migration of an old Delphi 7 app to Lazarus Windows x64 i detected a remaining flaw with ReadDirectoryChangesW (in combination with GetOverlappedResult).

The scenario is that i have a old grown dirwatcher component for a file listview's folder elements.
The folder is same as the related folder in a directory tree.

The component, somehow adapted to Lazarus, works fine in nearly all use cases, except one.
The problamatic case is when i rename a folder in the tree (with DO_RENAME in SHFileOperationW) and _afterwards_ the parent folder of that already renamed child folder,
then the file operationi failds with error ui "already in use".

Before those rename actions the filewatcher thread is terminated  and a CancelIO done, somehting like "CancelWait" in related articles.

The problem appears to me to be that the ReadDirectoryChangesW resp. GetOverlappedResult is not cleaned up properly resp. is sticky on a folder once having renamed it so that i cannot continue on a parent folder. of it
Tried a lot of modification on the stop thread / stop ReadDirectoryChangesW, but keep to be stuck.

Yes, it's hard to say something about that without code, but it would be very hard to condense the complex thing into a little test case.
Maybe a theoretical thought might help, convering "how to finish the IOs of a ReadDirectoryChangesW properly".
Somebody encountered such?
« Last Edit: May 24, 2021, 09:35:30 pm by d7_2_laz »
Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

ASerge

  • Hero Member
  • *****
  • Posts: 2493
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #1 on: May 20, 2021, 08:29:35 pm »
Are you using CancelIO or CancelIoEx? Do you close the directory handle when the thread terminates?

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #2 on: May 20, 2021, 08:54:50 pm »
CancelIO  (if handle <> 0 and <> INVALID_HANDLE_VALUE), then CloseHandle

It's worthy to try CancelOEx for this special case?
Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

ASerge

  • Hero Member
  • *****
  • Posts: 2493
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #3 on: May 20, 2021, 10:07:16 pm »
It's worthy to try CancelOEx for this special case?
From docs: "The CancelIoEx function allows you to cancel requests in threads other than the calling thread." You didn't say where the cancellation was.

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #4 on: May 20, 2021, 10:56:02 pm »
Yes really,  that (in threads other than the calling thread) sounded promising, but i didn't notice a difference.

I tried to test the cancellation from various places. For instance from inside the control thread execute of later after having called the thread terminate.

Maybe it is better for speaking to print an extract from the coding here.  The control thread execute:

Code: Pascal  [Select][+][-]
  1. procedure CtrlThread.Execute;
  2. var
  3.   pBuffer :   Pointer;  // Buffer for return values of ReadDirectoryChangesW
  4.   dwBufLen :  DWORD;
  5.   dwRead :    DWORD;
  6.   PInfo :     PFILE_NOTIFY_INFORMATION;  
  7.   dwNextOfs : DWORD;
  8.   dwFnLen :   DWORD;
  9.   Overlap :   TOverlapped;
  10.   WaitResult: DWORD;
  11.   EventArray: Array[0..2] of THandle; // Array der Handles für WaitForMultipleObjects
  12.   pFileNameForMessage: PWideChar;
  13.   lasterrcode:Cardinal; ErrorMessage: string;
  14.  
  15. begin
  16.   try   // try .. except
  17.     Cancel_IO;
  18.     if not DirectoryExists(FsDirPath) then begin
  19.        Terminate;
  20.        exit;
  21.     end;
  22.  
  23.     FhFile:= CreateFileW(PWideChar(UTF8Decode(FsDirPath)),
  24.                         FILE_LIST_DIRECTORY or GENERIC_READ,
  25.                         FILE_SHARE_READ or FILE_SHARE_WRITE or FILE_SHARE_DELETE,
  26.                         nil,
  27.                         OPEN_EXISTING,FILE_FLAG_BACKUP_SEMANTICS or FILE_FLAG_OVERLAPPED,
  28.                         0);
  29.     if (FhFile = INVALID_HANDLE_VALUE) or (FhFile = 0) then begin
  30.         Terminate;
  31.         exit;
  32.     end;
  33.  
  34.     FileEvent:=CreateEvent(nil,FALSE,FALSE,nil);
  35.     Overlap.hEvent:=FileEvent;
  36.     TermEvent:=TEvent.Create(nil,FALSE,FALSE,TermEvName);
  37.     SuspEvent:=TEvent.Create(nil,FALSE,FALSE,SuspEvName);
  38.     EventArray[0]:=FileEvent;
  39.     EventArray[1]:=TermEvent.Handle;
  40.     EventArray[2]:=SuspEvent.Handle;
  41. }
  42.     EventArray[0] := FileEvent;
  43.     EventArray[1] := plocaleventrec(TermEvent.Handle)^.Fhandle;
  44.     EventArray[2] := plocaleventrec(SuspEvent.Handle)^.Fhandle;
  45.  
  46.     dwBufLen := 65535;      // Dirwatch:  BufferSize (=32) * SizeOf(TFILE_NOTIFY_INFORMATION);
  47.     pBuffer := AllocMem(dwBufLen);
  48.     try
  49.       while not terminated do begin
  50.         dwRead := 0;
  51.           if ReadDirectoryChangesW(FhFile, pBuffer, dwBufLen,
  52.                                  prWatchSubTree,    //  true or false didn't make a difference here
  53.                                  FFilter,        //FILE_NOTIFY_CHANGE_FILE_NAME or FILE_NOTIFY_CHANGE_DIR_NAME,
  54.                                  @dwRead, @Overlap, NIL) then
  55.         begin
  56.           WaitResult := WaitForMultipleObjects(3, @EventArray, FALSE, infinite);
  57.           case WaitResult of
  58.             WaitDir:
  59.                 begin                 // WAIT_OBJECT_0
  60.                   if GetOverlappedResult(FhFile, Overlap, dwRead, False) then begin
  61.                      PInfo:= pBuffer;
  62.                      if dwRead = 0 then begin   // check overflow
  63.                         ErrorMessage := SysErrorMessage(ERROR_NOTIFY_ENUM_DIR);
  64.                         SignalError(ErrorMessage, ERROR_NOTIFY_ENUM_DIR);
  65.                      end;
  66.                      repeat
  67.                        dwNextOfs := PInfo.dwNextEntryOffset;
  68.                        FAction := PInfo.dwAction;
  69.                        dwFnLen := PInfo.dwFileNameLength;
  70.                        FsFileName := WideCharLenToString(@PInfo.dwFileName,dwFnLen div 2);
  71.  
  72.                        GetMem(pFileNameForMessage, dwFnLen + SizeOf(WideChar));
  73.                        Move(PInfo.dwFileName, Pointer(pFileNameForMessage)^, dwFnLen);
  74.                        PWord(Cardinal(pFileNameForMessage) + dwFnLen)^ := 0;
  75.  
  76.                        PostMessage(FWndHandle, WM_DIRWATCH_NOTIFY, FAction, LParam(pFileNameForMessage));
  77.  
  78.                        pChar(PInfo) := pChar(PInfo)+dwNextOfs;
  79.                      until dwNextOfs=0;
  80.                   end else begin
  81.                       lasterrcode := GetLastError;
  82.                       ErrorMessage := SysErrorMessage(lasterrcode);
  83.                       SignalError(ErrorMessage, lasterrcode);
  84.                       break;
  85.                   end;  // GetOverlappedResult
  86.                 end;
  87.             WaitTerm:                      // WAIT_OBJECT_0+1
  88.                  // dont call Terminate  again here
  89.               ;
  90.             WaitSusp: Suspend;             // WAIT_OBJECT_0+2
  91.             else
  92.                 break;
  93.           end;    // case WaitResult ReadDirectoryChangesW
  94.         end else begin
  95.            ErrorMessage := SysErrorMessage(GetLastError);
  96.            SignalError(ErrorMessage);
  97.         end;
  98.       end;
  99.     finally    
  100.       FreeMem(pBuffer,dwBufLen);
  101.       if FhFile <> 0 then
  102.          if FhFile <> INVALID_HANDLE_VALUE then begin
  103.             CancelIOEx(FhFile, 0);   // or nil, no change
  104.             CloseHandle(FhFile);
  105.             FhFile := 0;
  106.          end;
  107.       SendMessage(FWndHandle, WM_DIRWATCH_THREADTERMINATED, 0, 0);
  108.       with TFldrControl(Owner) do
  109.            ThrdTerminated := True;
  110.     end;   // try
  111.  
  112.   except
  113.      on E :Exception do begin
  114.         ErrorMessage := E.Message;
  115.         SignalError(ErrorMessage);
  116.      end;
  117.   end;
  118.  
  119. end;
  120.  
Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

ASerge

  • Hero Member
  • *****
  • Posts: 2493
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #5 on: May 21, 2021, 05:51:52 pm »
Let's go back to the beginning to better understand.
Let the directory tree look like this: Level_0_A\Level_1_A\Level_2_A\. You start monitoring Level_0_A from subtrees (by starting a thread). At some point, you stop monitoring (via stopping the thread).
Then rename Level_2_A to Level_2_B (for example) - there are no errors.
Then rename Level_1_A to Level_1_B - an error occurs.
Do I understand correctly?

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #6 on: May 21, 2021, 07:46:32 pm »
Hm, maybe it's easier with a small picture -> attached pic1.
And for simpliness, for the moment forget all monitoring of the tree itself
and think in terms of: monitoring of one single dedicated folder, which's filelist does appear on the right side.

(i need the ReadDirectoryChangesW only for the monitoring of a single folder. That needs much more details information
about changes, ie. file size, date, attribues etc., which would not be needed and overwhelming for a directory tree monitoring at all).

So, clicking onto child node "Level2_A", monitoring will start for folder Level2_A, who's details are needed for the file list informations.

Now: edit caption on the tree node "Level2_A" and rename it to "Level2_A_test".
The thread will be stopped, waiting until terminated, CancelIO and CloseHandle.
Then, after a certain security delay (avoid timing issues), SHFileOperationW is called for to rename the folder.
If successful, the changed folder name will be passed to the file list and the monitoring will restart for the new folder name "Level2_A_test".

Now, click on parent node: Level1_A.
Monitoring is stopped and restarted --> now for the Level1_A folder, and it's details will be displayed in the file list.

Now, When editing Level1_A eg. to "Level1_A_test". SHFileOperationW will fail with "already in use". That was not the case with Delphi 7 (resp win x32?).

The other way round vice versa:
Rename the parent folder Level1_A to Level1_A_test first, and afterwards it's child node (sub folder) Level2_A to Level2_A_test: this does NOT lead to locking  issues.

In short: it appears to me that after stopping thread/monitoring for a sub filder, it will be still locked by ReadDirectoryChangesW, forbidding to rename it's parent.
What is needed to achieve is: after monitoring of a folder, get rid of the locking by the monitoring mechanism. Unfortunately i got no clue why this locking happens. So any idea about possible causes is very appreciated.

What i even tried;  directly after a folder rename (with SHFileOperationW):  get the its handle using CreateFileW and do a CancelIOEx(FFileHandle, 0)  //  (0 (or nil?) for to stop _all_ iI/O requests onto this handle). No difference.

Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

ASerge

  • Hero Member
  • *****
  • Posts: 2493
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #7 on: May 22, 2021, 12:17:39 pm »
There may be a problem in the SHFileOperationW function. Try renaming the directory using the MoveFile function.

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #8 on: May 23, 2021, 02:02:18 pm »
Hm, wouldn't have suspected that side, but, after test, with MoveFile basically it did behave the same (already in use within the scenario described).
So i did spend some time to prepare a minimalistic test case, and at least i can say it did reproduce the issue indeed.
Very simple Gui, 2 entry fields representing a parent and a sub folder for to paste in,
and, if valid, one can select parent or child by button(analogous to treenode click)  and then trigger a simple rename action on it.
Instrutions are in a memo field.  -> project attached

When preparing that code, i came more and more close to the impression that the issue might be due to an inproper thread termination so that the ReadDirectoryChangesW stays sticky on an old folder and hinders to do a rename of the parent after having renamed a child.
Maybe Lazarus or x64 may require a slightly different coding here. But where ..

Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

ASerge

  • Hero Member
  • *****
  • Posts: 2493
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #9 on: May 23, 2021, 08:13:36 pm »
After changing in Fldrwatch_buffered.pas
Code: Pascal  [Select][+][-]
  1.     EventArray[1] := plocaleventrec(TermEvent.Handle)^.Fhandle;
  2.     EventArray[2] := plocaleventrec(SuspEvent.Handle)^.Fhandle;
to
Code: Pascal  [Select][+][-]
  1.   EventArray[1] := THandle(TermEvent.Handle);
  2.   EventArray[2] := THandle(SuspEvent.Handle);
everything works without errors.

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #10 on: May 23, 2021, 09:50:57 pm »
Yes ASerge, that brings the rename capability back (and proves it's not an issue withReadDirectoryChangesW).

Unfortunately now it does hinder the monitoring completely, as the WaitResult of WaitForMultipleObjects now will bring an WAIT_FAILED  (=4294967295), where GetLastError is 6 = ERROR_INVALID_HANDLE.

Curious; exactly that had been my problem in https://forum.lazarus.freepascal.org/index.php/topic,53887.msg399495.html#msg399495
which leaded to the cast to be replaced.

You can verify that with the testcase by: selecting one directory,  do a file change in this directoy ie. copy a file herein, and inspect the WaitResult of WaitForMultipleObjects.
Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #11 on: May 24, 2021, 02:10:13 pm »
I've the following picture:

// 27.03.2021: https://forum.lazarus.freepascal.org/index.php/topic,53887.msg399495.html#msg399495
Code: Pascal  [Select][+][-]
  1. EventArray[1] := plocaleventrec(TermEvent.Handle)^.Fhandle;
  2. EventArray[2] := plocaleventrec(SuspEvent.Handle)^.Fhandle;

Remaining problem: ReadDirectoryChangesW does work fine, but it keeps locking the folder ... because the thread does not terminate correctly?

//  23.05.2021 this thread   https://forum.lazarus.freepascal.org/index.php/topic,54704.0.html
Code: Pascal  [Select][+][-]
  1. EventArray[1] := THandle(TermEvent.Handle);
  2. EventArray[2] := THandle(SuspEvent.Handle);

Problem: WaitForMultipleObjects fires WAIT_FAILED  (=4294967295), where GetLastError is 6 = ERROR_INVALID_HANDLE
Monitoring does not happen at all,  and so the folder will not be locked of course

I extended and corrected the test case a bit so that it shows up a minimal monitor reporting for to prove that this really keeps to work, and throws a message if the nandle for the terminate event is invalid.

Project attached.

Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #12 on: May 24, 2021, 09:33:54 pm »
Finally i found it:  one line to change:

Code: Pascal  [Select][+][-]
  1.  // PulseEvent(StopEvent.Handle);   //  With Delphi 7
  2. PulseEvent(THandle(StopEvent.Handle));  // Previous attempt with Lazarus
  3.  
  4. // ---- replace by:  (SOLUTION):
  5.  PulseEvent(plocaleventrec(StopEvent.Handle)^.Fhandle );

Thread's termination will be fired,  "while not terminated" loop can be left, folder will be freed and parent folder can be renamed.
For everybody who is interested the corrected test case is attached.
Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

ASerge

  • Hero Member
  • *****
  • Posts: 2493
Re: How to cleanup ReadDirectoryChangesW operations properly?
« Reply #13 on: May 26, 2021, 06:10:52 pm »
Code: Pascal  [Select][+][-]
  1. // ---- replace by:  (SOLUTION):
  2.  PulseEvent(plocaleventrec(StopEvent.Handle)^.Fhandle );
It is a pity that TEvent.Handle is not compatible with Delphi. This leads to such errors.

About the mistakes. In the above code, I didn't see where dwBufLen is initialized. Apparently the random value from the stack turns out to be quite large.

In addition to the differences between Lazarus and Delphi, I will give an example (I did it quickly) where such monitoring can be carried out without additional threads in Lazarus.

d7_2_laz

  • Hero Member
  • *****
  • Posts: 660
Re: [SOLVED] How to cleanup ReadDirectoryChangesW operations properly?
« Reply #14 on: May 26, 2021, 09:50:17 pm »
Hello ASerge, thanks again for your attention!

About the dwBufLen initialization, unfortunately it had become a victim of editorial cuts from obsolete stuff in the first zip file.
I noticed that soon (ouch) and upfrom test_ReadDirectoryChangesW_2.zip it was in again
 dwBufLen := 65535;
apparently close to yours: CBufferSize = 64 * 1024;

The intention of my own updates of the source i once found and adapted many years ago simply had been to report file object changes in a bulk. For to avoid frequent individual screen updates by the using application (think of e.g. inserts or deletions of dozends of objects at once).
I'm happy it does work again smoothly, but without the chance to communicate within this great forum i wouldn't have had a chance.
It's really amazing that an old source basically kept working when doing the migration to Lazarus x64, after changing a very few couple of things.

I had not the possiblity to take a closer look at your example yet, but will try it the next days
and rethink which use case scenario exactly might have had motivated to use threads for that and if i could avoid them.
Lazarus 4.4  FPC 3.2.2 Win10 64bit  ==> Lazarus 4.6  FPC 3.2.2 Win11 64bit

 

TinyPortal © 2005-2018