Lazarus

Programming => General => Topic started by: user5 on November 24, 2020, 02:54:26 am

Title: Text file problem
Post by: user5 on November 24, 2020, 02:54:26 am
    I have a problem with text files that only happens with Windows 10, not with other operating systems. There's no point in including the code because there's way too much of it and it's not relevant because the problem occurs only randomly. I have no where to turn except for this forum.
    The program creates a text file (image5.txt) and absolutely no changes are made to that file until it's deleted. In most cases the file is deleted fine but on rare unpredictable occasions under the exact same conditions the deletion fails because Windows 10 'thinks' that the file is open. I know for a fact that neither the program nor any other application has opened that text file.
    Of course the reason that the file is created is so that at some point it can be opened with AssignFile, Reset and Readln but this error can happen even before the file has been read by the program. The text file is created in a form that is shown by the main program.
    When this happens it is very bad since the only thing that frees up that file is to shut the program down. I need some kind of failsafe operation to run when the deletion fails. It only happens every once in a while and it's impossible to recreate the problem on demand.
    I've had the program run the try statement to close the TextFile variable that created image5.txt but this too has failed. Is there some way for the program to find out what opened the file?
    I've tried to use Command Prompt and have even considered using a batch file to delete, unlock or close the file but nothing works. Does anyone know some way to force close or delete a file that is open by some unknown action?
Title: Re: Text file problem
Post by: user5 on November 24, 2020, 03:13:14 am
    I forgot to mention that an old version of Lazarus was used to compile the program which was run on Windows 10. The program runs fine on Windows 10 except for this problem. I suppose that I could use my new version of Lazarus to create an external program that creates the text file and have the old main program launch that external executable instead of the main program launching another form but I don't know if that would fix the problem.
Title: Re: Text file problem
Post by: jamie on November 24, 2020, 04:21:42 am
its possible  you have a left over IO results or something …

before doing Pascal IO I do this..

InOutRes := 0;

if for some reason that isn't 0 when you start the next pascal IO it could fail silently if you have the IO check turned off and thus not close the file.

 The file does need to be closed...

 also, if the OS is scanning the file at the time it can't be deleted.
Title: Re: Text file problem
Post by: user5 on November 24, 2020, 04:41:00 am
    Very interesting jamie. So I assume that the following is what you meant, huh?

   
Code: Pascal  [Select][+][-]
  1. InOutRes := 0;
  2. if FileExists('image5.txt') = true then
  3.  DeleteFile('image5.txt');

    I'll bet that this might be the solution but if it isn't then I could have the main program create tempfile.txt and send it to a small external program that creates image5.txt.
    When the external program terminates then image5.txt should be free.
    Thank you so much!!!!
Title: Re: Text file problem
Post by: Thaddy on November 24, 2020, 08:17:49 am
Code: Pascal  [Select][+][-]
  1. if FileExists('image5.txt') then
  2. begin
  3.  InOutRes := 0;
  4.  DeleteFile('image5.txt');
  5. end;
= true is not necessary and resetting InOutRes is only relevant for the delete operation.

Furthermore you can test if a file is really closed like so:
Code: Pascal  [Select][+][-]
  1. // Suppose your text file hande is F
  2. if TTextRec(F).mode <> fmClosed then closefile(f);{first flush it on write mode}
Don't forget to check that IOresult is really zero after calling closefile. If not the file is locked, so you need a kind of retry mechanism.
Which leads us to this small example (without using sysutiles):
Code: Pascal  [Select][+][-]
  1. {$I-}
  2. var
  3.   F:Text;
  4.   E:word;
  5. begin
  6.   Assign(F,'image5.txt'); // this has no IOResult
  7.   Reset(F);
  8.   E:=IOResult;  // Always buffer IOResult, because it resets on read
  9.   if E  = 2 then writeln('File does not exist') else
  10.     if E = 0 then
  11.     begin
  12.       if TextRec(F).mode <> fmClosed then close(F);
  13.       E:= IOResult;
  14.       if E = 0 then
  15.       begin
  16.         Erase(F);
  17.         E := IOResult;
  18.         if E <> 0 then writeln('Can not delete file, IOResult is ', E);
  19.       end else writeln('Can not close file, IOResult is  ', E);
  20.     end else writeln('Operation failed, IOResult is  ', E);
  21. end.
Of course this is not complete there are more things to check, see:
https://www.freepascal.org/docs-html/rtl/system/ioresult.html
But it covers most cases and the IOresult reason can always be checked since it is buffered.
 
Title: Re: Text file problem
Post by: user5 on November 24, 2020, 08:56:02 am
    Thaddy, I know that "= true" is not necessary but I've written code in that old version of Lazarus in which leaving it out caused problems so I acquired the habit of being redundant in this case ever since. It doesn't hurt anything.
    As for InOutRes, I assume that what you are saying is that it has to be used with each delete or other such operation.
    I plan to wait until the text file problem happens again before I try this solution so that I know whether or not it fixes the error, else I might end up wondering if I fixed it or not. I'll use something like what is shown below when the problem happens again.
    Thanks for your reply and thanks for the code that tests if a file is open or not. I've never seen that before and I'll keep it in mind.

   
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button25Click(Sender: TObject);
  2. begin
  3.  InOutRes := 0;
  4.  DeleteFile('image5.txt');
  5.  if FileExists('image5.txt') = true then
  6.   memo2.text := 'The deletion failed.'
  7.  else
  8.   memo2.text := 'The deletion succeeded.';
  9. end;
Title: Re: Text file problem
Post by: Thaddy on November 24, 2020, 09:33:57 am
I have adapted my example a bit to cover most scenario's
Title: Re: Text file problem
Post by: marcov on November 24, 2020, 10:20:09 am
This is a known (windows) issue.  Software monitoring directories (shell plugins like tortoisesvn) and Windows defender can acerbate this.

In a distant past I wrote and erased metafiles (.wmf) under Windows XP, and an the explorer preview function could do funky things, and in the end I had to move everything to a thread. 

FPC itself suffered from it in its build tools (fpmake), basically it tries multiple times with some pauzes, from packages/fpmkunit/src/fpmkunit.pp :

Code: Pascal  [Select][+][-]
  1. procedure TBuildEngine.SysDeleteFile(Const AFileName : String);
  2. var retries : integer;
  3.     res : boolean;
  4. begin
  5.   if not FileExists(AFileName) then
  6.     Log(vldebug,SDbgFileDoesNotExist,[AFileName])
  7.   else
  8.     begin
  9.       retries := 2;
  10.       res := SysUtils.DeleteFile(AFileName);
  11.       while not res and (retries>0) do
  12.         begin
  13.            log(vlWarning, SWarnRetryDeleteFile, [AFileName]);
  14.            sleep(5000);
  15.            dec(retries);
  16.            res := SysUtils.DeleteFile(AFileName);
  17.         end;
  18.      if not res then
  19.        Error(SErrDeletingFile,[AFileName])
  20.      else
  21.        Log(vlInfo,SInfoDeletedFile,[AFileName]);
  22.    end;
  23. end;
  24.  

Similarly Removedir is retried 3 times too:

Code: Pascal  [Select][+][-]
  1.   // There were reports of RemoveDir failing due to locking-problems. To solve
  2.     // these the RemoveDir is tried three times, with a delay of 5 seconds. See
  3.     // bug 21868
  4.     retries := 2;
  5.     result := RemoveDir(ADirectoryName);
  6.     while not result and (retries>0) do
  7.       begin
  8.         log(vlWarning, SWarnRetryRemDirectory, [ADirectoryName]);
  9.         sleep(5000);
  10.         dec(retries);
  11.         result := RemoveDir(ADirectoryName);
  12.       end;
  13.  
Title: Re: Text file problem
Post by: user5 on November 24, 2020, 01:52:45 pm
    Thaddy and marcov, that code you posted is really amazing. You guys are better than me.
    I tried and tried to make the error happen again but it hasn't happened yet. I should have known. I did a number or reboots so maybe that had something to do with it. That and what marcov said suggests that something other than the program alone failed. This potential bug is so bad that the program should be ready for it so if the program fails to delete image5.txt etc. (filename) then the following will happen (If you don't disapprove):

   
Code: Pascal  [Select][+][-]
  1. procedure TForm1.RemoveTextFile(var filename:string);
  2. var
  3.   F:Text;
  4.   E:word;
  5.   retries : integer;
  6.   res : boolean;
  7. begin
  8.  form199.statictext1.caption := 'An error has happened. Please wait while the program fixes it.';
  9.  form199.show;
  10.  delay(2200);
  11.  application.processmessages;
  12.  
  13.  InOutRes := 0;
  14.  DeleteFile(filename);
  15.  
  16.  //If filename still exists then do the following.
  17.  if FileExists(filename) = true then
  18.   begin
  19.    Assign(F,filename); // this has no IOResult
  20.    Reset(F);
  21.    E:=IOResult;  // Always buffer IOResult, because it resets on read
  22.    if E = 0 then
  23.     begin
  24.      if TextRec(F).mode <> fmClosed then
  25.       close(F);
  26.      E:= IOResult;
  27.      if E = 0 then
  28.       Erase(F);
  29.     end;
  30.   end
  31.  else
  32.   begin
  33.    form199.statictext1.caption := 'The error has been corrected.';
  34.    delay(2200);
  35.    application.processmessages;
  36.    form199.hide;
  37.    exit;
  38.   end;
  39.  
  40.  //If filename still exists then do the following.
  41.  if FileExists(filename) = true then
  42.   begin
  43.    retries := 2;
  44.    res := SysUtils.DeleteFile(filename);
  45.    while not res and (retries>0) do
  46.     begin
  47.      log(vlWarning, SWarnRetryDeleteFile, [filename]);
  48.      sleep(5000);
  49.      dec(retries);
  50.      res := SysUtils.DeleteFile(filename);
  51.     end;
  52.   end
  53.  else
  54.   begin
  55.    form199.statictext1.caption := 'The error has been corrected.';
  56.    delay(2200);
  57.    application.processmessages;
  58.    form199.hide;
  59.    exit;
  60.   end;
  61.  
  62.  //If filename still exists then do the following.
  63.  if FileExists(filename) = true then
  64.   begin
  65.    form199.hide;
  66.    form219.statictext1.caption := 'A very rare Windows related (Outside) bug occurred. Please shut down and restart the program.';
  67.    form219.show;
  68.   end
  69.  else
  70.   begin
  71.    form199.statictext1.caption := 'The error has been corrected.';
  72.    delay(2200);
  73.    application.processmessages;
  74.    form199.hide;
  75.   end;
  76.  
  77.  
  78. end;
Title: Re: Text file problem
Post by: Thaddy on November 24, 2020, 05:01:58 pm
As both Marco and I stated: you should have some retry mechanism.
Not in my code - just the remark - but Marco's code shows a retry mechanism.
Only when the number of retries exceeds a certain limit, you should warn the user that something is wrong.
Title: Re: Text file problem
Post by: marcov on November 24, 2020, 05:13:48 pm
Note that the delays are relatively long. If this is a problem (e.g. GUI not responsive) you need to use a thread for it.
Title: Re: Text file problem
Post by: user5 on November 24, 2020, 08:12:05 pm
    Everything compiles fine though I had to change assign and close to assignfile and closefile respectively. I also removed the remaining log report in the code that I posted. The text file/s that I have been talking about are each associated with a picture or video import in the program so instead of advising the user to take the serious step of shutting down the program if the file can't be deleted then the program will simply disable the affected import partially so that the user can still use it.
    Thank you so much. This forum saved my bacon once again.
Title: Re: Text file problem
Post by: Thaddy on November 24, 2020, 08:16:18 pm
    Everything compiles fine though I had to change assign and close to assignfile and closefile respectively.
That should - almost - never be the case. You do not need xxxFile in most cases, just system and no dependency on sysutils.
But it does not hurt, except larger binaries.
Title: Re: Text file problem
Post by: user5 on November 25, 2020, 12:45:31 am
    I hadn't planned on adding any more posts on this thread but just so that anyone who is interested knows, the program discussed in previous posts finally failed to correctly delete the relevant text file so I applied InOutRes := 0 and then tried to delete it again.
    Unfortunately the file still wouldn't delete but I have hopes on the procedures that Thaddy and marcov provided. Man, that text file was locked like a son of a gun.
    As for assign vrs. assignfile Thaddy, all I know is that the code wouldn't compile unless I used assignfile and closefile. I might be missing something here.
    You mentioned larger binaries and I guess that you mean larger program files. The program in question is pretty big. Thanks again.
Title: Re: Text file problem
Post by: winni on November 25, 2020, 02:09:04 am
Man, that text file was locked like a son of a gun.
   

Hi

If  a file (any kind) is locked by the OS or other programms on a Windows System there s no chance but to unlock the file.
From the users point of view there are some free UnLockers around.

From programmers view it seems to be hard stuff. You need Admin rights, System rights and whatever.
I know there is some C code flying around but I could not find it.

Winni
Title: Re: Text file problem
Post by: Thaddy on November 25, 2020, 06:26:32 am
There is also some Pascal code around. I seem to remember Bart? has some code.
There is also Pascal code for whoislocking.
I used it in the past, I guess it is win32 only.
Title: Re: Text file problem
Post by: PascalDragon on November 25, 2020, 07:25:02 am
    Everything compiles fine though I had to change assign and close to assignfile and closefile respectively.
That should - almost - never be the case. You do not need xxxFile in most cases, just system and no dependency on sysutils.
But it does not hurt, except larger binaries.

It's necessary to rename calls to Assign to System.Assign or AssignFile inside descendants of TPersistent due to their own Assign method and Close to System.Close or CloseFile inside descendants of TCustomForm due to its own Close method.

Also the *File overloads are part of the ObjPas unit, not SysUtils and are only really forwarders, so that don't hurt that much.
Title: Re: Text file problem
Post by: loaded on November 25, 2020, 07:35:58 am
A simple solution method I use for problems that may occur in systemic situations that improve after a certain period of time;

Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   err:Boolean=false;
  4. begin
  5.   repeat
  6.     try
  7.     // Start of Creation, Deletion Operations
  8.     //....................
  9.     //....................
  10.     //....................
  11.     //....................
  12.     //....................
  13.     //....................
  14.     //....................
  15.     //....................
  16.     //End of Creation, Deletion Operations
  17.     err:=false;
  18.     except
  19.       on E : Exception do
  20.       begin
  21.       err:=true;
  22.       caption:=E.ClassName+' error raised, with message : '+E.Message;
  23.       end;
  24.     end;
  25.     Application.ProcessMessages;
  26.   until not err;
  27.   caption:=' Transaction Completed Successfully' ;
  28. end;  
  29.  
Title: Re: Text file problem
Post by: Thaddy on November 25, 2020, 09:06:09 am
Since he is using old school IO routines I strongly disagree with using exceptions in this case.
Both code bloat and slow in a loop that can be predicted to fail sometimes.
Although Marco's code is also predictably slow because of his long timeout value, as he wrote himself.
Title: Re: Text file problem
Post by: marcov on November 25, 2020, 09:10:36 am
Since he is using old school IO routines I strongly disagree with using exceptions in this case.
Both code bloat and slow in a loop that can be predicted to fail sometimes.
Although Marco's code is also predictably slow because of his long timeout value, as he wrote himself.

In my experience these locks range from 50ms to 500ms. Retrying immediately then only causes busywaiting. This is the reason for the timeouts.
Title: Re: Text file problem
Post by: user5 on November 25, 2020, 08:30:49 pm
    Thanks winni, for looking for that C code. The code that I got from this thread was a valiant effort and I'm very appreciative but testing showed that they will not work except to tell you if a file is open or not. Like you say, aside from very hard-to-find solutions the only way to delete a text file is to first close it and to close you have to know what TextFile variable opened it.
    I changed my program so that the text files that I had trouble with are created by an external program that is launched by the main program. When the external program closes then the files are freed up. After the files are created then there is only one page in the main program that opens them and it opens them for reading only. The only other thing that is done to them is to delete them.
    I still don't know what caused the text files to have an open status but I think that the error is fixed. If I have any more problems then I'll use try statements to make sure that the files are closed before they are deleted, using the TextFile variables that I know opened them. If a file is open then the try statement will close it. If it's not open then the exception error is handled by the try statement and no harm is done.
Title: Re: Text file problem
Post by: dseligo on November 27, 2020, 01:38:32 am
Like you say, aside from very hard-to-find solutions the only way to delete a text file is to first close it and to close you have to know what TextFile variable opened it.
From this I assume you are opening multiple files or using multiple variables for files. This opens possibility that some file is not closed because of an error in your program. I suggest that you log in a file all you opening and closing files along with variable names (and/or maybe unique ID). When error occurs you take that file and look for pairs - an unmatched pair is location of you problem.

Something like this:
Code: Pascal  [Select][+][-]
  1. const LogName='mylog.txt';
  2.  
  3. procedure LogInit;
  4. var fLog:TextFile;
  5. begin
  6.   AssignFile(fLog,LogName);
  7.   Rewrite(fLog);
  8.   CloseFile(fLog);
  9. end;
  10.  
  11. procedure Log(s:String);
  12. var fLog:TextFile;
  13. begin
  14.   AssignFile(fLog,LogName);
  15.   Append(fLog);
  16.   WriteLn(fLog,FormatDateTime('yyyy-mm-dd hh:nn:ss.zzz ',Now),s);
  17.   CloseFile(fLog);
  18. end;
  19.  
  20. procedure MyCreateFile(var f:TextFile; sFileVarName, sFileName:String; ID:Integer);
  21. begin
  22.   AssignFile(f,sFileName);
  23.   Rewrite(f);
  24.   Log('create '+sFileVarName+' '+sFileName+' '+Inttostr(id));
  25. end;
  26.  
  27. procedure MyCloseFile(var f:TextFile; sFileVarName:String; ID:Integer);
  28. begin
  29.   CloseFile(f);
  30.   Log('close '+sFileVarName+' '+Inttostr(id));
  31. end;
  32.  
  33. ...
  34. LogInit; // at start of the program
  35. ....
  36.  
  37. //
  38. MyCreateFile(fTest,'fTest','testfile1.txt',1);
  39.  
  40. ...
  41.  
  42. MyCreateFile(fTest2,'fTest2','testfile2.txt',3);
  43.  
  44. ...
  45.  
  46. MyCloseFile(fTest,'fTest',2);
  47.  
  48. ...
  49.  
  50. MyCloseFile(fTest2,'fTest2',4);
  51.  

If you do this everywhere in your program where you use AssignFile, CloseFile, Rewrite, Reset or Append you should find if and where you left your file open.
Title: "Text file problem" update
Post by: user5 on December 01, 2020, 10:30:06 pm
    On November 24 I posted a note ("Text file problem") about a problem I was having in which a text file that was legitimately closed by my program ended up being considered still open by Windows. marcov said "This is a known (Windows) issue". This problem was unpredictable and would only happen every once in a while but testing did show some interesting results. Sometimes when I run Windows Explorer I notice that its progress bar runs very, very slowly and never really ends, even going beyond the end of its containment box. Rebooting the computer usually fixes this.
    When the progress bar is behaving like this then the chances of the error occurring are vastly increased. My computer is a Windows 7 machine that was upgraded to Windows 10 and it seems to behave oddly at times. If I use YouTube a lot then the computer crashes, shutting itself off. I changed the power settings but it still happens if I play many YouTube videos.
    I did some research on the slow progress bar. I have been using Task Manager to check processing performance and I ended up changing Windows indexing options but I don't know if this is the solution to the text file problem.
    I did find an interesting Lazarus Forum topic titled "Delphi, Lazarus Pi Performance Benchmark" and I was wondering if it might be possible to programmatically do some kind of performance check before the program creates and uses those text files. My program is currently set up to provide work around options when the error happens but I haven't been able to solve the problem except by shutting the program down and thus freeing up the file/s involved.  I'm really just groping in the dark here.
Title: Re: "Text file problem" update
Post by: dseligo on December 03, 2020, 01:27:48 am
My computer is a Windows 7 machine that was upgraded to Windows 10 and it seems to behave oddly at times.
Are you experiencing this "Text file problem" only on this one computer?
If you have other issues with it, maybe it is time to reinstall Windows?
TinyPortal © 2005-2018