Lazarus 4.6 may erase your hard disk during "cleanup" after compilations

[Benoni_Edin]

Once a program has been compiled in Lazarus (whether debug or release version), Lazarus deletes temporary files generated during compilation.

There seems to be a very dangerous glitch in the way this is achieved: it can cause not only the temporary files to be deleted but also parent folders, e.g., a whole folder tree representing a Lazarus poroject. In fact, I just got most of my D:\DropBox content deleted (>80 000 files) while compling a program in Lazarus.

I don't know how Lazarus 4.6 cleans up temporary files but my guess is that what triggers its misbehaviour is that DropBox locks files that are changed and then uploads them to its web server.

The problem is that when Lazarus 4.6 is unable to delete a temporarily locked file, it doesn't simpoly ignores this file but to recursively starts to delete files up the folder tree.

I can, of course, restore the deleted using DropBox, but it's time consuming (a really dangerous behavior and very annoying.

I have used DropBox for > 10 years and never had an similar problems with Lazarus 3.8.

In short, the fault is definitely with the compiler of Lazarus 4.6.

I have worked around this issue by simply excluding the relevant "Debug" and the "Release" folders from monitored by DropBox. Yet, today it happened again and Lazarus more or less wiped out everyting in D:\DropBox.

A safe solution is, of course, to manually inactivate DropBox completely while working in Lazaurs 4.6 but then I loose the ability to regret silly coding.

For an non-professional programmer like me, it seems strange that the names of generated temporary files are not collected and once the compilation has complete these specific files are deleted. Why rely on something that recursively deleted files in a file tree.

Anyhow, please look into this ASAP.
Also, if anyone can suggest a solution that I can implement without inactivating DropBox, it would be very welcome!

/Ben

Benoni B. Edin (Benoni.Edin@gmail.com)
Örnsköldsvik, Sverige

[Martin_fr]

I have not worked on the compilation parts of the IDE, so I don't know anything about that code. But since your message may not be seen here, I advice to report it on the bug tracker.

You will probably have to provide more details, though I don't know.

I did a quick test, and set breakpoints on anything I could find that would delete something. => Including entry points in the kernel for DeleteFileW

I then compiled and run a project. It called deletefile to delete the oldest backup. That is expected and correct.

I did not see any other call to delete any file or dir.

I may have missed some method that does the job... Or maybe your IDE is configured differently, and it only happens with that different config?

As I said, I don't know, but with the level of detail (or lack thereof), it may turn out that no one can reproduce it.

[marcov]

The dropbox angle is of course the problem. It is hard to understand what happens there. I guess a first step would be to reproduce it on a micro scale, narrow it down to specific operations, and then maybe diagnose with additional logging or syscall/api monitoring.

But for that it first needs to be narrowed down and become reproducable, which is probably hard if transient locks are the origin.

If it deletes arbitrary files, wildcards must be involved, and that is actually relatively rare.  Most cleanup deletes a list of specific files. Only the build processes of FPC/Lazarus themselves delete unit dirs with wildcards (either in fpmake or with RM), and sometimes then only when distclean is executed.

I can also imagine that when a change directory fails, and the RM command is executed anyway in the current dir before the change directory, that could cause a problem.

[JuhaManninen]

AFAIK Lazarus IDE does not delete temp files with wildcards or otherwise.

A safe solution is, of course, to manually inactivate DropBox completely while working in Lazaurs 4.6 but then I loose the ability to regret silly coding.

Use a revision control tool to store history of your coding. Git with its local commits is good for that.
DropBox clearly is not the right tool here.

[Martin_fr]

AFAIK Lazarus IDE does not delete temp files with wildcards or otherwise.

A safe solution is, of course, to manually inactivate DropBox completely while working in Lazaurs 4.6 but then I loose the ability to regret silly coding.

Use a revision control tool to store history of your coding. Git with its local commits is good for that.
DropBox clearly is not the right tool here.

Well depends what causes the delete... If the .git folder gets deleted too, then that is no good.

[rvk]

Well depends what causes the delete... If the .git folder gets deleted too, then that is no good.

Then at least your code is stil on the server 
If you remove the .git, it won't be able to erase the server.

Using dropbox sometimes has problems with locked files (for example dropbox is uploading the executable while Lazarus is just beginning compiling a new version, yikes).

I usually stay away from 'live' backup folders when working on projects.

[Martin_fr]

Well depends what causes the delete... If the .git folder gets deleted too, then that is no good.

Then at least your code is stil on the server 
If you remove the .git, it won't be able to erase the server.

What server? I have quite a few local git repo on my disk... (Ok some of them have a remote on a 2nd local disk, some...)

But back to topic.

We need clear steps how to make the issue happen. Without it happening on the system of anyone who can (and will) trace it in a debugger => more or less impossible to do anything about it.

[marcov]

AFAIK Lazarus IDE does not delete temp files with wildcards or otherwise.

A safe solution is, of course, to manually inactivate DropBox completely while working in Lazaurs 4.6 but then I loose the ability to regret silly coding.

Use a revision control tool to store history of your coding. Git with its local commits is good for that.
DropBox clearly is not the right tool here.

Local commits don't guard against disk failure.  Anyway, regardless of methodology, that does raise another point:  there might be multiple intrusive extensions other than dropbox active.

Thinking the rest of the afternoon about this, I think even the makefile risk is minimal, since it usually descends using make -C before emitting the delete(rm) commands. If the make -C fails, the makefile with the RM won't be executed.

[rvk]

The first question is: does Lazarus do any cleanup after FPC, or is this actually an FPC problem?

Although... TS didn't have the problem with 3.8 and FPC hasn't changed in ages

[JuhaManninen]

Local commits don't guard against disk failure.

OK, yes. Using a repository in the net solves that.
Disk failures happen rather seldom. Copying a backup of the whole project dir may be enough in most cases.

The first question is: does Lazarus do any cleanup after FPC, ...?

AFAIK no.
Lazarus does not have essential changes in the build process since 3.8. I wonder how that could make a difference.

[Martin_fr]

Lazarus has "build" and "clean" commands. But they clean before the compile... Not aware of after.

Some of them, do work with reading the dir, and removing file by file. But they do filter . and .. (and iirc symlink on Linux). I don't know how windows junctions and stuff would do for those (but if you put a junction the the rood dir into your lib folder .... => well...  assuming that stuff runs in lib folders only / I don't know.

The locking of a file can be simulated, just have a small app that opens it, and keeps it open read/write exclusive. That should do.
But that doesn't help if (like for me) there isn't any delete called at all. (or not that I can detect it).

So, whatever settings, add on packages, other stuff, the original poster has, he needs to give a detailed description
- setup
- config (upload all xml from primary conf)
- complete project, with lpi and lps, ready to trigger the issue.
- steps to perform with that project in the IDE