The problem doesn't seems to me as complicated as it is presented.
It depends. If you have complicated solutions (building file list, reading PPU state, tracking everything), you want the result to be pretty airtight. If you want stuff built into the compiler, you want it to be really airtight.
I'm getting mixed signals about changing the compiler algorithm vs the IDE "decide compile or build" algorithm. The former is complicated, as the compiler doesn't exclusively cater as much to the "compile the whole project in one go with one commandline" as the lazarus ide does. And even Lazarus doesn't 100%.
So what are you talking about, extending codetools to make better decisions about compile or build, or changing the compiler?
After a build for each unit source file there is a corresponding ppu file. In that ppu there are entries for all source files and units it depends.
It _directly_ depends, and only if the last build was successful. If it was unsuccessful, a PPU might have been deleted, breaking the chain
To decide whether to re-compile or not, we should check whether there is any of the listed files/unit on which it depends on has been changed.
There is no list. Maybe codetools have something but not sure if that is complete.
Usually we do that by comparing the current timestamps of the actual files against the saved timestamps into the ppu. If there is a change, then we must recompile the current unit.
That's what the compiler already does automatically. But that doesn't take defines into consideration.
Let's say we have an additional entry into the ppu, containing the command-line defines at the time of compilation.
Assuming it is built in one go. Lazarus can have projects depend on packages, and might auto rebuild them too creating a different cmdline for those units (from the project .lpk of that package).
Now we can compare if it matches with the current set of defines. If isn't - we must re-compile again.
- From the file included with the {$I file} directive - since the file will be listed into the ppu as one on which the current depends on, the file timestamp will be newer
No, it might be older. The other (b.lpr in my example iirc) project has a different but older include file. The .ppu does not list paths, only names.
- From the source file itself, {$DEFINE xxx} - it is also listed at the topmost position into the ppu, the file timestamp will be newer
Locally defines are afaik not in the ppu. Only the ones on source file compilation entry.
Thus, I can't see the reason of multi-level research through the dep-tree and corresponding ppu's.
You don't have a complete list of files in the IDE. In the compiler you can do more, but that also poses problems as where to stop ? The compiler doesn't know any package bounderies including lazarus packages and precompiled FPC units. It only knows ppus. The only when it recursively stops the ppu loading is when it encounters .ppu's compiled with -Ur. (FPC release .ppu, not sure if e.g. fpcdeluxe enables this though)
As already mentioned, to tackle this, you could make a -Ur multi level (system, package, application) and give some parameter to relax the rebuild algorithm at some package boundary.
But that is all quite massive, and I think there will be many practical problems with define checking. But the only way to find out is to try.
Claiming up on the dependency tree (I'm assuming the leafs should be examined first) we can detect the change in the command-line defines for the shared unit and recompile it, even its source has not changed.
What dependency tree? As said, there is only executing the compiler on the first module, nothing more.
You can execute the compiler or resort to ppu/source scanning in the IDE, nothing else. There is no makefile with all items listed that can be transformed into a tree.
There will be different defines for a.lpr, thus unit3 will be recompiled when the change in defines detected.
That that is not true was the point of the exercise. The defines are in an includefile and thus not part of the commandline (and so not in the .ppu), so invisible to your proposed plan. So no, that doesn't work. You might maybe able to detect something fishy though because the includefile datetimestamp changed. (either while compiling or when tracing .ppu's in the IDE)
The inline defines will be changed in some included source file, their list is kept into the ppu and their timestamps will be compared.
Inline defines are not listed in the ppu, since they are not global to the ppu. Moreover, even if it did, that would break your algorithm, since then local unit defines would be in the .ppu and always mismatch your commandline.
This is a bit more convoluted case (but not unthinkable, think of all the projects that use an includefile for project settings)
E.g. take a.lpr project that includes directory a/b/c/incdir1 (-Fi) for include files and b.lpr project that includes directory a/b/c/incdir2.
That included file will be listed into unit3.ppu and it's timestamp will be checked.
It will, but it won't be necessarily newer.
Put in other words, can't the command-line definitions (which I presume are same as the Laz project defines) be treated as saved in a temporary file and each source to have a dependency on it?
No. As said there is no global list to create some dependency on it.
Maybe a simple hash on the command line stored into the ppu could do that job just for detecting the change.
The problem in leaving it to the compiler is when to stop. Even for the IDE the problem is what to do if it can't find a piece of source or ppu. Is that a missing file in the project, or does it belong to preinstalled packages? Quite involved to figure all that out.
Anyway, we are approaching the borders of my superficial knowledge of the compiler. The first thing to do is to decide approach. Change the compiler or change the IDE ?