Forum > General

Sharing same source file in two projects

<< < (9/10) > >>

marcov:

--- Quote from: y.ivanov on October 06, 2021, 07:56:43 pm ---The problem doesn't seems to me as complicated as it is presented.  O:-)

--- End quote ---

It depends. If you have complicated solutions (building file list, reading PPU state, tracking everything), you want the result to be pretty airtight. If you want stuff built into the compiler, you want it to be really airtight.

I'm getting mixed signals about changing the compiler algorithm vs the IDE "decide compile or build" algorithm. The former is complicated, as the compiler doesn't exclusively cater as much to the "compile the whole project in one go with one commandline" as the lazarus ide does. And even Lazarus doesn't 100%.

So what are you talking about, extending codetools to make better decisions about compile or build, or changing the compiler?


--- Quote ---After a build for each unit source file there is a corresponding ppu file. In that ppu there are entries for all source files and units it depends.

--- End quote ---

It _directly_ depends, and only if the last build was successful. If it was unsuccessful, a PPU might have been deleted, breaking the chain


--- Quote ---To decide whether to re-compile or not, we should check whether there is any of the listed files/unit on which it depends on has been changed.

--- End quote ---

There is no list. Maybe codetools have something but not sure if that is complete.


--- Quote ---Usually we do that by comparing the current timestamps of the actual files against the saved timestamps into the ppu. If there is a change, then we must recompile the current unit.

--- End quote ---

That's what the compiler already does automatically. But that doesn't take defines into consideration.


--- Quote ---Let's say we have an additional entry into the ppu, containing the command-line defines at the time of compilation.

--- End quote ---

Assuming it is built in one go. Lazarus can have projects depend on packages, and might auto rebuild them too creating a different cmdline for those units (from the project .lpk of that package).


--- Quote ---Now we can compare if it matches with the current set of defines. If isn't - we must re-compile again.


* From the file included with the {$I file} directive - since the file will be listed into the ppu as one on which the current depends on, the file timestamp will be newer
--- End quote ---

No, it might be older. The other (b.lpr in my example iirc) project has a different but older include file.  The .ppu does not list paths, only names.


--- Quote ---
* From the source file itself, {$DEFINE xxx} - it is also listed at the topmost position into the ppu, the file timestamp will be newer
--- End quote ---

Locally defines are afaik not in the ppu. Only the ones on source file compilation entry.


--- Quote ---Thus, I can't see the reason of multi-level research through the dep-tree and corresponding ppu's.

--- End quote ---

You don't have a complete list of files in the IDE. In the compiler you can do more, but that also poses problems as where to stop ? The compiler doesn't know any package bounderies including lazarus packages and precompiled FPC units. It only knows ppus.  The only when it recursively stops the ppu loading is when it encounters .ppu's compiled with -Ur.  (FPC release .ppu, not sure if e.g. fpcdeluxe enables this though)

As already mentioned, to tackle this, you could make a -Ur multi level (system, package, application) and give some parameter to relax the rebuild algorithm at some package boundary.

But that is all quite massive, and I think there will be many practical problems with define checking. But the only way to find out is to try.


--- Quote ---Claiming up on the dependency tree (I'm assuming the leafs should be examined first) we can detect the change in the command-line defines for the shared unit and recompile it, even its source has not changed.

--- End quote ---

What dependency tree?  As said, there is only executing the compiler on the first module, nothing more.

You can execute the compiler or resort to ppu/source scanning in the IDE, nothing else. There is no makefile with all items listed that can be transformed into a tree.


--- Quote ---There will be different defines for a.lpr, thus unit3 will be recompiled when the change in defines detected.

--- End quote ---

That that is not true was the point of the exercise. The defines are in an includefile and thus not part of the commandline (and so not in the .ppu), so invisible to your proposed plan.  So no, that doesn't work. You might maybe able to detect something fishy though because the includefile datetimestamp changed. (either while compiling or when tracing .ppu's in the IDE)


--- Quote ---The inline defines will be changed in some included source file, their list is kept into the ppu and their timestamps will be compared.

--- End quote ---

Inline defines are not listed in the ppu, since they are not global to the ppu. Moreover, even if it did, that would break your algorithm, since then local unit defines would be in the .ppu and always mismatch your commandline.


--- Quote ---
--- Quote from: marcov on October 06, 2021, 04:54:37 pm ---This is a bit more convoluted case (but not unthinkable, think of all the projects that use an includefile for project settings)

E.g. take  a.lpr project that includes directory  a/b/c/incdir1 (-Fi) for include files and b.lpr project that includes directory a/b/c/incdir2.

--- End quote ---

That included file will be listed into unit3.ppu and it's timestamp will be checked.


--- End quote ---
It will, but it won't be necessarily newer.


--- Quote ---Put in other words, can't the command-line definitions (which I presume are same as the Laz project defines) be treated as saved in a temporary file and each source to have a dependency on it?

--- End quote ---

No. As said there is no global list to create some dependency on it.


--- Quote ---Maybe a simple hash on the command line stored into the ppu could do that job just for detecting the change.

--- End quote ---

The problem in leaving it to the compiler is when to stop. Even for the IDE the problem is what to do if it can't find a piece of source or ppu. Is that a missing file in the project, or does it belong to preinstalled packages? Quite involved to figure all that out.

Anyway, we are approaching the borders of my superficial knowledge of the compiler. The first thing to do is to decide approach. Change the compiler or change the IDE ?

marcov:

--- Quote from: prof7bit on October 06, 2021, 08:17:02 pm ---The quickest way would be to just store a hash over the compiler command line in the ppu and recompile if it changes. This would still cause some unneeded recompilation if a unit does not actually use a certain define but it would at least prevent units from NOT being recompiled if they really should.

--- End quote ---

See the compiler halting problem described in the post to y.ivanov. The compiler only knows .ppus. It doesn't know which belong to project, lazarus package or FPC package.

Sooner or later it will load a FPC or lazarus unit that wasn't compiled with the exact parameters.


--- Quote ---The more complicated approach would be to store all compiler options individually in the ppu and mark those that did not affect the compilation of that particular unit as "DontCare". This would be a whole bunch of code, some of it in the depths of the compiler.

--- End quote ---

This precludes any source precompiled from outside the project. As said multiple times, many pieces of code (e.g. lazarus packages and the fpc packages) need extra defines and options, so that "global one compile" model is not viable.


--- Quote ---But it would also change the PPU format and probably also force other tools that parse it to make some changes or additions.

--- End quote ---

I think it would be possible, but you would have to flag all preexisting non (project .ppus in a special way, so the compiler knows the hash match doesn't need to be performed. IOW, define a project as a leaf and compile the rest as branch and tell the compiler only apply it to units that are a leaf.  Maybe even compile some as trunk (the current -Ur)

But as all of this is quite invasive and reaches across the whole codebase, the question is how practical all this would be .

Moreover it relies on having a reliable classification as leaf/branch/trunk and it might only shift the problem.

y.ivanov:
@markov,
What I can see now is that maybe I've failed to explain my proposal clearly. It is basically what prof7bit wrote into his the reply #39, i.e. to put into the ppu some footprint of the actual command-line at the time of compilation. Either as a hash or as a string, etc. - whatever is more convinient - and then compare against it.

Anyway, despite my bad explanation, now I can see the actual problems you're writing about:

--- Quote from: marcov on October 07, 2021, 12:02:51 pm ---The .ppu does not list paths, only names.

--- End quote ---
Is the first trouble. And then:

--- Quote from: marcov on October 07, 2021, 12:02:51 pm ---In the compiler you can do more, but that also poses problems as where to stop ? The compiler doesn't know any package bounderies including lazarus packages and precompiled FPC units. It only knows ppus.  The only when it recursively stops the ppu loading is when it encounters .ppu's compiled with -Ur.  (FPC release .ppu, not sure if e.g. fpcdeluxe enables this though)

--- End quote ---
Is probably the actual stopper.

And just to clarify what I've meant by dep-tree (It is not relevant anymore)

--- Quote from: marcov on October 07, 2021, 12:02:51 pm ---
--- Quote ---Claiming up on the dependency tree (I'm assuming the leafs should be examined first) we can detect the change in the command-line defines for the shared unit and recompile it, even its source has not changed.

--- End quote ---
What dependency tree?  As said, there is only executing the compiler on the first module, nothing more.

--- End quote ---
When compiler executed on the top module, it starts checking dependencies, walking through the used units. ppu files. For each such .ppu file in turn, it's own dependencies has to be checked (recursively) until a leaf .ppu found. Thus, the compiler walks a dep-tree which is defined as visited .ppu files as nodes. 
That is the dependency tree I'm talking about, it is not contained in a single file, it is a multi .ppu file structure. But it's there :)

marcov:

--- Quote from: y.ivanov on October 07, 2021, 01:18:46 pm ---Is probably the actual stopper.

--- End quote ---

Not absolute, but things like this make it very complicated, not just compiler algorithm, but all buildsystems must be made aware (and that assumes my hunch even works out in practice) And then also the cost/benefit factor comes into play.

Dep-tree: that's what I meant. The compiler traverses recursively, and that can be seen as a tree pattern, but that is not the same as first building a tree and then traversing it. Also compiler unit building order is not entirely predictable, at least not intuitively.

Anyway, the discussion is still good, also for me as core member but compiler-outsider. Keeps me on my toes :-)

y.ivanov:
@marcov,
What if the suggested check is performed only on the .ppu's residing into intermediate directory? The compiler should know that directory by its command-line parameters, I believe.

Navigation

[0] Message Index

[#] Next page

[*] Previous page

Go to full version