Best I can think off is to have defines ( -dSomeDefine _ set per buildmode, and then $IFDEF the units in the uses clause.
The package may be included and build, but if no units are used, then it wont be in the resulting exe. (you just loose some time, if some of the unused units are rebuild).
It be more complex, if want to choose between 2 packages that provide the same units. So your project always "uses UnitFoo", but that could be of one or the other package. If you add neither package, you can set the include path for the ppu and o files per buildmode. But you then have to always manually rebuild the units.
Or you need to find a way to hide the path from the IDE, so you can add both packages, but none of their ppu would be found, until you manually add that path to the buildmode.
Whatever you do, I would advice against including the path to the source /pas files per buildmode => then you get ppu files in the project, rather than the package. And if you change stuff, it will generate chaos. (People have tried that often enough, and regularly lost the battle)