You do not need to go for a package at all costs.
I keep my units in one (central) directory, and in
Project Options > Compiler Options > Paths I add the path to this directory and to each of its sub-dirs.
This works for you, locally, but not the best way if you want to publish your code. In such case, you need sub-dirs in your project dir, like you already have.
As for the package, you need one OS-independent unit, and that is the one you'll refer to in the package file.
This unit will have uses section with IFDEF for the OS-es:
uses blah...blah units,
{$IFDEF darwin}
darwin_midi_unit,
{$ENDIF}
{$IFDEF windows}
windows_midi_unit,
{$ENDIF}
.
.
.
After that, this unit should provide the API (procedures, functions etc.) to the app, and the sub-units (darwin_midi_unit, windows_midi_unit etc) should provide the underlying functions used by this main unit. In the best case, all the underlying units would have the same-named procedures, with the same argument types, so that you don't need to IFDEF in the main unit.
Example: main midi unit has procedure
procedure SetMidiBuffer(AVal: string);
begin
SetMidiBufferInSubUnit(AVal);
end;
and your both darwin_midi_unit and windows_midi_unit have the procedure SetMidiBufferinSubUnit(AVal: string), but each unit with its own code that is OS-specific.
The IFDEFs are respected at installing the package in Lazarus, so the compiler will not complain about windows_midi_unit if you install the package on MacOS.