An alternative to modularize with a git construct would be to use submodules. I use them all the time and find them very easy to work with. When you have a repository that uses them you just add a --recursive flag to a git clone and the source plus all submodule sources will be fetched. They also allow for using versioned libraries in projects and can be updated as needed by simply issuing a git checkout xxx in modules directory.
Doesn't help however with things like compilation time and such when you do a clean build.
Very true, just an alternative suggestion with a different set of benefits