Well, Windows handles are unsigned. So logically you'd keep everything unsigned. That you don't introduces the problems.
Note that the optimization doesn't matter. In any Delphi like that has int64, a integer<>cardinal comparison or addition will be upscaled, and this comparison will always fail because int64(-1) <> int64(cardinal(4294967295)).
In FPC and Delphi after 4, this generally got cleaned up, but if you are recycling ancient code this might trip you.