Warning: rambling thoughts ahead:
Now if you could just fence off that unsafeness such that user code could benefit from the "magic" in the GC, RC or other low-level management units without themselves doing something unsafe, you'd be getting somewhere.
Actually when I had discussions with Benjamin K about M10 this came up. Somehow confining low level tricks to certain modules, and back then I was sold on it. The idea of the unsafe/safe module separation is to confine the unsafe gore to the unsafe modules, maintained in a different way from the safe more business like code.
However later when I met an old college friend that I used to discuss language design with, and he questioned if safe code that calls unsafe code with high granularity all the time is really that safe.
Can you actually enforce enough of a separation, and stronger, does it really matter? In cases where safety is really important (like e.g. if you run as a plugin in a shared services webserver), Java and C# often don't allow calling unsafe code either. The VM itself is of course unsafe, but that is vendor supported.
So one could argue a safe/unsafe separation within one codebase is a bit pointless from the practical use viewpoint.
As it was quite irrelevant for me(my work programs can be described as machine firmwares rather than applications), I never did much with that thought.