Just call free on the control, the control itself handles the rest.
Also, you may have other items in the control that needs to notify others of what is happening, losing a link to the owner or parent while this is happening is not such a good idea.
So enumerate backwards as you are and call free on it, you don't even need to define a local control, you can take it out of the array of controls directly.
For I := ControlCount-1 Downto 0 to Controls[I].Free;
Unless I missed something, that should work.