You are only allowed to modify property of the frame components, but you cannot delete them - that's probably your problem.
Since you cannot distinguish components on the frame from components added separately the work with frames can very easily become very frustrating.
Whenever I use frames I only insert them at runtime so that these issues cannot occur. The must prominent application using Frames, the Lazarus IDE, does the same, and it is very successful this way.