Not sure what you mean by "closed polygon towards the edge of the 3D grid". That the contour line suddenly leaves its correct path and jumps to the edge? This is certainly not correct, but without having seen your code and your data I can't tell very much. How are the data points distributed? Add code which shows you where the data points are located. Maybe there are no data points near the edge?
If you don't find a solution yourself, try to extract the contour creation and drawing code from your program into a little demo which shows the issues and which you can upload here, if possible along with data file (or information how I could get it if it is too big). Pack pas, lfm, lpr and lpi files of the demo into a single zip and upload it (no exe and no ppu please).