Try putting {$linklib newton32} after your function declarations. FPC doesn't require this with external routines on other platforms, but does for OS X. I've been told that this requirement will eventually be eliminated, but for now you need to do this.
I believe {$linklib} corresponds to the OS X linker (ld) -l switch, which tells the linker to first search for libnewton32.dylib and if it can't find it search for libnewton32.a, which is what you have.
You could also specify this on the FPC command line if you want to make it clearer about what you're doing link-wise: -k'-lnewton32' -- the -k switch is how you pass stuff to the linker via FPC.
Not sure what the archiver (ar) error message is telling you. The approach looks correct: extract the archive file's (.a) individual .o files to a temp. folder, then combine them (*.o) into a .dylib (specified with -o switch).
You don't mention what version of OS X you're using or whether you're on the Mac PowerPC or Intel platforms, but these could also be factors, particularly if the .a file is old.
Let us know if any of this helps you. I know it seems like it shouldn't be this complicated - afterall, we're working in Pascal, not C or Java - but sometimes it is due to the cross-platform nature of FPC.