Hey eli,
Ok, let's do some maths: 2*4096*4096 = 33'554'432 Cells
Let's assume were doing 64b integers: 33554432*64 = 2'147'483'648 bits = 268'435'456 bytes = 256 MB
That's nowhere near 3GB since its only a quarter of a Gig. On that amount of RAM you can have 12 of those matrices, well less since you do need the OS to be running.
And with the second matrix, like Davo mentioned, even with 512 bit integers, that would be only 4MB.
So, it's definitely nothing to do with memory but everything to do with the way you're manipulating that memory.
I'm unaware of how the GMP unit uses those MpIntegers, so I'm not really comfortable blaming the GMP unit just yet.
But, if you're running Windows on a 3GB system, well, how much RAM do you really have free, I wonder. I'm guessing that you have at least some 300MB of spare RAM to run the code and allocate the data space.
If you're able to have Lazarus open and running the code, I'm guessing your system can spare the 300MB...
So, again, back to how your code is manipulating that data.
If you would provide us with a test project, or a subset of your code so we can have a better look at how it's running, maybe we could me of more help.
Hope that helps!!
Cheers,
Gus