SetLength has GetMem in it. Maybe more than one. So yes it will be slower. 
How many times are you going to call SetLength?
If the memory access is fast enough, then calling SetLength once doesn't really matter.
If GetMem gets called thousands of times (each with different dimensions), then yes, you could go with the method taazz or Remy showed you. (the upside of taazzs one being it has the [][] notation)
(But do your benchmarking beforehand to check if it's really necessary.)