In the second case, presumably the compiler converts the longint N to a byte for you (so it is compatible with the "in" set operator), since Pascal set elements must be byte-sized or less. So you have a behind-your-back truncation of N's value. I'm not sure if this is intelligence built in to the compiler, or just luck that using the truncated N gives a correct result (since if N is larger than a byte value it cannot by definition be part of a Pascal set, so will never be "in" it in that case).
In the first case, the compiler has to allow N mod 5 to be a longint, since for most values of N the expression would be a longint. So it issues the valid warning.