I don't get why Unicode is a problem.
Obviously, machine code for i386 platform (and EXE file format is the same).
The rough outline of the EXE format is the same. there are possible differences in details. Pascaldragon might know more about that than me.
Runtime library has many functions, which know nothing about Unicode and WinAPI (sin, cos, log, abs, ...)
True. But those (sin,cos,log,abs) are mostly only CPU dependent, not OS dependent, so I didn't count them as "special" anyway.
But anything that communicates a string to an OS call is affected by unicode. Open file, findfirst, but also calls that get parameters, environment strings and special directories.
Some functions are WinAPI wrappers (sockets) or use WinAPI in implementation low-level (WriteLn, ReadLn, ...)
Sockets are a good example. Not all win9x targets support winsock2, which is the corner stone of most Windows networking. Only very late (win ME) and with all updates afaik do.
That looks like high-portable part of RTL (just copy implementation).
About Unicode. I guess strings should support Unicode (cause UTF-8 have no same character length) and functions like Assign.
In Windows versions before 2019-05 Windows 10, unicode means UTF16 and thus -W API calls that win9x doesn't have. The usual forum parrots point to the kernelex and MSLU enhancements to make such calls work on win9x, but I don't know of no practical experiences of such avenues. It is just a gentle remind for a future win9x maintainer to research them, as they could save quite some time.
Because of Windows 98 use "local" single byte encoding (ASCII, Windows-****),
Don't forget the console uses OEM encoding.
I see two ways: set single byte only encoding for Windows 9x or use UTF-8 implementation, but before system call (kernel32.dll call) convert UTF-8 string into "local" single byte string.
What if try to modernize RTL (Windows 9x) epoch FPC 2.x to modern RTL API?
You are free to try any avenue you want. As long as you keep it separate from regular (NT5+ aka w2k+) windows. That is a hard request. No sharing, period. Copy existing (NT) code if it helps you forward, but then maintain it in isolation from the mainstream Windows port.