Expanding of the tilde (~) is noramlly done by the shell (bash, sh etc.), before the filename is passed to the proper API.
If you need expanding, then you can use ExpandFileNameUtf8() function.
This function will take care of the tilde on Unix like systems.
So, if the file in your example exists, then this should work:
if (Key = VK_F1) or ((ssCtrl in Shift) and (Key = VK_H)) then
if FileExistsUTF8(ExpandFileNameUtf8('~/.mydir/help/help.html')) then
begin
ShowMessage('Help file found.');
OpenDocument(ExpandFileNameUtf8('~/.mydir/help/help.html'))
end
else ShowMessage('Help file not found.');
Bart