Should Windows and Linux work in the same way when using the LCL?Of course not, since Linux is case sensitive with respect to filenames and Windows is (almost always) case insensitive.
*.txt|*[Tt][Xx][Tt]
Digging the widgetset code, I found out that dialogs in Gtk2 use something called "glob pattern (https://en.wikipedia.org/wiki/Glob_(programming))" and that, according to this (http://teaching.idallen.com/cst8207/15w/notes/190_glob_patterns.html#using-to-match-case-insensitive), you will get case-insensitive behaviour with ...
So, for now, if you want the dialog to behave case-insensitevely on all platforms, as a workaround you can use something like this:
OpenDialog1.Filter := {$ifdef LCLGtk2} '*.txt|*.[Tt][Xx][Tt]'; // Gtk2 {$else} '*.txt|*.txt'; // win32/64 and Qt4 {$endif}