You can either set it from the file's extension or look for some characteristic of the file. For example, if a file has a extension pattern of ".?htm*" or starts (inside a certain threshold, to deal with files starting with a comment) with "<!doctype html" or "<html>" then it's an html file, and so on. That's basically what the file *nix utility and most (hilighting-)editors do.
IIRC, there was (is?) a function somewhere in SynEdit's source that did something but I don't remember exactly where or how it's called, sorry.