I guess the problem with any non-native help system on Mac is that users generally expect help to look and behave and be accessible in the same way with Mac software. Have you gone through various programs on Mac to see whether any programs use their own help viewer? If not, that should be a signal on what the normal way of doing help is on Mac.
The Mac native system also supports features that might be a lot of work to support in a custom viewer, such as video. If you look at the Xcode 4 help, I believe there are videos embedded in it. Also, the native system obviously supports localization.
I have a bunch of apps that use custom help viewers (or just opening some text file or pdf file), although most of them are cross-platform apps.
I'm not very concerned about following exactly the Mac style since from what i've seen this is a lost cause with Lazarus, unless some features are added, like different "versions" of a form per widgetset (the expected style and layout is different under Windows, GNOME and Mac OS X for example).
LazHelp doesn't provide much features, but this isn't the goal. Instead i want something simple that a) works and b) works across platforms, in that order. Currently the best way with Lazarus (and many other cross platform applications) is to open a help site in the browser. This is good (and
i've written a tool to assist in that) but it has it's own problems and limitations.
(as a sidenote, while generally i like Mac OS X, i'm not sure i like its help system interface)
Maybe another approach you could take would be simply to write a little converter that takes your editor's HTML and adds the tags and what-not that make it Mac-friendly. Then just use the Mac help viewer.
Writing a HTML exporter for the format that LazHelp currently uses should be trivial since the document structure is very HTML-like and is something i'm planning to do. However that would be far from cross-platform since people will need to have special code for Mac. Although if someone uses Lazarus to make Mac apps which behave as Mac as possible, he'll already have such issues (in a Mac app of mine i had a "Macize" procedure which did stuff like changing properties of some controls -removing borders, etc- and converting Ctrl+<blah> shortcuts to Command+<blah> :-P).
But as i said, having "proper" Mac OS X support is rather low in my priorities. Having it work without crashing is much more important :-). And since the Carbon widgetset is far from stable, i'm not sure if the crashing issue is in my code or the widgetset code. I'll need to checkout from the svn and see if i can replicate the crash with a simpler program.