You spoke about introducing scopes (as in one type of token being a subset for another):
Yes. I spoke about:
a) The scope of a nested HL (range or block).
b) The ability for defining subsets of a token with a different Attribute (like Keywords).
I'm not sure what do you refer exactly?
1) This is fol your HL only, and not generic?
There is no code/definition or other reference to has in any other HL?
None of the base classes for HL need to provide anything special for this?
For creating my HL, I have been learning of some Lexer's and editors (Notepad++ and UltraEdit). One lexer can support nesting, but it's too much power (and process) for a highlighter. I had to simplify the flexibility for gain speed.
As I have seen, Nested HL are a need if we can make a reasionably flexible scriptable HL. By now I haven't implemented the nesting. (I'm taking a rest of HL by now), but I expect to develop it, on a future.
I had not structured yet, how it will be defined a case of nested HL. I have some ideas buy nothing clear by now. Not even if this will need to modify the base class.
But what I can say by now, is that (and this is material for another Topic) it's related to the Folding and Code-Completion features.
If we work with nested HL, we should consider, that even the whole HL have attibutes, like the background color, or the default font. We could see all the background of a procedure on a different color, if we use nested HL for blocks.
2) This is.. or This is not ... related to the IDs for GetDefault attribute?
In some way. Because when working with scriptable HL, it's necessary to make generalization about the attributes. If the HL works at the lexical level, it can name their attributes on any way (probably need just a few constant SYN_ATTR_COMMENT, SYN_ATTR_IDENTIFIER, SYN_ATTR_KEYWORD , etc.).
If we work at the syntantic level, we can managed a lot of attribute (SYNS_AttrASP, SYNS_AttrAssembler, SYNS_AttrAttributeName, SYNS_AttrAttributeValue, SYNS_AttrBlock, ...)
How is that different from choosing some identifiers as a keyword? Or from defining some symbols as operators (which you mentioned, by declaring them as subset).
For visual results, no differences.
The operator "and" could be defined as a Subset of Identifiers with the attribute OPERATOR. For visual efects, it will be similar to defining the operator "&&" like a subset of Symbol.
There is not inconsistence.
But if we don't define "and" like a OPERATOR token, it will be sure considered as an identifier (and coloured as such). And it's lexically correct. Probably, syntactically we know that "and" is an operator, but the HL have not way for know it.
Again, there is not inconsistence.
Remember when I said that managing subsets of tokens is some kind of sintactical approach of a HL.
The point is, a language could define +++-* as identifier (allowing it as name for a variable or function. (In brainfuck there are only symbols).
OK we can define some like:
<Token Start= "+" Content = "+-" Attribute='IDENTIFIER'> </Token>
Or
<Token Start= "+" End = "-" Attribute='IDENTIFIER'> </Token>
The rule is than at "Lexical Level", an attribute is just THE WAY A TOKEN IS SHOWED in the screen. This mean that a VARIABLE attribute, no necessary have to correspond to the definition of a VARIABLE in the syntax of a language.
In fact I could define my identifiers like:
<Token Start= "A..Za..z_" Content = "A..Za..z0..9_" Attribute='DIRECTIVE'> </Token>
And if the attribute DIRECTIVE have the same properties that I expect to see on the editor for an IDENTIFIER, the user won't have idea (and really don't care) the name of the attribute, except for Config.
About dinamyc attributes:
We can define dynamically, many attributes on a syntax. This definition create a new attribute, with the label NUMBER:
<Token Start="0..9" Content = '0..9' Attribute="NUMBER"> </Token>
We can say that the NUMBER attribute always exists, but in a scriptable HL, you have to create it.
Currently I can not create dinamicaly attributes on my highlighter (But I have analysed that possibility), but I have defined two "empty attributes" for using on defining new token struct:
<Token Start="{%" End="}" Attribute='EXTRA1'></Token>
Craete dinamicaly an new attribute on a scriptable HL should consider the name of the attribute ("MYSTRING") and the label (STRING).