New ID attributes on SynEditHighLighter unit.

[Martin_fr]

Probably I'm not understanding the issue. Can it be shown on a practical case? Would you please put some practical case when this issue occurs?. 

Depends on how it is implemented.

But every implementation that forces a sub-categorization, will add limits. And some text/language may not be able to be highlighted in that limit.

Again, there will be limits anyway. but why add.

That was my example:
ASSUME your implementation forces operator, to be a subcategory of symbol (an example you gave), then "and" can not be an operator, because it is not symbol(s).

By now I have it hardcoded.

That is exactly the problem.

It is one thing to say keywords, identifiers etc must entirely consist of a given set of chars (word token chars). Ideally configurable, but may even be hardcoded.

But the last sentence does not put a relation between identifiers and keywords.

Each of them may be a different subset of word-tokens. word tokens themself have no attribute.

defining the concept of word-token chars, allows the scanner to determine word boundaries, and then match the found words against the configured group.

This allows for
* word token char A-Za-z (0-9 maybe)
* identifiers start a-m
* keywords start n-z

Which requires keywords to NOT be a subcategory of identifiers.

[Martin_fr]

If it were not hardcoded, but configurable, then it was not so much an issue.

Because I as user could define any outer group I need, even if the outer group does not describe any highlightable set

[Edson]

That was my example:
ASSUME your implementation forces operator, to be a subcategory of symbol (an example you gave), then "and" can not be an operator, because it is not symbol(s).

My implementation doesn't forces. I've tried to explain this, in many ways. Let's see this hipothetycal definition:

Code: [Select]

<Token Start= "+-:;" Content = "+-:;" Attribute='SYMBOL'> </Token>
<Subset Set='SYMBOL' Attribute='OPERATOR'> +  - </Subset>

<Token Start= "A..Za..z_" End = "A..Za..z0..9_" Attribute='IDENTIFIER'> </Token>
<Subset Set='IDENTIFIER' Attribute='OPERATOR'> and or </Subset>
Can I say "I have forced OPERATOR, to be a subcategory of SYMBOL"?.
Can I say "I have forced OPERATOR, to be a subcategory of SYMBOL and IDENTIFIER"?.

No. I just have defined the symbols "+", "-" and the identifiers "and", "or" to have the attribute OPERATOR.

"Is this definition of OPERATOR restricted to be an SYMBOL or IDENTIFIER ?"

No. Because I could have included other token definition for to have the attribute  OPERATOR with nothing to do with identifiers or symbols.

Formally: There is not a definition for an OPERATOR (like we understand syntactically) on a HL.

When I define:

<Subset Set='IDENTIFIER' Attribute='OPERATOR'> and or </Subset>

I don't say "Define all the operators for to be a subclass of IDENTIFIER".
I am saying: "By the way: Use the parser of IDENTIFIER like a filter for easy identify this tokens 'and,or' with the attribute OPERATOR"

I repeat:

In the HL we just define attributes for tokens. Lexically we can define operator in diferentes ways: Like a "delimited token", like a "token by content", "Like a subcategory of a token". And we can use one or more definitions at the same time. Syntactically, it doesn't care.

Even in my simple implementation, I don't force a token KEYWORD, to be a subset of IDENTIFIER. I can define tokens KEYWORD with any other rule.

[Martin_fr]

Ah, I see now (at least one part)

I have always read your statements that the attribute itself is also part of the subcategory. (and therefore *all* it's tokens)

But the Attribute (itself) is NOT a sub category (neither a subset) of any other attribute.

Some (not necessarily all) tokens belonging to an attribute are (or may be) a subset of tokens from another attribute.

This is different. And OK.

This was a long discussion for getting us synced. But interesting.

-------------------------------

Leaves one last point: Why/How does that need GetDefaultAttribute?

Does/Will/Do you plan: The user can define it's own attributes ?

<Subset Set='IDENTIFIER' Attribute='MYCOLOR1'> and or </Subset>

[Edson]

Well, it's sure I haven't expressed it correctly in English, and this post have grown unnecessarily.

Leaves one last point: Why/How does that need GetDefaultAttribute?

Actually, it can live without that. But when it's needed to access to common attributes of a HL through the editor, for config, it's desirable to have an easy way for that. And NUMBER are a common attribute.

Does/Will/Do you plan: The user can define it's own attributes ?

I haved considered. It would be relative easy. But I'm afraid of that imply to change the struct of the Base Class of the HL.

By now I have predefined these Attributes:
tkNull, tkIdentif, tkKeyword, tkDirective, tkVariable, tkNumber, tkSpace, tkString, tkComment, tkSymbol, tkLabel, tkAsm, tkExtra1, tkExtra2

Firts I am worried on defining the properties of the attributes:

<Attribute Name='MYSTRING' type='STRING' ForeColor="$$$$" Backcolor="$$$$">
</Attribute>

Also I want to include the process of symbols, and operators.

But what is important for me, is struct someway for to manage folding on a scritptable HL (for me, it belong to the syntactic level) and the nested HL. I hope you can help me on this issue.

Later, I would like to improve the code-completion, making it part of the "syntax file". In that way, all the highlighting, folding and completion could be joined on a uniq "syntax file".

[Martin_fr]

Well as you pointed out earlier, not all I wrote was always expressed clearly. That is the fun of human languages....


GetDefaultAttrib for conf makes some sense.
Personally I prefer other ways, but that is personal preference.


Adding attributes from user conf should be no problem.

The highlighter can keep them in a list/array and use them. They do not need to have a property.
If you want them accessible it Object inspector, you can add a TCollection.

Configuration such as the IDE options can use the Attrib[n]/AttribCount properties. That is what the IDE does.

In fact IIRC the IDE stores extra attibutes in this list. (such as block selection color, which is not handled by the HL)

-------------------
Could you use "range based" "state driven" or similar instead of nested? nested to me always is using an asm HL inside a pas HL (2 different HL, one usde within the other)

Folding, start here, and the example folder in the IDE installation.
http://wiki.lazarus.freepascal.org/SynEdit_Highlighter

It is a complete tutorial. Feel free to ask question.

[Edson]

WinMerge can generate patches. But you need the original, and the modified.

Since you should base a patch on the latest SVN, I recommend to use TortoiseSVN, which also has the option to create a patch.

@Martin
I have made a Ckeckout on my PC using TortoiseSVN from "http://svn.freepascal.org/svn/lazarus/trunk"

I have located the Highlighters files on my local copy on "D:\Lazarus\components\synedit", so  it's where I have to make the changes and generate the patch.

Is't OK?

[Martin_fr]

Yes make the changes there.

Then check the context menu of the windows (file) explorer. In the tortoise section is "create patch"

[Edson]

Ok.

I have found these new HL files:

* synhighlighterjscript.pas
* synhighlighterposition.pas

that I don't find on the Component Palett (on 1.0.14).

Do they have to be changed too?

[Martin_fr]

if it is easy, then yes. But no big deal.

If one is missed, not a problem. It may get reported one day on mantis, and *one* is quick ho fix.

Only, if a lot are missing, and it gets reported, then it would be a lot of work. That is why I did ask to implement it for all/majority of HL.

[Edson]

I've prepared this patch.

I've included these new constants:

+  SYN_ATTR_NUMBER            =   6;
+  SYN_ATTR_DIRECTIVE         =   7;
+  SYN_ATTR_ASM               =   8;   
+  SYN_ATTR_VARIABLE          =   9;

and updated 14 highlighters, with these new values.

This is the first I do using TortoiseSVN, so I hope it will be OK. I have to learn more about SVN.

First of all, I would like to know how to to test this patch?. I suppose this doesn't work on Lazarus 1.0.14.

[Martin_fr]

just acknowledging, have seen it, and will look at asap.

[Martin_fr]

Just tried, it does not even compile.

    SYN_ATTR_SYMBOL: Result := tkSymbol;

Should return the attribute (as in the object holding the colors) not the token-kind.

Same for all HL.

[Edson]

Sorry Martin, I have been doing a copy-paste from the tokend's ID instead of the attribute.

I've corrected the patch by hand, so I don't know if this is OK.

Would you please tell me, what should be the correct way for test this patch.?

[Martin_fr]

I still have to check the patch.


You should be able to actually run, and recompile Lazarus from the SVN directory. Then you can test (which is part of supplying a patch).

Copy a lazarus.exe into the folder.

When calling it, use a primary-config-path (search forum and wiki), so it use a diff config-dir.

It will ask for fpc. point it to the fpc from your normal install (do not copy, move, rename the fpc folder. that will not work (or you must edit fpc.cfg)