Recent

Author Topic: New ID attributes on SynEditHighLighter unit.  (Read 44861 times)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #15 on: November 22, 2013, 10:21:00 pm »
Quote
And we can also add a constant for CompilerDirectives in that case.

What it would be for?

That was what you originally wanted (only you named it macro)
pascal:
{$IFdef}

c (actually preprocessor...)
#include

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #16 on: November 24, 2013, 07:05:07 am »
Quote
That was what you originally wanted (only you named it macro)

Ok. I didn't realize.

I was checking the range of "TtkTokenKind" on the 20 Highlighters in the Lazarus 1.0.12. And it's clear that we need to include:
* SYN_ATTR_NUMBER

These others are common too:
* SYN_ATTR_TEXT
* SYN_ATTR_VARIABLE
* SYN_ATTR_DIRECTIVE (used by Pascal and C++)
* SYN_ATTR_ASM  (used by Pascal and C++)

There is a "tkPreprocessor" type, in SynHighlighterAny, that I suppose is some kind of "tkDirective".

Another values could be:
* SYN_ATTR_CONSTANT  (used by SynHighlighterAny)
* SYN_ATTR_FUNCTION  (used by SynHighlighterSQL)
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #17 on: November 24, 2013, 01:45:59 pm »
Quote
These others are common too:
* SYN_ATTR_TEXT
* SYN_ATTR_VARIABLE
* SYN_ATTR_DIRECTIVE (used by Pascal and C++)
* SYN_ATTR_ASM  (used by Pascal and C++)
* SYN_ATTR_CONSTANT  (used by SynHighlighterAny)
* SYN_ATTR_FUNCTION  (used by SynHighlighterSQL)
Before I comment, I need to know:

What would be examples for TKText? Does it differ from string?

Which highlighters do variables at the moment?

---

If constant is only used by SynAny, then I don't see it that important. And any number and string are a constant. So what to return as default for constant, if there are more than one?

ASM, would be ok. But some day will break. The idea (though no plan yet when) is to highlight asm with an embedded hl. (so asm will have many attributes. then which one to return as default?

Also with asm highlighted like that, there may be number(none asm), and number(asm) which can be different. Which one to return as number?

---
Even today the concept is flawed.

String/char in pascal has a default attribute (default: blue). But the pascal highlighter can mix attributes. if the char is a case label
Code: [Select]
case foo of
  'A': ;
  'B': ;
end;
then (if configured) it will have a different attribute (result of mixing or replacing).

And not only that, but the mixed attribute (the same object, with either the same or different values) is also returned for numbers and identifiers, that are case labels.

So you can not use the returned attribute, to find all numbers, because some numbers are not using the default.

----

I will still add the extra, since this concept is already there. I will place a comment there with warnings, that its usage may become less and less reliable....



In the end, an extension of the Attr[n]/ AttribCount idea might be able to cope with this better. Each attibute can belong to more than one class, The attribute can then be asked
Attrib[n].IsInTokenClass(SYN_ATTR_NUMBER)


But if you do the patch for GetDefaultAttib, then I accept that. It is only about which classes/types to add.

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #18 on: November 24, 2013, 06:14:48 pm »
Code: [Select]
What would be examples for TKText? Does it differ from string?
In SynHighlighterIni, it refers to the string value (text that follow the "=" after the Key, that is not a number).  In INI files, it usually don't have quotes.

In SynHighlighterXML it refers to the content beetwen to labels (tags). It is consider like a string even if it hasn't quotes.

Basically the difference is that tkText are strings without quotes. But I would consider them like some syntactic approach.

I have not decided yet about the scope of the highlighters. I consider that the HL should be just a lexer, but it this way, it wouldn't have to analyze KEYWORDS, that IMHO are syntactic elements. So in some cases we need to extend the scope of the HL.

In my highlighter's documentation, I have moreless, this definition:

Constants, Variables, Directives, Functions could be consider like a sub-category of Identifiers, in most of the cases, but in some syntaxs they could have some lexical special definitions, like the variables in PHP.

For me tkText, is a similar case. It Could be consider as a sub-category of Identifier of it could be a special token (lexical different to identifiers).

So tkText, could be at the same level of Constants, Variables, Directives, and Functions too.
« Last Edit: November 24, 2013, 06:16:39 pm by Edson »
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #19 on: November 24, 2013, 06:23:24 pm »
Code: [Select]
Which highlighters do variables at the moment?
According to I have pointed, the languages who can easy identify the variables like special tokens (lexical level).

They are:

SynHighlighterBat
SynHighlighterSQL
SynHighlighterPHP
synhighlighterunixshellscript
SynHighlighterPerl
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #20 on: November 24, 2013, 06:46:05 pm »
Code: [Select]
If constant is only used by SynAny, then I don't see it that important. And any number and string are a constant. So what to return as default for constant, if there are more than one?
Constants are only used by SynAny. In this case, they are considered as a subcategory of the identifiers, (syntactic level), so they are at the same level that Keywords.

Numbers and strings are constant too, but at the Syntactic or Semantic Level. At the lexical level they are differents tokens category. It depends on what level we want to carry the HL.
« Last Edit: November 24, 2013, 06:50:01 pm by Edson »
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #21 on: November 24, 2013, 07:24:17 pm »
Code: [Select]
What would be examples for TKText? Does it differ from string?
I have not decided yet about the scope of the highlighters. I consider that the HL should be just a lexer, but it this way, it wouldn't have to analyze KEYWORDS, that IMHO are syntactic elements. So in some cases we need to extend the scope of the HL.

In my highlighter's documentation, I have moreless, this definition:

Constants, Variables, Directives, Functions could be consider like a sub-category of Identifiers, in most of the cases, but in some syntaxs they could have some lexical special definitions, like the variables in PHP.

This attempt to structure tokens, on a  global (across all Highlighter) level, is adding to my dislike of the initial idea.

You can define that for a selected set of HL. But any HL is free to break this rules.

By definition, HL is not even restrained to be  a lexer. IT could be anything. Of course a higher level of complexity would bring new problems (speed). But there may be a usecase where this does not matter (e.g. it is known that the data will always be small).

The point is that those ID should not limit the future of HL. This is why I already pointed out that a warning will go along with them, that the concept of such ID may be broken in future.   
As in the example where a HL dynamically generates new Attributes (SynPasSyn already does). Such dynamically generated attributes are only known while parsing the tokens. they can not be returned by getDefaultAttr.


HL may also be nested (asm HL as part of the pas HL) This will lead to more than one default for the same token kind.
With asm, in pas, you may define the default as taken from pas.

But with the SynMulti, there is an outer HL (that may have almost none attribs of its own), and then there are many HL all nested at the same Level. What then?


---
So in conclusion: Those ID, will be an extension, that in a limited number of cases may be used succesful, but with no guarantee of support in future versions (other than in HL written by the person themself)

For that reason, I want to keep any future work resulting out of this at a minimum.


So if it must be

* SYN_ATTR_TEXT    (not my favourite)
* SYN_ATTR_VARIABLE
* SYN_ATTR_DIRECTIVE (used by Pascal and C++)
* SYN_ATTR_ASM  (used by Pascal and C++)

can be added.

 SYN_ATTR_FUNCTION   would be a bad name, they a like a 2nd group of keywords., but next is a HL, wit 3 group of keywords. Or 4 group of numbers.
Pascal has 3 sort of comments(but they shore one attribute / until now, but what if pasdoc will be recognized?)

Pascal has 2 directives (fpc {$...} and ide {%...}
the 2nd has a dynamically created attribute.

---
Absolutely no promise of maintenance.


Still not sure how this will be of use....

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #22 on: November 24, 2013, 07:41:24 pm »
There is also the option of a class helper, that searches through
Attrib[n] / AttribCount.

Using the StoredName

for i := 0 to hl.attribCount do
  if hl.attrib[ i ].StoredName = SYNS_XML_AttrString,

SYNS_XML_AttrString, is a constant in SynEditStrConst

That way all existing attributes can already be found.

Of course it does not solve dynamically created attributes,

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #23 on: November 26, 2013, 06:22:41 pm »
There is also the option of a class helper, that searches through
Attrib[n] / AttribCount.

Using the StoredName


It could be an option, but it depends on attributes of HL have been defined using the correct constant. And there is a lot. (I have counted more than 100, without consider XML ), very likely to confuse.

In fact "SynHighlighterAny", have some attributes defined with some different strings.
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #24 on: November 26, 2013, 06:38:33 pm »
Code: [Select]
This attempt to structure tokens, on a  global (across all Highlighter) level, is adding to my dislike of the initial idea.
It's necessary to have some kind of generalization for working with Scriptable and Multi-Sintax HL.

Of course, not all the Syntax can be included, but they can be managed with a special HL. Far all I can see, most of the current HL of Lazarus can be replaced with a Scriptable HL.
 
Code: [Select]
HL may also be nested (asm HL as part of the pas HL) This will lead to more than one default for the same token kind.
With asm, in pas, you may define the default as taken from pas.

I agree. HL should have the ability for be nested.  I can guess that the current DefaultAttribute depends on what is the current HL. But it's a problem that I haven't faced seriously by now.
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #25 on: November 26, 2013, 06:45:56 pm »
Well but if you want to access *all* attributes by an ID (which is not an option, because I will not enter the maintenance hell of 100 IDs), but in theory, you would need the same amount of ID.

The different ones in SynAnySyn are a bit of an issue.
1) They shoulfd really be updated.
2) That breaks compatiblity, if the StoredName is used as an ID to save the attribute to a file.

SynAnySyn, is really not maintained. It just happens to be there. Hence I newer noticed the problem.

--------------------
Quote
but it depends on attributes of HL have been defined using the correct constant

So does the ID, it must be added, and maintained on each HL. It just adds to the amount of things needing maintenance. And with that to the amount of things that can break.

---------------------
I said I add a certain subset, if patch is supplied. So I will stand by that word of mine.
But I will keep telling you why I thing they are a problem.

The only case where somehing like them is needed, would be, if you wantt to be able to offer a global config, for values that are common across more that one highlighter (e.g. if the IDE would allow to configure the color for comments, and apply it to all supported languages.)

But since there could be multiplry comment attributes in some languages, a way is needed that allows to retrieve *all* comment attributes from a highlighter. Not just one.

In sofar such an ID may be usable. But the current way is wrong.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #26 on: November 26, 2013, 06:49:35 pm »
Code: [Select]
This attempt to structure tokens, on a  global (across all Highlighter) level, is adding to my dislike of the initial idea.
It's necessary to have some kind of generalization for working with Scriptable and Multi-Sintax HL.

Why, example?

Quote
Of course, not all the Syntax can be included, but they can be managed with a special HL. Far all I can see, most of the current HL of Lazarus can be replaced with a Scriptable HL.
 
Different topic. And that is mainly because most of them are kept very basic.

The only not basic one, is pascal, and that (maybe possible) will be hard to make script-able.

Quote

Code: [Select]
HL may also be nested (asm HL as part of the pas HL) This will lead to more than one default for the same token kind.
With asm, in pas, you may define the default as taken from pas.

I agree. HL should have the ability for be nested.  I can guess that the current DefaultAttribute depends on what is the current HL. But it's a problem that I haven't faced seriously by now.

But adding something new, should consider such possibilities. Even if not yet faced. Otherwise we end up with code that we may regret later.

Edson

  • Hero Member
  • *****
  • Posts: 1011
Re: New ID attributes on SynEditHighLighter unit.
« Reply #27 on: November 26, 2013, 07:40:48 pm »
Quote
Well but if you want to access *all* attributes by an ID (which is not an option, because I will not enter the maintenance hell of 100 IDs), but in theory, you would need the same amount of ID.

The idea is not to identify all the possible attributes of a HL.

Code: [Select]
The only case where somehing like them is needed, would be, if you wantt to be able to offer a global config, for values that are common across more that one highlighter (e.g. if the IDE would allow to configure the color for comments, and apply it to all supported languages.)
That's the idea.


It's necessary to have some kind of generalization for working with Scriptable and Multi-Sintax HL.

Why, example?


Do you refer to why it's necessary to make generalization?

I think it's obvious that a Scriptable Highlighter must have a finite quantity of elements and rules in order to cover many syntax. But trying to cover all the existent syntax would be impossible, and it will require a lot of code and process.

Sorry, probably I don't understand the question.
« Last Edit: November 26, 2013, 07:43:50 pm by Edson »
Lazarus 1.6 - FPC 3.0.0 - x86_64-win64 on  Windows 7

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #28 on: November 26, 2013, 08:16:18 pm »
Quote
Quote
The only case where somehing like them is needed, would be, if you wantt to be able to offer a global config, for values that are common across more that one highlighter (e.g. if the IDE would allow to configure the color for comments, and apply it to all supported languages.)
That's the idea.

Ok, but then we only need ID that occur in more than one highlighter.

Here is another thing how, that may need to be handled.

Example perl
   $var1 = "text$var2"

$var2 is both, variable, and string.

If I ever have to much time, and work on the perl highlighter, then either string, or var-attrib will be made a modifier, so that the final result can be a mix.

A modifier attribute has things ilke alpha (how to blend with the other attribute).

Now some HL will have string as basic attib, some as modifier-attrib. How do you copy values.

This problem exists of course, never mind how the groups are done. So it is NOT an argument against any solution.

---
Similar:

pascal already has modifiers:
- there is a normal attribute for directives (fpc style:  {$Ifdef}_
- a modifier for directives (ide style {%region} {%w-}
   the 2nd is based on the 1st

now the default attrib for directives would obviously be the fpc style one. But if you copy some value to it, you should know there is a dependent attrib. So you can have an option to reset it.

Otherwise, if you have: ide-directive = red background + foreground copied from fpc directive
and you change fpc directive to red foreground, then ide directive is hard to read....

---
As for modifiers, they may also modify *unrelated* tokentypes
Modifier for case label, can modify string, number or identifier

Case foo of
  CONST_LOW, 5: ;
end

CONST_LOW, 5 will be modified.

--------------------

Which actually means there is no safe way. ...

Anyway, I indicated which attribs are acceptable. So you can write a patch for them if you want (I will still keep telling you, why I do not like it / but I made the offer, and it stands)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5389
    • wiki
Re: New ID attributes on SynEditHighLighter unit.
« Reply #29 on: November 26, 2013, 08:27:10 pm »
Quote
Do you refer to why it's necessary make generalization?

I think it's obvious that a Scriptable Highlighter must have a finite quantity of elements and rules in order to cover many syntax. But trying to cover all the existent syntax would be impossible, and it will require a lot of code and process.

All rules are user supplied, that makes them finite, since a users lifetime is finite, and that acts as limitation (also available hard disk space)

Otherwise there is no limit.

The HL is a state engine.

Each state is define by either:
- one value out of a userdefined set of values
- a combination of values, each value out of a userdefined set of values

Each state has rules that lead to the next state,

The rule may for example be a keyword. In that case a  keyword may only act in some states.

The same word, may be 2 different keywords, depending on the state.

---

Comments are no exception.

(* is the keyword that enters the state. And for the state only *) is defined as rule.

----
In order to handle nested states a stack is needed.
So that (* does not simply replace the state, but puts the previous state on the stack.
*) then pops the previous state of the stack.

You also need that for coed blocks (In perl or C: {} / pascal: begin end)

---
Some keywords only exist in other blocks. So you do need to keep track of all that. Then you may do it in a generig way.

A script able highlighter has now idea about strings, comments, ... They are all just states.


« Last Edit: November 26, 2013, 08:30:54 pm by Martin_fr »