[SOLVED] Bug in TCustomSynEdit.FindMatchingBracketLogical ?

[EganSolo]

I hope not, but I can't explain the behavior of the code other than by stating there's a bug.

Context: I am writing a custom highlighter for a new programming language. I only mention this to explain why I am not using one of the available highlighters.

The program is raising a 201 inside FindMatchingBracketLogical.

I've carefully traced the code and tried to understand what this code is doing. Without comments, it's a bit tricky. I assumed the error was in my highlighter -- and perhaps it is -- but I can't explain the following behavior:

Here's the trace:

First: The code I am writing in my editor consists of the following two lines

Code: Java  [Select][+][-]

1. {
2. }

That's it. There's nothing else in the SynEdit editor. I've made sure there is no return after the '}'

Now, before the main form becomes visible, the Next function of the highlighter is called 7 times and the problem occurs on the seventh. The function Next returns '}' so it has grabbed the symbol from the second line. 

As the stack unwinds, we go back to SetLine, then to TSynCustomHighlighter.StartAtLineIndex. From there, we go back to line 8566 inside the SynEdit unit. We're in a sub-function called FindMatchingBracketLogical.

Line 8566:

Code: Pascal  [Select][+][-]

1. {8566}  fHighlighter.StartAtLineIndex(PosY - 1);  // PosY = 2, so we're starting at line 1.
2. {8567}  TokenListCnt := 0;

Here, the code is asking the highlighter to begin scanning one line above the current one. So far so good.

Then, on line 8570: I've added my own comment to help along.

Code: Pascal  [Select][+][-]

1.   {8570}  fHighlighter.Next; //We're getting the next token. No issues
2.   {8571}  i := TokenListCnt; //Now i is set to 0. That's the critical issue.
3.   {Skip a few lines..}
4.   {8577} TokenPosList[i].X := fHighlighter.GetTokenPos + 1;
5.   {The token is the right curly brace at column 1, so X should be set to 2, which is exactly what happens}
6.   {8579} If TokenPosList[i].X > PosX then begin //TokenPosList[i {0}].X = 2 from line 8577 and XPos = 1 so the condition is true.
7.   {8580} TokenListCnt := i+1;  //TokenListCnt = 1 now since i = 0
8.   {8581} MaxKnownTokenPos := TokenPosList[i].X // So MaxKnownTokenPos = 2.
9.   {8582} Result := TokenPosList[i-1].Attr := BracketKind // <=== offending line
10.  

That last line is trying to access position -1 of TokenPosList, which does not exist.

So, is the code I wrote in the highlighter leading me to land in the wrong place or is there a missing condition here?

Your help is appreciated. 

[EganSolo]

One additional note: replacing the offending line

Code: Pascal  [Select][+][-]

1. {8582} Result := TokenPosList[i-1].Attr := BracketKind // <=== offending line
2.  

With

Code: Pascal  [Select][+][-]

1. {8582} Result := TokenPosList[i].Attr := BracketKind // <=== offending line
2.  

Seems to work. Nevertheless, I am not certain about the ramifications and I'm hoping someone with greater knowldge of SynEdit than I could shed some light.

[EganSolo]

After carefully tracing through the code, it turns out that the bug was on me  , which is a good thing!
My implementation of GetTokenPos was returning the position of the cursor _after_ the token had been parsed. This then led to the wrong part of the code to be triggered. Once I fixed that, the code as originally created worked without a hitch.

One parting thought: The highlighting architecture of TSynEditor is as powerful as it is complex. Nevertheless, the HighlighterTutorial example program that ships with Lazarus is a good start, but you'll need to study it carefully. The good news about this architecture is that it allows me to plug in my Tokenizer to power the highlighter which means every time I use the IDE, I am actually testing the tokenizer and that is a great boon.

If you are in a similar boat, there are few things you'll need to consider.

1. Your tokenizer must be able to work in Line and Text mode. Normally, your compiler will want to work accross all the text, ignoring line boundaries whenever this is meaningful. For instance, if you're parsing a block comment, your compiler would be perfectly happy if the tokenizer jumped over the entire block in one go (asssuming there are no meaningful bits there, such as directives).  Not so with the highlighter: it operates on a line by line basis, therefore, your tokenizer will have to maintain state and act accordingly.

2. Debugging the highlighter can be tricky since it is responding to events generated by the editor, especially when in the initial phase, your code trips over memory boundary. One helpful thing I did was to create a simple persistent log: In important methods such as SetLine, Next and GetToken, I would load the log from disk into a string list, create a meaningful dump and save it immediately back to disk. This slowed the editor to a crawl but provided a good trace to help me see at which token or which line the code was blowing up.

[Edson]

it allows me to plug in my Tokenizer to power the highlighter which means every time I use the IDE, I am actually testing the tokenizer and that is a great boon.

That's why I have designed a highlighter (https://github.com/t-edson/SynFacilSyn), who is a very goog lexer too: scriptable and supporting basic regex.

1. Your tokenizer must be able to work in Line and Text mode. Normally, your compiler will want to work accross all the text, ignoring line boundaries whenever this is meaningful.

That's why I have created an additional layer over my highlighter (https://github.com/t-edson/t-Xpres). It supports ignoring or not, the line ending, scanning over multiples files, and includes an expresion evaluator.

[Martin_fr]

for debugging the highlighter, search ide/SourceSynEditor. It has TIDESynGutterDebugHL

This displays an additional gutter, that you can fill with data for each line. So you can run a SynEdit (on a form or in the IDE) and display the HL state for each line.

[EganSolo]

Thanks Martin, definitely helpful.

Thus far, I've managed to get the highlighter to work for the list of attributes below. The folding works correctly across all these groups as well.

• Line comments
• Block comments (nested)
• Directives (similar to free-pascal)
• Keywords
• oIdentifiers
• Symbols
• Constants: strings, numbers, and characters.
• variables

[li]
[/li][/list]

I've been studying the highlighter for HTML, XML and the one for Pascal. The trickiest thing was to figure out how much of that code is for parsing versus for actual highlighting. In studying this code, I've got now a renewed respect for SynEdit; it certainly is a complex piece of software. I wish I could find a wiki that explains some of the classes I have seen in more detail.

What I've got to still look into is

• Dim the colors when the directive is not active (Similar to what Lazarus does).
  
  • Make {$Else} be the end of the {$IfThen} block and the start of the next block. That should be relatively straightforward, relying on EndCodeFoldBlock and StartCodeFoldBlock.
  • Rely on my tokenizer to tell me which of these two parts is active. Should be relatively straightfoward based on what was defined.
  • For the part that is not active, I'm thinking the easiest thing to do is to let the parser parse this part and return the appropriate information but then replace the standard attributes with 'dimmed' versions. Hopefully this make sense
• Enable the functionality that allows one to chance one keyword inside a selected block (similar to Lazarus). Is that even possible in TSynEdit?
• Deal with any unforeseen difficulty. That's where your tip will come handy.

[EganSolo]

Hey Hudson,

Thanks for the pointers. I've studied your scriptable highlighter and the t-xpres framework (after an auto-translation from Spanish to English).

They look awesome and I would have used them without a hesitation if it weren't for the fact that the language I am working on is more complex than what the t-xpres framework can handle at the present. Also, I have built a lightweight and very fast in-memory engine to power not only the parser but the actual execution as well. Given that I could not simply abandon what I have in favor of t-xpres, I resolved to stay with my highlighter so that I could leverage the parser for this language.

I will definitely keep an eye on your framework though. Great piece of work!

[Edson]

Hi EganSolo.
 
T-xpres is still in development. There much work to do with it, but it can be used in simple languages. I would say, the main advantages of using T-xpres are: the lexer, and the expresions analyzer (usually a lot of work).
I have used it in some applications:

* https://github.com/t-edson/Tito-s-Terminal  (implements the interpreter)
* https://github.com/t-edson/PicPas (implements the Compiler)

On the other hand, SynFacilSyn is a mature library and has been used in some important projects.

There is another library based in SynFacilSyn, it is SynFacilCompletion, and it is a completion tool using the same lexer-parser of SynFacilSyn.

The SynFacilSyn, SynFacilCompletion and T-Xpres are created to work together, and when they do, they really reuse code.

But like I said, T-Xpres is still incomplete for big projects, so I understand if it doesn't feet your needs.

[Martin_fr]

- dimming:

You may want to consider a Markup (search main synedit unit for the exact class) / markup colors are always merged, so you can dim, by setting a foreground of whatever your background is, and then set alpha channel.

But you can do merging in the highlighter too. See "case" (in synpassys) for example.

Code: Pascal  [Select][+][-]

1. Enable the functionality that allows one to chance one keyword inside a selected block (similar to Lazarus). Is that even possible in TSynEdit?

You mean find/replace in selection only? synedit has a search replace method.

Or you mean when editing the word in one occurrence, all others are edited at the same time?
For the latter, see syncro edit (unit in synedit) / see ide/synSourceEditor for example