How many CustomFoldBlocks are created for a given line?

[EganSolo]

I'm making steady progress on creating a full-fledged highlighter for the new language / parser I'm working on.

I'm documenting quite a bit of it and will share this docs with the group when I'm done.

Here's one thing I'm stuck on:

Consider the following SINGLE line of code, that is, all statements are one line (just be extra clear):

Code: Pascal  [Select][+][-]

1. {This is a comment}begin i := 4; end; while i > 0 do begin dec(i); end;

How many FoldBlocks are created here? I'm thinking the answer is one of the following:

• One: because it's one foldblock per line. I think this is a wrong answer, but not sure.
• None: because everything starts and ends on the same line. I hope that's not the case
• Four: One for the comments, one for the begin-end, one for the while and one for the last begin-end.[/l]
  

I'm hoping it's Four because it will really simplify things in managing Directives.If it is indeed four, can I rely on TSynEdit to set the TopFoldBlock appropriately as the cursor moves through the line?

Would love to know!

[Martin_fr]

Afaik 4 foldblocks.
But no they do not act on cursorpos.

TopFoldBlock can only access them while the line is scanned (with Next()).

Also be aware. They are reused. If you repeat the same line, it will re-use the 4 existing foldblocks (unless the repeated line is nested in other outer foldblocks)

You should use TLazSynEditNestedFoldsList to access fold blocks (note you need to Invalidate it, if the text changes, or it gives old/invalid info)

Search in files, there should be examples in SynEdit folder.

If you are only interested in one line, you can use
IncludeOpeningOnLine and OpeningOnLineCount
that should include folds closing on that line

-----------
Less recommended, but you can also get a single line info
TLazSynFoldNodeInfoList
HL.FoldNodeInfo[LineIdx]


TSynEditMarkupWordGroup.FindMatchingWords shows how to go through the info. (ignore the calls to HL.FoldBlockMinLevel)

Make sure to release the object, or you get leaks
Also again, make sure to refresh it after editing.

------------
If you are in Begin/EndUpdate and text changed , none of them works. You must first EndUpdate

[EganSolo]

Hi Martin,

Thanks as usual,

I've got the folding mechanism working pretty well for my needs. The only thing I want to resolve is the ability to dim a fold block based on directives. In the language I'm designing, things would look like so:

Code: Java  [Select][+][-]

1. $Define $Debug
2.  
3. $Region $MyRegion {
4.   $IfDef $Debug {
5.      //Code goes here
6.      Class MyClass {
7.  
8.       } //MyClass
9.   }//$Debug
10. }//$MyRegion
11.  

As you can see, there's only one fold group in which the directives, regions and the code fit. It's more restrictive than ObjectPascal since you can't have an $IfDef $EndIf block mixed with a class block.

Suppose the editor is now focused on the Class MyClass line. I can navigate the fold chain up to determine if I am inside an $IfDef block, or use a similar technique as in TSynPasSyn where the range is attached to each line. Either way, I can find out if the current line is inside a directive block.

Now, the missing piece is to find out if the directive $Debug is enabled. I had thought that these foldblocks persist as long as the bit of code they are tied exist in the file code. For instance, if I create a non-foldable block for the $Define $Debug, then I could somehow determine if this block exists in the fold tree.

But judging from what your response it doesn't look possible to leverage the TSynCustomFoldBlock that way. I can think of several other approaches:

1. Handle the OnProcessUserCommand and parse the line before it's changed, then parse it after it's changed in SetLines and do a diff on the two list of tokens. If the old had a directive such as $Define and the new list doesn't I would know that there was a directive that has been removed or modified.

 The drawback is dual processing of the same line, before highlighting and after highlighting.

2. Rely on TLazSynEditNestedFoldList: If the fold blocks are reused, I think I'm out of luck.

3. Create an outside process that scans the file OnIdle and updates the IfDef block outside the highlighter. The drawback here is that (a) the response is not immediate and (b) additional code leading to higher complexity.

But thanks to your answer, I know that at least the initial thinking process I was following won't work... that's progress.

Thanks.

[Martin_fr]

I can navigate the fold chain up to determine if I am inside an $IfDef block, or use a similar technique as in TSynPasSyn where the range is attached to each line. Either way, I can find out if the current line is inside a directive block.

Now, the missing piece is to find out if the directive $Debug is enabled.

Well if you go up the chain, you also find the line number, and then you can look at the directive. (
TLazSynEditNestedFoldsList  lets you do that, it should get you the line number too.

If there can only be one directive on each line, and if you want to do it in the HL, then the HL can set a flag in the range of that line. If the line changes, the new state of the flag is set (and you can have an event if the flag changes.)
(The flag would be on the directive line only / not carried forward to the next line)
You can then use the nestedfoldlist do to the rest.


Similar to your point (3):
In the IDE things like define are done by codetools, outside the HL. Therefore define also works, if it is defined on the command line (codetools looks at the project settings).

[EganSolo]

Martin,

If I've understood your suggestion, I would do the following:

• Declare a field of type TLazSynEditNEstedFoldsList in my highlighter
• In the highlighter's constructor, create the NestedFoldsList by passing it SynEdit.TextView.FoldProvider. Please see the attached Class Diagram.
• On SetLines, do I call NestedFoldsList.AcquireFoldNodeInfoList or can I just set Line to the LineNumber Param? I suspect it's the latter since Line is read /write and I've already assigned the HL.
• In the HL, if I set RangeType, it gets stored in TSynCustomHighlighterRange. If I set BlockType, it's stored in an object of type  TSynCustomFoldBLock. But NestedFoldsList doesn't access either. It accesses a TSynFoldNodeInfo, which holds the FoldType. Am I to assume that when I set the Line to LineNumber, some magic happens to populate the SynFoldNodeInfo with the right info about the block type?

Assuming, this make sense, this approach would not only allow me to check if the line is inside one directive group but it would allow me to check if I'm inside several nested $ifDef blocks.

The second portion is the one I'm still struggling with:

if you want to do it in the HL, then the HL can set a flag in the range of that line. If the line changes, the new state of the flag is set (and you can have an event if the flag changes.)
(The flag would be on the directive line only / not carried forward to the next line)

Let me try and break down what you wrote:

• The HL can set a flag in the range of that line: this is similar to what is done in the TSynPasSyn code where the field fRange is a set of flags. I would then add the directive flag to this variable. I think I got that
• The interesting part is here: If the line changes, the new state of the flag is set:
  
  • Does this occur even if the line is cut? Is the HL alerted when a line has been fully deleted? If that's not the case, then I could handle the TSynEdit OnUserProcessCommand and look for the commands that delete a line or a block of lines.
  • Where is that event declared and where can I handle it?

I think I'm almost there...

Thanks for any light you can shed on this.

[Martin_fr]

Ok, a lot of info. Not all of them immediately clear to me.

Lets recap the basics:
* goal is a lowlight like inactive $IFDEF in the ide
* I assume you read the HL tutorial at http://wiki.lazarus.freepascal.org/SynEdit_Highlighter#The_Basics:_Returning_Tokens_and_Attributes  (your knowlegde suggests you did)

Just in case, look at svn too. Some classes may have moved: TLazSynEditNestedFoldsList is in SynEditHighlighterFoldBase (not sure when it moved)

I am not sure why you want to bring in Fold-Provider. I am quite sure you should not need that.

Does this occur even if the line is cut? Is the HL alerted when a line has been fully deleted?

This is why I mention the tutorial. Yes it does. A cut line may affect the following lines (e.g.   "(*\n", if cut will uncomment further lines).

There are 2 different ways to archive your target. So first decide which one

1) The HL does the lowlight
2) You add a Markup to do the lowlight (based on info by HL)

The IDE does 2.
But 1 may be easier (yet less flexible)

Question:
Is "$Define $Debug" the ONLY way to define a debug? And it will always be in the file itself?

In other words, can  the lowlight be done, with info from scanning the current ONE file only? No info from any other place needed?

If that is the case, then 1 is possible. Otherwise I recommend 2

If you go for 1, forget most of what I posted before.

TSynCustomCodeFoldBlock are created for a Blocktype (e.g. cfbtBegin)

You need to dynamically create new Blocktype, and maintain a list of them.

If you encounter  $Define $Debug
you create a Blocktype for that (and use it for all "$Define $Debug" (but only $Debug))
(if the user edits/delete that line, then the HL will rescan until the end of ifle (or an $UnDefine, if exists)

If you encounte $Define $Foo, you create one for Foo, and so on.

In your list (TStringList) you keep the define-name $Debug $Foo...
The Blocktype is x+indexOf, where x is the first free value (if you have 20 cfbt____, then x is 20).

That way you can do all work while scanning.
At the time you scan an ifdef, you can walk up the fold tree.
But you must store on lines, if they are lowlighted, so that if the info changes, then they get rescanned. (the range must change)


I will write the rest later....
But if you do use a markup then all you need is TLazSynEditNestedFoldsList

The higlighter should set a flag in the range, if a line is in an "ifdef" (even if the "ifdef false")
Then the markup knows to go looking for the condition of the IFDEF, or to skip lines without the flag.

Sorry that is a bit cryptic.... more when I have more time....

You really do not need fold provider or selection or any of that.

[Martin_fr]

Does this occur even if the line is cut? Is the HL alerted when a line has been fully deleted?

This is why I mention the tutorial. Yes it does. A cut line may affect the following lines (e.g.   "(*\n", if cut will uncomment further lines).

Oh, I see why you ask.

WEll yes it will be alerted, but...

It will be told, that it needs to scan from that line again.
But the line has already gone, so it will not know that it once was a "special" line.

So if you have a flag, on that line only, then you do not know that it changed.

If you have the flag on all lines (until eof, or $Undefine), then you will note, that the remaining lines have the flag, yet should not have it.
(same for if you have a foldblock)

So my idea with the flag on the line itself will not work.
You can still use it, if you want to compile a list of $DEfine outside the hl, and then use events like
                           senrLineCount,        // Lines Inserted or Deleted (if not empty, they will trigger senrLineChange too)
                           senrLineChange,       // Lines modified (also triggered by senrEditAction)
to update the list.

That gives you inserted, deleted and edited lines.
You do not see what was deleted, but if line 138 to 143 was deleted, and your list contains 139, then you know it no longer is.
Also if your list has line 160, then adjust it to 154.

If you have a rather large list of defines, then I can point you to a tree, that allows such updates with one operation for ALL entries. (So that is an optimization you would want to leave for when all else is done)

[EganSolo]

Hey Martin,

It's been a few days... and I've been trying to understand how to achieve my goal.
Recall that what I am trying to do is to update a conditional block based on the status of the nearest defined (or undefined) directive.
To do so, I need access to the conditional blocks for this directive. For instance:

Code: Java  [Select][+][-]

1. $Define $Debug
2.  
3. $IfDef $Debug {
4.    //This block should be defined.
5. }
6.  
7. $UnDef $Debug
8.  
9. $IfDef $Debug {
10.   //This block should not be defined. It should be dimmed.
11. }
12.  

Now that I understand TSynEdit a bit better, I'm wondering if I could piggy-back off TSynEditFoldedView?TSynEditFoldedView maintains a fold tree of type TSynTextFoldAVLTree, which uses TSynTextFoldAVLNode to represent its nodes. Please see the attached diagram. I've highlighted the classes of interest in that diagram.

Unfortunately, TCustomSynEdit instantiates its TextView by directly calling the constructor on TSynEditFoldedView instead of relying on a dynamic approach that would allow us to create a descendant of TSynEditFoldedView. Likewise, TSynTextFoldAVLTree.NewNode calls TSynTExtFoldAVLNodeData directly to create a new node, and there does not seem to be any way to replace this node with a subtyped node.

I've looked also at TSynEditFoldProvider and this class does not have any public or protected virtual methods that could be used to react to changes in the TSynTextFoldAVLTree. Incidentally, TSynTextFoldAVLTree does not have any events either when it inserts, delete or modify a node.

So, the other way to approach this is to insert event handlers into the TSynEditStringList to react to its events which I can do. Say that a line has been inserted. I can react to this event and update my own version of TSynTextFoldAVLTree (this is really an interval graph in a balanced tree). The difficulty here is that I'm duplicating a structure that already exists (TSynTextFoldAVLTree) instead of using it.

Any suggestions? Would it make sense to modify some of these classes so that they become a bit more dynamic and allow descendants to handle added functionality?

What do you think?

[Martin_fr]

I think nothing in the unit FoldedView will be of help (except maybe by serving as example)

But lets look at the classes you picked out:

Unfortunately, TCustomSynEdit instantiates its TextView by directly calling the constructor on TSynEditFoldedView

That may get fixed, but first the inheritance of TSynEditFoldedView  must be fixed (it needs to be a TSynEditStringsLinked).
And there needs to be a manager for all TSynEditStringsLinked. And that would be where hooks can go in.

I've looked also at TSynEditFoldProvider and this class does not have any public or protected virtual methods that could be used to react to changes in the TSynTextFoldAVLTree. Incidentally, TSynTextFoldAVLTree does not have any events either when it inserts, delete or modify a node.

TSynTextFoldAVLTree.
This ONLY store currently folded nodes. (if you collapse a fold in the editor, it will be added to that list)

TSynEditFoldProvider
Allows to find where (on which line) text can be folded.
It mostly forwards info from the highlighter.
But it also adds the current selection (as you can hide selected text)

TLasSynDisplayView (in general, not specific to the one in FoldedView )
Well this could be used to modify highlights. But it really by design is not the place. (not for your case)
It *maps" the text in memory, to the display-able text.
For example it adds the [...] (the marker far a collapsed block) into the text, so this is painted with the normal text.
It can modify highlights, but really be careful with that.

There are 2 places, and only 2, that you should consider for highlight.
1) the highlighter (and yes it can mix colors (see pascal "case" labels)
2) the markup classes (SynEditMarkup...)
Both will work for you.

The difficulty here is that I'm duplicating a structure that already exists (TSynTextFoldAVLTree) instead of using it.

Actually you would be using it for other data.
Yes you can use this class to keep track of your locations.

The existing instance, is empty if nothing is currently collapsed. So it does not contain the data you need.

-----------------------------------

1) choice you need to make
how to maintain the location of define/undef/ifdef
a) you can do in your own code
b) you can do in your highlighter.
But for b, you must add extra foldblocktypes. (I did hint on that before)

If you do use foldblocks for anything, and you need to find them:
   TLazSynEditNestedFoldsList
You can give it a line, and it gets you info for that line, and finds previous lines with foldblocks still open at the current.

The only way to find all lines with a certain foldblock-type, is to scan through all line.
If you need an index, you need to keep it yourself.
(In the IDE codetools keeps track of define/ifdef // though the HL has ifdefs, like any other fold, but only accesible if you know the line already)

2)
if you tread all of them as fold blocks, you can do the highlighing in the highlighter too.
If you keep info outside the HL, then you will need a Markup for it.

-----------------------
folding of a define/ifdef
- say you have 19 blocktyes already (begin/while/try/procedure/....)

you have a stringlist with the names of defines

when parsing a "DEFINE foo"
  n := stringlist.indexOf(foo) // case sense ??? // add if not yet in list
  n := 20 + n *4 ; // 3 is enough but 4 maybe easier / we use 20 because it divides by 4
  StartCodeFoldBlock(n)

same for UNDEF, close the folds

for IFDEF foo, ENDIF again the same, but
  n := 20 + n *2 + 1; // use +1, so they do not conflict
OR
  n := 20 + n *2 + 2; // use +2, if this should lowlgiht

when you encounter an IFDEF, during scanning (in procedure Next)
so you do have the current range, and you can walk up the tree of foldblocks.
 n2 := n and $fffffffC // last 2 bits   // value dividable by 4
now when you open

search the tree of foldblocks, for n2, if it is in the tree, "foo" is defined.

if foo is defined use "+2" otherwise "+1"

if you use +2 set a flag on the range. Use the flag when you decide if you lowlgiht the token.

on endif, clear the flag (but check outer nodes, if they have +2 then keep the flag)

[EganSolo]

Martin,

I think this will work.
Assume the functions IsIfDefBlockType(BlockType) and IsIfNotDefBlockType(BlockType return true if the block type is of type bt_IfDef and bt_IfNotDef respectively. this can be easily achieved by reserving the low word of BlockType to the block type and the high word to the directive index. Please take a look at the code below and let me know if this is moving the way you were thinking.

Code: Pascal  [Select][+][-]

1. function DoDim(aToken) : Boolean;
2. begin
3.      if aToken.TokenType = tt_Directive
4.      then case aToken.DirectiveType of
5.        dt_Define  : If fRange * [rf_Active] = [rf_Active] //we're in an active range.  
6.                           then begin
7.                               n := ListOfDirectives.IndexOf(DefineName);
8.                                if n = -1
9.                                then begin
10.                                    ListOfDirectives.Add(DefineName);
11.                                    n := Count;
12.                               end;
13.                               n := 20 + n*4;
14.                               aBlockType := MakeBlockType(bt_Define, n);
15.                               StartCodeFoldBlock(aBlockType,{Folding = } False);
16.                           end;
17.                
18.        dt_Undefine: If fRange * [rf_Active] = [rfActive]
19.                             then begin
20.                               n := ListOfDirectives.IndexOf(DefineName);
21.                               if n = -1
22.                               then begin
23.                                 ListOfDirectives.Add(DefineName);
24.                                 n := Count;
25.                               end;
26.                               n := 20 + n *4;
27.                               aBlockType := MakeBlockType(bt_Define, n);
28.                               EndCodeFoldBlock(aBlockType);
29.                             end;
30.                
31.        dt_IfDef,
32.       dt_IfNotDef:  begin
33.      //Process block even if not in an active block to allow
34.      //the folding of IfDef even when dimmed.
35.                       n := ListOfDirectives.IndexOf(DefineName);
36.                       if n = -1
37.                       then begin
38.                          ListOfDirectives.Add(DefineName);
39.                          n := 20 + ListOfDirectives.Count;
40.                         If DirectiveType = dt_IfDef
41.                         then begin
42.                           System.Inc(n); //Directive not defined Add 2 to n.
43.                          fRange := fRange - [rf_Active];
44.                       end
45.                       else begin
46.                          System.Inc(n,2); //Directive not defined and it's an IfNotDef block. Add 1 to n.
47.                         fRange := fRange + [rf_Active];
48.                      end      
49.                  end
50.                  else If fRange * [rf_Active] = [rf_active]
51.                  then begin //We're in an active range but we need to confirm that it is still active.
52.                         n2  := 20 + n * 4;
53.                         Fold := CodeFoldRange.Top; //Fold is of type TSynCodeFoldBlock
54.                   While (Fold <> Nil) and (Fold.BlockType <> n2) do
55.                      Fold := Fold.Parent;
56.                   if Fold = Nil
57.                   then if DirectiveType = dt_IfDef
58.                        then begin
59.                          n := 20 + n * 4 + 2; //Directive not defined, dimm IfDef block
60.                         fRange := fRange - [rf_Active];
61.                        end
62.                       else begin
63.                          n := 20 + n * 4 + 1; //Directive not defined, don't dim IfNotDef block
64.                         fRange := fRange + [rf_Active];
65.                      end
66.                   else //switch the statements above: activate the block for IfDef and dim it for IfNotDef.
67.                  aBLockType := MakeBlockType(DirectiveType, n);
68.                   StartCodeFoldblock(aBlockType);
69.                  end;
70.                end;
71.        dt_Else    : begin //Since I'm assuming one fold group, then
72.                   if fRange * [rf_Active] = [rf_Active]
73.                   then fRange := fRange - [rf_Active]
74.                  else begin
75.                    //Range is inactive. But is it inactive because of the IfDef portion or is it
76.                    //inactive because the entire block if IfDef -- Else -- EndIf was already in an inactive
77.                   //block?
78.                   If BlockType < 20 //something went wrong. We should be in an IfDef or IfNotDef block.
79.                   then SignalError('Malformed Conditional block')
80.                   else begin
81.                     n := BlockType and $FFFFFFC;
82.                           Fold := CodeFoldRange.Top; //Fold is of type TSynCodeFoldBlock
83.                          While (Fold <> Nil) and (Fold.BlockType <> n2) do
84.                        Fold := Fold.Parent;
85.                     if Fold <> Nil
86.                           then //the directive is defined. Now do the opposite of the IfDef or IfNotDef block
87.                             If fRange * [rf_Active] = [rf_Active]
88.                       then fRange := fRange - [rf_Active]
89.                       else fRange := fRange + [rf_Active];
90.                         end          
91.                end;  
92.        dt_EndIf   : If BlockType < 20 //something went wrong. We should be in an IfDef or IfNotDef block.
93.                then SignalError('Malformed Conditional block')
94.                else begin
95.                   EndCodeFoldBlock(BlockType);
96.                        Fold := CodeFoldRange.Top; //Fold is of type TSynCodeFoldBlock
97.                   If Assigned(Fold)
98.                   then Fold := CodeFoldRange.Parent; //go to the next block.
99.                      While (Fold <> Nil) do
100.                   begin
101.                           If (IsIfDefBlockType(Fold.BlockType) or IsIfNotDefBlock(Fold.BlockType)
102.                     then begin
103.                      fRange := fRange + [rf_InConditionalBlock];
104.                             If Fold.BlockType mod 2 = 0
105.                      then begin
106.                         fRange := fRange - [rf_Active];
107.                         exit;
108.                      end;
109.                     end;  
110.                     Fold := Fold.Parent;  
111.                    end;
112.                   If (Fold = Nil)
113.                   then begin
114.                      //None of our parents is a disactivated conditional fold.
115.                      fRange := fRange + [rf_Active];
116.                      fRange := fRange - [rf_InDirectiveBlock];
117.                   end;
118.                end;  
119.     end ;//case
120.     Result := fRange * [rf_Active] = [];
121. end;
122.  
123.  

[Martin_fr]

I will have a look... hopefully soon.

just one thing I noted

Code: Pascal  [Select][+][-]

1.                               n := ListOfDirectives.IndexOf(DefineName);
2.                                if n = -1
3.                                then begin
4.                                    ListOfDirectives.Add(DefineName);
5.                                    n := Count;
6.                               end;

n:= count -1;

or
                                   n := Count;
                                   ListOfDirectives.Add(DefineName);

[EganSolo]

Yep sorry. it was late while I was coding this.
I'm cleaning the code as we speak and will repost once I see how this is running.
Thanks.

[Martin_fr]

Any update, did you get it to work?