Folding in SynEdit

[Morton]

There had been an AnyHighlighter by James Jacobson using a previous version
of the state machine.
The AnyHighlighter in Lazarus has nothing to do with it.

If someone creates a state class for every imageable generic expression the state machine could handle everything imaginable.
Complete Compilers can be implemented as state machines.
Indeed everything that can be broken down into states.

A scriptable folding highlighter for any imageable language would be no problem.
It just has to done, but not by me.

[Edson]

Here I'm again with folding.

I was analyzing the code of folding on the SynLFMSyn and some others HL, and my question is:

What's the difference on implementing folding, using:

Code: [Select]

function TSynLFMSyn.StartLfmCodeFoldBlock(ABlockType: TLfmCodeFoldBlockType): TSynCustomCodeFoldBlock;
var  FoldBlock: Boolean;
  p: PtrInt;
begin
  FoldBlock :=  FFoldConfig[ord(ABlockType)].Enabled;
  p := 0;
  if not FoldBlock then
    p := PtrInt(CountLfmCodeFoldBlockOffset);
  Result := StartCodeFoldBlock(p + Pointer(PtrInt(ABlockType)), FoldBlock);
end;

procedure TSynLFMSyn.EndLfmCodeFoldBlock;
var  DecreaseLevel: Boolean;
begin
  DecreaseLevel := TopCodeFoldBlockType < CountLfmCodeFoldBlockOffset;
  EndCodeFoldBlock(DecreaseLevel);
end;

And simply:

Code: [Select]

    StartCodeFoldBlock(nil);

    EndCodeFoldBlock();

I know it has to be with the manage of ranges of TLfmCodeFoldBlockType. But how it works?
There is not documentation about StartCodeFoldBlock().

[Martin_fr]

I added some info

http://wiki.lazarus.freepascal.org/SynEdit_Highlighter#Step_3:_Add_Folding

The pascal variants:
- typecast the

Code: [Select]

  TPascalCodeFoldBlockType = ( // Do *not* change the order
    cfbtBeginEnd,      // Nested
    cfbtTopBeginEnd,into a pointer
- encode (by adding an offset number), if a fold starts here.
  In the EndFoldBlock is a check if "TopCodeFoldBlockType" is greater/equal to the offset.

FCatchNodeInfo should be ignored for now. It will neither be documented, nor will it be kept compatible, until it moves to the base class.

[Edson]

Thaks Martin.

It's good to have more documentation on the Wiki  . It clears some of my doubts.

Would you please tell me if the parameter 'ABlockType' (a pointer)  have the same use that the parameter 'Value', when we use Ranges:

Code: [Select]

procedure SetRange(Value: Pointer); virtual;
 function GetRange: Pointer; virtual;
 procedure ResetRange; virtual;

I mean,
It's stored internally, for each line of the text?
Is it read at the end of each line?.
Can I use it for storage (point to) some complex object?

[Martin_fr]

No they are different.

Look at the doc, if you use the fold-hl, then the RANGE is managed by the base class.
It becomes an object(you can change the class of it, if need it, as long as it has the right base class)

Your range, is then in
 FCurRange := PtrInt(CodeFoldRange.RangeType);

The ID only identifies the fold type e.g. in pascal
begin=1
try=2
repeat=3
...

The range PtrInt(CodeFoldRange.RangeType) can still be used for other things. Thinks where you do not want a foldblock.

Foldblocks have the advantage that
- they can be nested
- you can find the start line very fast (see the section on "EndLvl")

Yet if you have range_InAssignment, (that is between ":=" and ";") then you need no foldblock.
You can use the range as enum/set, and flag the state there.

[Edson]

Understood.

I have enough information for continuing working.

In my case, I can't use enumerated ID's for the fold type, because I need to create them dynamically, and store different properties on them.

I'm going to use real pointers for objects on ABlockType of StartCodeFoldBlock().

[Martin_fr]

I am not sure what you want to store, but it does not sound ... well ...

look also at

    function GetRangeClass: TSynCustomHighlighterRangeClass; override;

unit
SynHighlighterPas

It returns TSynPasSynRange

For the pas HL, on each line an object of this class is stored.

IMPORTANT: it must have a compare method that compares *ALL* (each and every) field (or piece of data) on the class (see TSynPasSynRange.Compare). Inherited data is done in inherited.
(also assign)

PascalHL has

from inherited
  property RangeType: Pointer read FRangeType write FRangeType;
which is a set of enum (TRangeState)

folds for begin/end etc

   property BracketNestLevel: integer read FBracketNestLevel write FBracketNestLevel;
a counter for ()

counter for IFDEF folds

....

-----------------
Instead of subclassing, you can also store an object in
  property RangeType: Pointer read FRangeType write FRangeType;

But subclassing is more efficient. It will avoid storing duplicate objects.

[Edson]

This leaves me more questions.

    function GetRangeClass: TSynCustomHighlighterRangeClass; override;

What is this for?

For the pas HL, on each line an object of this class is stored.

Why is needed to store an object per line?

PascalHL has

from inherited
  property RangeType: Pointer read FRangeType write FRangeType;
which is a set of enum (TRangeState)

folds for begin/end etc

For me, "RangeType" is used for manage the "ranges" for highlight multiline tokens (or other information per line). Isn't it?

I'm getting some confused.

For all I know:

Ranges
=====================================
When we use a HL derived from "TSynCustomHighlighter", we only manage "Ranges", for store some information per line and/or whose state depend on previous line.
We do this, using an enumerated "fRange" and implementing:

Code: [Select]

procedure TSynMiColor.ReSetRange;
begin
  fRange := rsUnknown;
end;

function TSynMiColor.GetRange: Pointer;
begin
  Result := Pointer(PtrInt(fRange));
end;

procedure TSynMiColor.SetRange(Value: Pointer);
begin
  fRange := TRangeState(PtrUInt(Value));
end;


Blocks of Folding
=====================================
When we use a HL derived from "TSynCustomFoldHighlighter", we manage the "Ranges", and aditionally "Blocks of folding" (do we call them Ranges too?) for store information about the folding (included nesting).
We use too an enumerated "fRange" (for common ranges) and we have to implement:

Code: [Select]

procedure TSynFacilSyn.ReSetRange;
begin
  inherited;
  fRange := nil;
end;
function TSynFacilSyn.GetRange: Pointer;
begin
  CodeFoldRange.RangeType := fRange;
  Result := inherited GetRange;
end;
procedure TSynFacilSyn.SetRange(Value: Pointer);
begin
  inherited SetRange(Value);
  fRange := CodeFoldRange.RangeType;
end;

Is it OK? Probably I'm wrong in my concepts or terms.

I think they are different things. But, are they (ranges and blocks) related in some way?

[Martin_fr]

Ranges:

You may remember from some other thread, that apparently ranges were initially a "range of lines to which applies some condition".
For each line we store that "Range".

The range is stored on each line. If it is not changed, then the same range is stored again.

For me, "RangeType" is used for manage the "ranges" for highlight multiline tokens (or other information per line). Isn't it?

In some Highlighters this ranges are now very detailed.

Code: [Select]

type
  TFoo = class
  // ..
  end  ; 
  deprecated
Will NOT highlight the word "deprecated", but if you take the ";" away, then it will. So after the ";" starts a new range, or the other way round, there was a range that ended at the ";".

Being in a fold able block, is also part of the range. It may not (always) be something that affects how the code in this lines is parsed, but it is an attribute that applies to those lines (and besides it is a convenient place to store it)

So anything the HL needs to store, that is needed in a later line, goes into the range.

e.g. fold info is needed at the closing line. It is carried forward. So no loo back is needed.

Why is needed to store an object per line?

Because the amount of flags and info does not fit into the 32 or 64 bits of a pointer.

But yes, it is also true to extend the storage size in the array of ranges (1 per line), so there are 128 bits.

---------------
As before: The range is stored on each line. If it is not changed, then the same range is stored again.

If it is an object, ideally we do not copy the object, but duplicate the reference. However then the object stored must be immutable. If you changed it, that would affect many lines instead of just one.

The fold base HL does introduce all that is needed to deal with such an object. And it forces all inherited HL do use that object.

and aditionally "Blocks of folding" (do we call them Ranges too?)

The entire object is called the "range" now. (Actually the range is the part of code/text to which it applies).

The tutorial only showed the basics.

The range can hold whatever is needed.
But if designed well, you must look at implications such as memory.

Look at pascal, using objects. Open the IDE, and open a handful of units. You easily have 100.000 lines or more of source. Imagine a range object for each line. that are a lot of objects.
In fact, even with 1 million lines, the IDE only need approx 2000 objects In rare cases I have seen 4000.

-----------------------------
If you use the fold ID to point to an object, then you need to manage this your same.
You either get one per line, or must organize to re-use them. Well you get one for each fold-opening line, but worst case that is every line)

BTW: Those ID, are stored as part of the range-object.

[Martin_fr]

Also note, that ranges (and changing of their value) is used to find out how far the HL must scan.

Code: [Select]

if a then begin
  (* // Some commented initialization:
  x := 0;
  b := a;
  Init(Foo);
  // *)
  Foo.DoSome;
end;

Now if the line

Code: [Select]

  b := a;is edited, and changed into

Code: [Select]

  b := a; *)then only the text in that line changed.

That means to start with, only that line needs to be scanned. But of course that is not enough.
So a way is needed to know how far the scan must go. After all, we also do not want to scan to the very end of the file each time (that could be a million lines)

So when

Code: [Select]

  b := a; *)it will look at the range that it returns for the end of that line.
It compares it to the old value.

* If it is equal it is done.
* If it is changed, then the next line now start in a different range, and the next line must be scanned too,

And this is repeated, until a range is kept equal.
In the example, that is at the end of " Foo.DoSome;"

[Edson]

Thanks Martin.

I think I understand the concept of Range (like pointers stored on each line). I have implemented some hardcoded Highlighters, and in my Scriptable Highlighter I use the range like pointer to a record and it works OK (maybe I'm the only tester).

But what confuse me, is the terminology on folding. I see "ranges" and "blocks", and I have no definitions.
I can suppose that block is "A logical representation of some extension of text that can be folded and nested". Is it OK?

Probably for programming, the concepts are not important (we create our own in mind), but for documentation yes.

In some Highlighters this ranges are now very detailed.

Code: [Select]

type
  TFoo = class
  // ..
  end  ; 
  deprecated
Will NOT highlight the word "deprecated", but if you take the ";" away, then it will. So after the ";" starts a new range, or the other way round, there was a range that ended at the ";".

I guess you are talking about that, HL can use the information of Folding for parsing. If so, then it's clear.

But I still have the same my question:

Are Ranges and Blocks(of folding) different things?

I mean, that we can use one of then without implementing (or leave for default) the other?

I think the response is Yes. But it will be nice have your opinion.

[Martin_fr]

The terminology is used quite loosely. After all the original meaning of "range" was not known any more until recent (thanks to morton for explaining it).

"Range" is still used in many ways. Does identify a "code snippet"? Or some ID (the sum of the data stored in the field called range)?

If we go by "a range of lines" then I would say the actual code....
And it can start in the middle of a line. It can even end on the same line it started.
The info stored in the range field, is just a snapshots of the identification of the range (a snapshot at the end of each line).

That is why I like to think of it as "the internal state of the Highlighter at the end of a line"

----
Block (if used as foldblock) then yes, some form of reference to foldable code.

In that sense a block is part of the information stored about a range.

Are Ranges and Blocks(of folding) different things?

A fold block is one way to start a new range.

fold-block-info is part of the range info.

I guess you are talking about that, HL can use the information of Folding for parsing. If so, then it's clear.

It can. It does not always have too. It sometimes does not need it at all.

I was not trying to make any statement about this.

I mean, that we can use one of then without implementing (or leave for default) the other?

You can use ranges, without folding, every none fold HL does that.

The foldHLbase implements folds-info as part of the range. So it always uses both.

If you open/close a fold, then the base class changes the current range. (as it stores the fold info within)


You can of course use folds, and not add any of your own info to that range. (but the range info from the base class is still there.

[Edson]

It's clearer now.

Although the Range and Fold Block, have different uses (but could be related), they both use the same mechanism of ranges for to work. It has sense.

A fold block is one way to start a new range.

fold-block-info is part of the range info.

If the fold block info and the range info are shared. Is there some limitation on that.?
I say it, because I'm using pointer for ranges and for fold blocks (not enumerated ID's), so I'm using the 32 bits of data. I haven't noticed some problem by now.

Until now, I have been success with scritptable foldable HL. Just I'm having problems with nested foldable blocks. I have the next doubts:

A Start delimitier (like "BEGIN"), can only open a block. Is it OK?

But an End delimitier (like "END"), can close two fold blocks: PROCEDURE ... END and BEGIN ... END

Can it close 3 or more blocks?

[Martin_fr]

It is the highlighters decision how many blocks to open or close.

Say I wanted

try
except
end

to be able to fold either "try except" or "try end" then I would open 2 blocks at the try.

But there  can only be one [-] in the gutter, so it is very hard to access (gutter pop up menu)

Overall in pascal it makes no sense to open more than one block for a keyword.

-----
"end"
look at

Code: [Select]

function TSynPasSyn.Func23: TtkTokenKind;
  if KeyComp('End') then begin
It can close a lot.

Code: [Select]

Case a of
  1: foo
   else bar
end;
NOTE: no ";" before the case-else needed.

Code: [Select]

Case a of
  1: If x1 then
        If x1 then
           If x1 then foo
          //else bar
end;
     
Each "then" opens an internal block (it is marked not to be foldable / does NOT increase the fold-level), because it means the "else" belongs to the IF, and not to the case.

So above could have 3 else for the 3 if, and one for the case.

The end closes them all, and it closes the case.

[Edson]

Thanks Martin.

I can see. There is several figures, not all present on Pascal language.

Now I just have to find the way to express it on a XML description.

By now, I have:

<block> </block>