Print Page - [Solved] ShellTreeView Node Expand: why second expand is slower than first one?

Programming => LCL => Topic started by: d7_2_laz on November 27, 2022, 05:04:31 pm

Title: [Solved] ShellTreeView Node Expand: why second expand is slower than first one?
Post by: d7_2_laz on November 27, 2022, 05:04:31 pm

Coincidentally seen (Win x64): open (eg. via the ShellView demo) a highly populated folder (eg. Windows\WinSxS).
Collapse it. Expand it again ... appears to be much slower than the first expand. Why?
Would have expected that the second expand runs instantly, as the node is already populated.
Why the node is populated each time? (I assume there is a reason)

Changed simply for testing CanExpand with instant second expand:

Code: Pascal [Select][+]

function TCustomShellTreeView.CanExpand(Node: TTreeNode): Boolean;
var
  OldAutoExpand: Boolean;
begin
  Result:=inherited CanExpand(Node);
  if not Result then exit;
  OldAutoExpand:=AutoExpand;
  AutoExpand:=False;
  BeginUpdate;
  try
    Result := True;
    if Node.Count = 0 then begin     // Added simply for test
        //Node.DeleteChildren;       //  Not becessary regarding the condition
        Result := PopulateTreeNodeWithFiles(Node, GetPathFromNode(Node));
    end
    else  Result := True;           // Added
    AutoExpand:=OldAutoExpand;
  finally
    EndUpdate;
  end;
end;

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: wp on November 27, 2022, 06:38:01 pm

I had thought that child nodes would be destroyed when their parent is collapsed, in order to save memory load. But I cannot find anything related to "Collapse" in the ShellTreeView code - so, probably I am wrong, and your modification would be a solution.

On the other hand, why not implement that "Free-on-collapse" thing? Would there be a disadvantage in having it?

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 27, 2022, 07:37:36 pm

Hi wp, i'm simply not yet deep enough inside the Shell tools for to understand
- why at all a population might be triggered by a CanExpand query
- and why at all nodes might be cleared or childs destroyed when a node is collapsed.
To my simple understanding maybe intuition a node collapsed is a potential candidate to be re-opened, so i never would clear it when being collapsed.
I assume there is a good reason to do so, but i don't know it. Simply missing of info on my side. What's the benefit to clear? Save memory? Never did run into a bottleneck here.

Beyond intuition, most other file tools i know about re-open (re-expand) nodes instantly and that wuld be my own expectation too.

A nice side-effect i could imagine is that a re-populated node will on-the-fly catch all modifications of the folder subtree needed from actions outside.
But imo that would be rather a question of how to deal with file change monitors.

I was simply really surprised that a subsequent expand takes (felt:) the doubled time of the first one (may be internally the population is done twice?).
The disadvantage of a "Free-on-collapse" in my eyes: why not keep information that is already there but destroy it? Would lead to much redundant work.

I don't think the nodes are cleared intentionally (eg. at collapse). It's simply that they are ckeared and re-built each time a node is re-opened again. (no memory save advantage could be seen here).

For to avoid misunderstanding: i feel the shell tools are excellent - that's why I want to use them more and more for new things.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 28, 2022, 10:18:17 am

I experimented a while using the change mentioned above (within CanExpand) and did not find any negative indication why it should not be applied.

So i did open a issue note on that:
https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/40022

Supplementary remark: i'd expect that when selecting a subnode "Sub1" beyond parent node "Parent1", that when i collapse Parent1 by click on the expand sign, and re-expand it again, the previous selection marker on Sub1 is preserved.
Using the change, it stays preserved. By the current behaviour (= sub nodes cleanup) it goes lost. Not ok. Please keep my selection!

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: balazsszekely on November 28, 2022, 12:41:17 pm

@d7_2_laz
From the bugreport:

Quote

when re-expanding the node, this is done instantly without clearing/repopulating the node
No more node population is done again redundantly. And certainly not it should take twice the time.

I'd expect that an already populated node can be re-expanded instantly.

What if the collapsed node's children are physically deleted from the hard drive while the node is collapsed? Isn't the (re)read on expand intentional? I wouldn't call it redundant. Maybe you can introduce a new flag, when set to true, the disk changes are ignored for an already loaded node.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 28, 2022, 01:39:55 pm

I'dthought about that as above

Quote

A nice side-effect i could imagine is that a re-populated node will on-the-fly catch all modifications of the folder subtree needed from actions outside.
But imo that would be rather a question of how to deal with file change monitors.

Of course it makes sense to take care about external done file changes (implying fle change monitors). But then - consequently, amd not only at one single place (a re-opened node). What about siblings of the parent node, if a folder is deleted: why should they stay within the tree, whereas a re-expanded node should respect duch changes?
At this very basic level external file and folder changes necessarily are not yet in scope.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 28, 2022, 01:47:14 pm

Additionally.if you have a dirtree - filelist twin pair, you re-expand a node and expect to see folder changes hrtr, then one necesarily needs to re-read the listview's content completely too, otherwise dirtree and listview would go out of sync.
Expand = reread whole subtree. plus reread whole lfile - listview?

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: wp on November 28, 2022, 02:33:23 pm

I think I'll add an ExpandCollapseMode property:

Code: Pascal [Select][+]

type
  TExpandCollapseMode = (
    ecmDefault,          // Current behaviour (--> default)
    ecmKeepExpanded,     // Do not clear children of already-expanded, but collaped nodes (d7-2-laz's code)
    ecmCollapseAndClear  // Clear children when node is collapsed
  );

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 28, 2022, 02:53:04 pm

That'*s surely the best ... seems different preferences might exist :)

Else i had not other findings. Wow! And it's so fast! (i'm using ownerdata for the shell listview)
:)

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: Bart on November 28, 2022, 03:04:53 pm

Quote from: wp on November 28, 2022, 02:33:23 pm

I think I'll add an ExpandCollapseMode property:
Code: Pascal [Select][+][-]
type
TExpandCollapseMode = (
ecmDefault, // Current behaviour (--> default)
ecmKeepExpanded, // Do not clear children of already-expanded, but collaped nodes (d7-2-laz's code)
ecmCollapseAndClear // Clear children when node is collapsed
);

ecmDefault is not very descriptive.
The fact that you make it default should already be reflected with the default modifier for the property.

Bart

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: wp on November 28, 2022, 03:26:52 pm

ecmLegacy? ecmClearExpandingAndKeepCollapsing?

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 28, 2022, 03:42:28 pm

ecmRefreshedExpanding?

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: wp on November 28, 2022, 11:09:55 pm

In the attachment there is a modified shellctrls.pas in which I attempt to implement the ExpandCollapseMode. Make a backup of your original shellctrls.pas in the lcl folder of your lazarus installation and copy the attached file over the original one.

The attachment contains also a small project for testing. I tried to measure the time it takes to expand or collapse a folder by storing the clock times at which the OnExpanding/OnExpanded or OnCollapsing/OnCollapsed event fire. My impression however is that this is not very correct since it feels much longer when the WinSxS folder collapses. However the qualitative difference between the ExpandCollapse modes is as expected

I also called GetHeapStatus.TotalAllocated to measure the change in occupied heap memory with respect to the memory used in the form's OnCreate event. Maybe there is some doubt on the exactness of these numbers, but again, the memory usage is as expected.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 29, 2022, 10:17:36 am

Thank you wp!
Seems it needs to be applied on a recent laz-main extract (due to other external dependencies), not the 2.2.4.

Tried the test project.
Of course i was focused onto the option 2 "Do not clear expanding nodes, keep collapsing nodes".
For the re-expand: yes, perfect! Just as expected.
For the collapse: when using option 2, it appears to run same instantly too (if there should be any minimal delay, i think it's neglecticable).
Also, with option 2, a selected child node keeps to be selected when collapsing/re-expanding its parent node (no mystery, why); ok.

Exactly just how it should be ... :)

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: wp on November 29, 2022, 12:26:49 pm

Committed the new shellctrls to main.

What's strange is that deletion of the child nodes in the C:\Windows\WinSxS subtree takes much longer than adding them...

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 29, 2022, 12:56:57 pm

Quote

What's strange is that deletion of the child nodes in the C:\Windows\WinSxS subtree takes much longer than adding them...

(with option 1, previous behaviour, for to assure)
Yes, i saw that yesterday evening too. What i find strange: when retrying it now again, there is no delay at collopse (option 1). Sometimes yes, sometimes no? Hmm
Will try some more retests about that (especially collapse) this evening

But regarding the original topic, i think this could be marked as solved later.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: wp on November 29, 2022, 01:24:10 pm

Quote from: d7_2_laz on November 29, 2022, 12:56:57 pm

Quote
What's strange is that deletion of the child nodes in the C:\Windows\WinSxS subtree takes much longer than adding them...
(with option 1, previous behaviour, for to assure)
Yes, i saw that yesterday evening too. What i find strange: when retrying it now again, there is no delay at collopse (option 1). Sometimes yes, sometimes no? Hmm
Will try some more retests about that (especially collapse) this evening

I think there is some caching on the OS side...

When I repeat expanding and collapsing of WinSxS several times I usually (but not always) observe that
- option 1 expansion is fast at the first time, but slow subsequently
- option 2 is always fast in both cases
- option 3 is always slow in collapasing.
(felt times, not the times displayed).

I 'd interpret the difference between option 1 and 3 such that expansion by itself is always fast, but deletion is slow since it is involved in both cases (Node.DeleteChildren). In option 1, the expanded subtree is deleted before it is expanded again - therefore expansion is slow from the second time on. In option 3, the subtree is always deleted, therefore, it is always slow in collapsing.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 29, 2022, 10:33:11 pm

Tested again especially with regard to the “delay at collapse”, option 2 and 3

- Option 2 (keep childs, not clear them at expand)

Didn't encounter any(more) delay at collapse. Having tried various scenarios and different action sequences.

- Option 3 (clear collapsing nodes):

Yes,.the discrepancy between felt time for expand (short !) vs. collapse (long !) appears strange.
This differs surprisingly from the measured real times. Eg: for expand: 0,106s, for collapse 0,049s, which is faster.

This small number measured within "TForm1.ShellTreeView1Collapsed" would look like as if the processing itself is fast, but its gui display simply is deferred by some reasons!

On the other hand, this procedure (ShellTreeView1Collapsed) is really reached very late (seen by setting a breakpoint herein).
If we look in ShellCtrls TCustomShellTreeView.Collapse, apparently there's a lot of time spent for Node.DeleteChildren.
Which appears to me to take much longer as when being called from within option 1. May this be? Mysterious so far.
A next step could be to measure the runtime Node.DeleteChildren takes when being called from within option 1 vs. when from option 3. Do the numbers match? And do they match somehow the 0,049s as reported from the main form?

-------------

Quote

In option 1, the expanded subtree is deleted before it is expanded again - therefore expansion is slow from the second time on

Absolutely. Same opimion. Option 1; in a first expand, there's nothing to delete. In subsequent expands, there might be masses to delete. So they are longer by principle. This explains the question asked in the title of this topic.

Title: Re: Q: ShellTreeView Node Expand: why second expand is slower than the first one?
Post by: d7_2_laz on November 30, 2022, 12:00:27 pm

Here some numbers about "DeleteChildren" just for info.
The numbers are measured with GetTickCount and imo much more near to the real visual experience.

Option 1, seccond run: "DeleteChildren" takes: // the first open won't have anything to delete
1,282 ms

Option 3: "DeleteChildren" takes:
2,500 ms
zhr second and subsequent runs: 1,281 ms

Does mean: "DeleteChildren" takes (for "WinSxS") when using for option 1 vs. for option 3 each approx. 1,3 secs.
But for option 3, when the node is collapsed the first time, that will take twice as long.

Remark: personally for me option 3 is out of interest (option 2 would be my favorite).
But i could very well imagine a good use case for it, eg. a treeview action "Collapse all" resp. "Compact tree",
where under the hood the whole memory of opened node is freed.

Code: Pascal [Select][+]

function TCustomShellTreeView.CanExpand(Node: TTreeNode): Boolean;
.........
      ecmRefreshedExpanding:
        begin
 Tick := GetTickCount;
          Node.DeleteChildren;
 Tick := GetTickCount - Tick;
 Outputdebugstring(pchar('CanExpand delete children took:  ' + Node.Text + '   ' + Inttostr(tick) + ' ms' ));
 ...
 
procedure TCustomShellTreeView.Collapse(Node: TTreeNode);
.........
  if ExpandCollapseMode = ecmCollapseAndClear then
  begin
Tick := GetTickCount;
    BeginUpdate;
    try
      hadChildren := Node.HasChildren;
      Node.DeleteChildren;
      Node.HasChildren := hadChildren;
    finally
      EndUpdate;
Tick := GetTickCount - Tick;
Outputdebugstring(pchar('Collapse delete children took:  ' + Node.Text + '   ' + Inttostr(tick) + ' ms' ));
 ...

Lazarus

Programming => LCL => Topic started by: d7_2_laz on November 27, 2022, 05:04:31 pm