ShellTreeView and system icons

[wp]

The link mentioned by Ally (above) gives a good starting point. As well as that one:
https://forum.lazarus.freepascal.org/index.php/topic,35669.msg236441.html#msg236441
Here, see routine TShellListView1FileAdded  (although it is not needed and i would not do a call of SHGetFileInfoW for each file added; better do it only once for a folder).

Why don't you use Getmem's demo and merge it with ASerge's code? Did you check that?

I think before talking about the images we should fix other speed-limiting issues. I follow your example. open the Windows folder in the tree and click on the System32 folder in the tree to open System32 in the ListView. System32 has more than 4000 files in my system, population takes about 20 seconds (the Windows explorer opens it immediately). But after clicking I see the scrollbar running up, and this is an indication of a massive speed killer: redrawing of the listview after each addition! The standard way to avoid this is to surround the population code by an Items.BeginUpdate and Items.EndUpdate pair. So, I open shellctrls, and there's a PopulateWithRoot method. Putting an "Items.BeginUpdate" outside the "try" block and an "Items.EndUpdate" into the "finally" part should prevent that massive screen repainting. And in fact the population time goes down to about 4-5 seconds.

With the Treeview, there is a similar population routine, PopulateTreeWithFiles where the Begin/EndUpdate pair is missing as well. The same with PopulateWithBase. I wrote a quick-and-dirty program which generates 5000 folders. Opening the parent folder in the tree took 5 seconds in the original ShellViews unit, but was immediate after adding the Begin/EndUpdate.

Please make these modifications on your system and verify my observations. Then I can add them to Lazarus easily.

The speed difference between TreeView and ListView indicates that there may be something else to slow down the listview. I unhooked the OnFileAdded handler which creates the images - now the System32 folder populates immediately. I exited after the SHgetFileInfoW call -- 5 seconds again. So the following code for copying the image to the imagelist is negligible!

[ASerge]

ASerge, did you mean TListView  is entirely built by LCL?  if it is able to allow Listview_SetImagelist, the more it would be desirable that the treeview allows such similar.

Actually, I said that only TreeView renders the icons itself, but ListView uses the system. Therefore, it is useless for TreeView to set a non-LCL ImageList. But for ListView, it works.

[balazsszekely]

@wp

Thanks a lot. Very helpful. Where to put it? wiki? CCR/applications? Lazarus/examples/?

You're welcome. I don't have any preference, it's up to you.

@d7_2_laz
I attach a very fast solution(similar to windows explorer). After you build the application, make sure you run it outside the IDE, because the scrolling can be slow in debug mode. It looks like a TShellListView, but much faster. A TShellTreeView can also be implemented in the future.

[d7_2_laz]

ASerge: thanks for clarification, that had been my original assumption, but now i am clear that's useless to think any longer about treeview und set_imagelist.

wp:  you did ask about the merge. I'm late today and hopefully my  _attachment_  is not so far behind your recent thoughts.  It's the code sample from getmen with your adaption, and for the listview the adaption which uses listview_setimagelist. Note that a separate imagelist must not be attached to the listview at the same time, Regarding the treeview nothing had been changed. ASerge gave the idea (use ownerdraw for the icons)  but i had not time yet to implement that.
About the performance, i need to clarify that i'm not using the shell controls but, file system oriented, Searchrec, custom treeview and for the listview, ownerdatafetch (OnData, virtual approach). So myself i have no performance problems und simply the need to fetch system icons for the treeview correctly according to the Lazarus concepts, and hopefully not slower than before.
Regarding the shell controls:  we did speak about them because that icon stuff mostly is discussed around thiese controls, but using treeview or shell treeview doesn't make any difference in this respect.

About the BeginUpdate / EndUpdate (resp. LockWindowUpdate  & co) - yes absolutely, if this is not implemented in the shell controls yet, it should urgently, because this does a big deal. Actually i need to focus onto the most hindering issues forrced by component incompatibilities; next will be PageControl (replacement needed) and watch threads (argh).

getmem: many thanks for the sample! Will this trunk needed be part of the next release?  I think i can see the idea behind it i(it is based on VirtualTreeview control, and so i believe by my own usage of this control  that it will be really fast enough).
Abut the icon access within: same question i had but cannot answer fo rmyself: will the continuous usage of Imagelist IconAdd, when iterating maybe the same directory multiple times, let the imagelist grow and grow and grow??
   Data := VDT.GetNodeData(PaintInfo.Node);
   .....
    FileIcon.Handle := FileInfo.hIcon;
    Data^.FImageIndex := ilShell.AddIcon(FileIcon);
    Data^.FImageIndex := ImageList_ReplaceIcon(ilShell.ResolutionByIndex[0].Reference.Handle, Data^.FImageIndex, FileIcon.Handle);
    ImageList_Draw(ilShell.ResolutionByIndex[0].Reference.Handle, Data^.FImageIndex, PaintInfo.Canvas.Handle, R.Left + 2, R.Top + (R.Height - FileIcon.Height) div 2, ILD_TRANSPARENT);

Btw: you have a good forum here! As a newbie i'm impressed about the level of discussion and contributions.

[wp]

GetMem's solution with VirtualTreeView is amazingly fast. Why?

People usally say: "VirtualTreeView is much faster than TTreeView and TListView". But I think 4000 nodes is not so much that this would be visible.

Let's have a closer look at the OnFileAdded method of the D7_2Laz's version with TShellListView:

Code: Pascal  [Select][+][-]

1. procedure TfMain.SLVFileAdded(Sender: TObject; Item: TListItem);
2. var
3.   FullName: WideString;
4.   FileInfo: TSHFileInfoW;
5.   SysImageHandle: DWORD_PTR;
6.   ListHandle: HWND;
7. begin
8.   FullName := WideString(SLV.Root + Item.Caption);
9.   // <----- Test #1: add "exit" here
10.   SysImageHandle := SHGetFileInfoW(PWideChar(FullName), 0, FileInfo, SizeOf(FileInfo), SHGFI_SMALLICON or SHGFI_SYSICONINDEX);
11.   // <---- Test #2: add "exit" here
12.   if SysImageHandle = 0 then
13.      Exit;
14.   ListHandle := SLV.Handle;
15.   if ListView_GetImageList(ListHandle, LVSIL_SMALL) = 0 then
16.   begin
17.     SetWindowLong(ListHandle, GWL_STYLE,
18.                   GetWindowLong(ListHandle, GWL_STYLE) or LVS_SHAREIMAGELISTS);
19.     ListView_SetImageList(ListHandle, SysImageHandle, LVSIL_SMALL);
20.   end;
21.   Item.ImageIndex := FileInfo.iIcon;
22. end;
23.  

Populating c:\Windows\System32 takes about 5 seconds on my system. Let's find the speed-limiting step in this code. In a first test add an "exit" before SHGetFileInfoW -- now the listview is populated immediately. In a second step add an "exit" AFTER SHGetFileInfoW -- population takes 5 seconds again.

This observation confused me a lot because GetMem's VTV solution has exactly the same function calls. So why is it so fast?

The essential difference is: the ShellListView code has these calls in the OnFileAdded event; this is called for every file stored in the listview. VTV, however, has these calls in the OnDrawNode event handler, and this is called only for the visible files -- some tens compared to some thousands. That's the reason why the VTV Solution is so fast.

And I guess that if the TShellListView had an OnCustomDraw event which would draw only those icons needed it would be much faster. But hey, TShellListView inherits from TCustomListview, and this one does have several protected OnCustomDraw* events - so TShellListView can use them too once they are published or made accessible by a typecast to a class derived from TShellListView.

It's too late for me to demonstrate this today, but maybe tomorrow.

[balazsszekely]

@d7_2_laz

many thanks for the sample!

You're welcome!

Will this trunk needed be part of the next release?

The next release(2.0.12) is a bugfix release. After that, what is now trunk will become Lazarus 2.2. Anyways in my previous post I attached two demo versions, one for 2.0.10 and one for trunk. Unfortunately the VirtualTreeView shipped with 2.0.10, it's slightly old and buggy. The fix is easy, just install "VirtualTreeView V5" with OPM(Menu->Package->Online Package Manager).
So both attached demo versions should be usable.

I think i can see the idea behind it i(it is based on VirtualTreeview control, and so i believe by my own usage of this control  that it will be really fast enough).

Yes. I believe it will be fast enough, but please run a few test.

Abut the icon access within: same question i had but cannot answer fo rmyself: will the continuous usage of Imagelist IconAdd, when iterating maybe the same directory multiple times, let the imagelist grow and grow and grow??

Each icon is loaded only once. When you do a lot of drawing(scrolling the tree for example), the icons are re-used. The imagelist won't grow, it's cleared before the tree is populated.



@wp
Your assessment is spot on, the bottleneck is SHGetFileInfoW api. The fact that VirtualTreeView is only drawing the currently visible items helps a lot. I intentionally choose VirtualDrawTree(not VirtualStringTree) because you can fine tunning the drawing process. According to the documentation SHGetFileInfoW can be speed up by passing the SHGFI_ATTR_SPECIFIED flag, but then the folder icons must be retrieved from somewhere else. Unfortunately I cannot allocate more time to implement this idea.


PS: I just realised that both in my demos I made a typo, instead of SHGetFileInfo I should have written SHGetFileInfoW.

[d7_2_laz]

@GetMen: yes - your ShellView_VDT example is massively impressive ... i would have expected just soemthing like that when using the VirtualTreeView control
@GetMen and @wp:
Drawing only the parts of the visible items is the best one can do, and so - for a "normal" ListView - using ownderdatafetch aka OnData is a Must for expensive operations.
If i only could rewrite most of my app from the scratch (too many dependencies), for the TreeView i would bet on the Virtual* component now. As i don't want to do that, i need to refocus the treeview.
But for anybody who is interested: are you aware of the VirtualShellTools sample explorer app (they wrote it once based on the VST). Might be worthy to inspect or study anyway!

Spot back to the "normal" Treeview":
- i'm suffering not so much with the time spent for SHGetfileInfo[W]. My D7 app is using that too and was fast enought.
- Purely trivial i'll need the system icons, sigh ...
- i'm still not sure about the usage of AddIcon (triggered by OnGetImageIndex or OnData or whereever.

Each icon is loaded only once. When you do a lot of drawing(scrolling the tree for example), the icons are re-used. The imagelist won't grow, it's cleared before the tree is populated.

Could you please explain a little bit where (for to avoid to store the same object again and again) this clear is done when coming back to a previously visited folder again?
Please apologize my question, maybe this is trivial. The answer is important for my further steps.
- Just trying the possibilities of the customdraw (icons) approach. I got stuck on one certain issue (solely treeview lcl-related) and come back to this with a question later this day.

[d7_2_laz]

Ownerdraw treeview icons
Using AdvancedCustomDrawItem, as i already use that and appears to be the right place
Looking onto "regular" nodes (that are not cdsSelected or cdsFocused in State:
1. Simpe question (for to go sure)
Unless i set DefaultDraw to False (which i try to avoid because otherwise i would need to repaint the whole tree eleemts including vertical lines, expand buttons etc.)
the node caption / texts will be default drawed. I have no chance to avoid that. I only can / have to erase the already default drawed caption texts
eg. via FillRect in stage cdPostPaint and to DrawText the caption on an other position later.
Is this assumption right?
2. Problem :-(  :
If we draw the image and shift the subsequent Drawtext on another position:
   txtRect :=  Node.DisplayRect(True);
   inc(txtRect).Left, iconOffset);
   DrawText( ....., txtRect, .... ):
then there will be a conflict between the painted text position and the internal bookkeeping of the node offset eg. for mouse click, hitPosition et.al.
For mouse click, the control will think the caption is on the "old" offsets, and if we click for instance on the righter part of the caption, nothing will happen.
---> Painted offset und internal offset will be no longer in synch.
Any chanse to avoid? Can we tell the node by a single property that selection coordinates will start at pos X, but at pos X plus 20?
Or do we nned to apply a dummy imagekust with a dummy icon, that is default drawed and repainted afterwards, for to preserve the right selection offsets?

[wp]

You seem to make more  progress than me, I am still fighting how to understand that damned over-complicated customdrawing of the ListView. Can you post the code that you have so far?

[d7_2_laz]

wp, yes of course, but it is still a construction zone and the icon is still a pure placeholder
Currently i'm stuck with mouse click will not respect the changed offsets.
The procedure (only this procedure) is attached.

[wp]

Each icon is loaded only once. When you do a lot of drawing(scrolling the tree for example), the icons are re-used. The imagelist won't grow, it's cleared before the tree is populated.

Could you please explain a little bit where (for to avoid to store the same object again and again) this clear is done when coming back to a previously visited folder again?

I think d7_2_laz's argument is valid. I took the demo of reply #12 (which does add icons to the image list) and added the line Caption := SLV.SmallImages.Count.ToString; at the end of the SLVFileAdded event handler -- it simply displays in the form  caption the current count of icons of the ListView's imagelist. When I run the program and open the c:\Windows\System32 folder the icon count increases by about 100 on my system. Then I click some other folder, and the count increases again by some value. But when I return to C:\Windows\System32 the icon count increases by aonther 100 icons. This is a clear proof that the icons are re-added, otherwise the icon count should stay constant when a previously opened folder is revisited.

[wp]

wp, yes of course, but it is still a construction zone and the icon is still a pure placeholder
Currently i'm stuck with mouse click will not respect the changed offsets.
The procedure (only this procedure) is attached.

Sorry, you did say that you work on the treeview but I did not get it... I am struggling with the ListView, and this is even worse.

[EDIT]
Found the reason why it is so difficult for me to owner-draw the listview: I am working with Laz trunk, and the fix in 64204 (https://bugs.freepascal.org/view.php?id=25397) seems to have broken owner-drawing.

[balazsszekely]

@d7_2_laz

Could you please explain a little bit where (for to avoid to store the same object again and again) this clear is done when coming back to a previously visited folder again?

@wp

I think d7_2_laz's argument is valid. I took the demo of reply #12 (which does add icons to the image list) and added the line Caption := SLV.SmallImages.Count.ToString; at the end of the SLVFileAdded event handler -- it simply displays in the form  caption the current count of icons of the ListView's imagelist. When I run the program and open the c:\Windows\System32 folder the icon count increases by about 100 on my system. Then I click some other folder, and the count increases again by some value. But when I return to C:\Windows\System32 the icon count increases by aonther 100 icons. This is a clear proof that the icons are re-added, otherwise the icon count should stay constant when a previously opened folder is revisited.

The ImageList is only cleared in my latest example, the one with VirtualDrawTree(see method PopulateTree). This must be done in the previous examples too, otherwise the size of the ImageList will increase. If you choose to go with custom draw, imagelists might not be needed at all.

[d7_2_laz]

Sorry for the mistake wp, i add the sample for the listview ownerdraw here (two derivats), but i think it is not very meaningful iin our special context and is not scoping icons (which are still retrieved va listview_setimagelist).

Thanks GetMem for the reply, i take a look inside the example you mentioned this afternoon.

[wp]

Now the modified ShellViews demo is working in which ListView icons are no stored in an imagelist any more, but are painted on the fly at runtime. Opening c:\windows\system32 with about 4000 files on my system takes now about 1 second (strangely enough the folder opens instantly on my VM with Win7, other than my Win10 "real" machine).

You will not see this on an original Laz 2.0.10. You must add BeginUpdate/EndUpdate pairs are used whenever the Listview is populated (method SetRoot (around Clear, Items.Clear, PopulateWithRoot), method PopulateWithRoot (range like "Files")). The recent revision of trunk contains these changes already, it also contains the fix of the owner-draw regression.

There are still some minor painting issues (no painting of subitems, issues with painting the selected item).