MPHexEditor, large files scrolling

[DIMM_bl4]

Hello to all, my first message here.
Have a some trouble with MPHexEditor module, when open large files scrolling became very slow. This is slightly noticeable from 16-32MB size and go worst with increasing size.
First of all, I go to example project, and see exactly the same.
Lazarus is 2.0.10, FPC 3.2.0, MPHexEditor from here https://github.com/michalgw/mphexeditor

I tried to solve by myself, but looks like I'm not strong enough )
Found that things:
There are used two ways for scrolling
1. Scroll with keyboard/mousewheel do with changing of 'TopRow' property, and this way is slow.
2. Scroll with side scrollbar is totally differrent and uses 'WMHScroll(var message: TLMHScroll)' and then 'TrySmoothScrollBy(Col, Row)' from module Grids. This way work always fast, with any file size.

Would appreciate any help, or thoughts )

[korba812]

This is a known issue. The problem is with TCustomGrid, from which TMPHexEdit inherits. I have attached a sample project to illustrate the problem. Other problems with using TCustomGrid are excessive memory consumption and row limit.
This is the main reason why I stopped further development of TMPHexEdit. It should probably be redesigned from scratch.

[DIMM_bl4]

Thanks for reply.
I made a 'temporary' solution - modified procedures KeyDown, DoMouseWheelDown, DoMouseWheelUp from MPHexEditor.pas to also use TrySmoothScrollBy(aCol, aRow) from module Grids (as a scrollbar scrolling).
But for this I had to also modify module Grids.pas, for possibility to call TrySmoothScrollBy from outside. I moved it from 'private' declaration to 'protected', same as for some procedures, called from outside.
I'm not 100% sure about any side effects, but looks like working good.
I attach modified example .exe from MPHexEditor, now it can scrolls up to ~512MBytes files, original example freezed to death (with 100% load to one thread) with constant scrolling at about 32Mbytes files.
https://yadi.sk/d/7LU_sUxo1Zvk2Q

The problem is with TCustomGrid, from which TMPHexEdit inherits ...

I can not totally agree, because I also tryed to use 'parent' versions of procedures DoMouseWheelDown, DoMouseWheelUp, and scrolling became fast (no freeze)...but at one line per wheel's 'tick'.

[korba812]

IMHO, a better solution will be to expand functionality of TCustomGrid component. The TCustomGrid.UpdateCachedSizes method is called quite often. This function sums height of all rows, so it takes a while for a large number of rows. In many cases, all rows have the same height and there is no point in storing it. I would suggest adding a mode where all rows (or at least non-fixed) have the same height and not store their height values. This will eliminate the problem with slowdown and high memory consumption.

[Mastropiero]

Hello,

first of all, sorry for the thread necromancy.

I have recently faced this issue. It seems that the problem has been inherited from TCustomGrid. Is there any chance of having someday a new version of this component that is not based on TCustomGrid so the problem does not exist anymore?

Could the MPHexEditor component be redesigned so it only handles the rows that are actually shown on the screen instead of loading the whole file into RAM?

[korba812]

I have recently faced this issue. It seems that the problem has been inherited from TCustomGrid. Is there any chance of having someday a new version of this component that is not based on TCustomGrid so the problem does not exist anymore?

As I wrote, it requires a lot of work. Unfortunately, this is not in my plans. But maybe someone will?

Could the MPHexEditor component be redesigned so it only handles the rows that are actually shown on the screen instead of loading the whole file into RAM?

Theoretically yes - MPHexEdit uses TMemoryStream internally and theoretically could handle any stream. But it won't be easy, e.g. inserting or deleting data at the beginning of the file.

[Mastropiero]

Thank you for your comments.

I have just wrote a small proof-of-concept of buffered reading that uses only 3MB of RAM and it keeps constant and fast even when scrolling >5GB sized files.  I use a simple TMemo to print the HEX data in read-only mode but I think the concept could be applied to any descendant of TCustomGrid.

This would solve the speed and memory consumption problem, but the own component's scrollbars wouldn't work anymore. I think this can be easily solved by disabling scrolling and adding an external TScrollBar component synced with the grid.

However, I'm wondering how to handle data selection, tracking of modified bytes, undo operations, etc. with this approach.

Tons of analysis and work to do...

[korba812]

There is a component that can display (read only) huge files (> 2GB) in hex:
https://github.com/Alexey-T/ATBinHex-Lazarus
You can see how Alexey handled scroll bars in his component.

CNPack has a hex editor component which is based on TCustomControl:
https://github.com/cnpack/cnvcl/blob/master/Source/Graphics/CnHexEditor.pas

[AlexTP]

If the problem is inside LCL Grids code, can you take the Grids file, rename it + rename its classes, and include changed file into MPHexEditor?

[Mastropiero]

Theoretically yes - MPHexEdit uses TMemoryStream internally and theoretically could handle any stream. But it won't be easy, e.g. inserting or deleting data at the beginning of the file.

WinHex (the best hex editor for Windows) does insert operations directly on a RAM buffer. If the file does not fit the buffer, it creates a temporary file. It also offers an in-place edit mode for big objects.

[zeljko]

IMO, you should try TVirtualStringTree instead of TCustomGrid or any Grids derivative.

[Mastropiero]

IMO, you should try TVirtualStringTree instead of TCustomGrid or any Grids derivative.

Do you mean Grids is THAT bad?  Is there any plans to rewrite the Grids unit or something? TCustomGrid is probably one of the most used classes in the original Delphi and it can't understand why it isn't optimized on Lazarus.

By the way, I have never worked with the TVirtualStringTree but I'll give it a chance. Looks promising.

[zeljko]

Grids is ok for it's purpose, but it's not suitable for big data scenario. I'm using grids in my projects too, but for huge data I'm using TVirtualStringTree (eg 1 million items).

[korba812]

TVirtualStringTree does not solve problem of large files as it can only handle 2^32 rows, so maximum file size is 4GB x 16 bytes per line = ~68GB. It's not much nowadays.

We need something that has a row count of (at least) Int64 type.

[zeljko]

In that case you should implement your own control (if such beast does not exist already)