MPHexEditor, large files scrolling

[Mastropiero]

TVirtualStringTree does not solve problem of large files as it can only handle 2^32 rows, so maximum file size is 4GB x 16 bytes per line = ~68GB. It's not much nowadays.

We need something that has a row count of (at least) Int64 type.

I think there is something wrong in that approach. Most modern hex editors do not manage zillions of rows. They only store in the control a few rows (the ones being shown on screen and maybe a few more for buffering and fast scrolling) so the control only shows a small portion of the file. The rows are dynamically loaded into the control as the user navigates the whole file. This approach is very fast and uses a very small memory footprint. Size-modifying operations (insert, delete, concatenation...) are solved by using a temporary buffer (or a temporary file for huge amounts of data).

EDIT: but, of course, I think that a modern Pascal compiler should use primarily 64bit integers when possible. so maybe many of the classes/controls should be updated.

[korba812]

TVirtualStringTree does not solve problem of large files as it can only handle 2^32 rows, so maximum file size is 4GB x 16 bytes per line = ~68GB. It's not much nowadays.

We need something that has a row count of (at least) Int64 type.

I think there is something wrong in that approach. Most modern hex editors do not manage zillions of rows. They only store in the control a few rows (the ones being shown on screen and maybe a few more for buffering and fast scrolling) so the control only shows a small portion of the file. The rows are dynamically loaded into the control as the user navigates the whole file. This approach is very fast and uses a very small memory footprint. Size-modifying operations (insert, delete, concatenation...) are solved by using a temporary buffer (or a temporary file for huge amounts of data).

TCustomGrid and VirtualStringTree run "virtually" - they don't store any data. They only store basic information, e.g. total number of rows, current cell number, first and last visible cell, etc... Incase of TCustomGrid, user has to implement appropriate methods to handle data or draw a cell. As I said before, TMPHexEditor stores data in TMemoryStream.

The problem with TCustomGrid is that it stores height of each row separately, which is unnecessary in the case of TMPHexEditor. This can be easily fixed by modifying TCustomGrid code as Alex suggested.
A bigger problem is the ability to handle only MaxInteger row count. And that won't be easy to change in TCustomGrid.

EDIT: but, of course, I think that a modern Pascal compiler should use primarily 64bit integers when possible. so maybe many of the classes/controls should be updated.

I don't think it is necessary. The case of hex editor is special. Usually, no such scope is needed.

[Mastropiero]

TCustomGrid and VirtualStringTree run "virtually" - they don't store any data. They only store basic information, e.g. total number of rows, current cell number, first and last visible cell, etc... Incase of TCustomGrid, user has to implement appropriate methods to handle data or draw a cell. As I said before, TMPHexEditor stores data in TMemoryStream.

The problem with TCustomGrid is that it stores height of each row separately, which is unnecessary in the case of TMPHexEditor. This can be easily fixed by modifying TCustomGrid code as Alex suggested.

So, if I used a TCustomGrid with only 64 rows (the visible ones), that wouldn't be a problem anymore. Isn't it?

[korba812]

But scrollbar will only match 64 rows.

[Mastropiero]

But scrollbar will only match 64 rows.

In my previous proof-of-concept (as I described some posts ago) I fixed that by hiding the control's scrollbar and putting a TScrollBar component next to the grid. I did the test with a standard TMemo instead of a TCustomGrid, but it did the trick perfectly.  In any case, not only the scrollbars should be fixed; other features such as data range selection should also have to be redone with a different approach.

So, in my opinion, it is too much work for a complete MPHexEditor reworking, but I still think that a new (made from scratch) Hex component based on TCustomGrid is feasible.

[rant mode on]

Edit: never mind... I have just discovered that the min and max values of TScrollBar are also 32bit, so as a workaround I could use the ScrollBar to store the offset at the row level (so if I edit 16 columns, I can scroll files up to 2^32 * 16 bytes). The limit in that case would be 64GB. Still 'small' nowadays.

So, at this point, I see that most components in Lazarus have severe limitations to be used in a modern system (and for modern purposes). Definitely, if Lazarus/Free Pascal have become a niche language is for a reason.

Lazarus is not a RAD environment anymore if you have to build from scratch even a simple Scroll Bar component. Of course, in the case of the scroll bar there are third-parties alternatives such as ATControls (that I am in fact using for tabs, and I think it's a brilliant suite of components), but in my opinion, the stock classes should be usable without having to use external packages.

[rant mode off]