Recent

Author Topic: Question: Handling large image files  (Read 2403 times)

Timur Born

  • New Member
  • *
  • Posts: 16
Re: Question: Handling large image files
« Reply #15 on: December 22, 2020, 01:30:44 am »
It is possible to point directly to a library using the external keyword. That's the simple but I guess you can do that only if you are sure the library will be available.
Thanks for the hints and extra information. Via its installer Nomacs installs OpenCV as as dll files of about 16 mb and 28 mb respective file-size. That's okish, but the whole of FastStone IV (likely Delphi) fits into the 16 mb size, including language files and plugins. FastStone launches a bit quicker to draw the window frame, but it's not faster until you reach the point where you can really use it. Long launch times is one thing that hampers ACDSee (and its Quick View module is limited is functions).

Quote
Otherwise it is also possible to statically link an object file (I never tried it personally). Here is the doc on external:
Is there an advantage to doing this?

Quote
You can have more control over the dependency to the library. You need for this code that loads the library and retrieve the pointers to the procedures.
...
On the other hand, you can choose when to load/unload the library, do something else if the library is not present, etc.
Is the load/unload the main advantage? It could be used to unload libraries like LibRaw when they are not needed (no raw files to be processed).

circular

  • Hero Member
  • *****
  • Posts: 3668
    • Personal webpage
Re: Question: Handling large image files
« Reply #16 on: December 22, 2020, 09:29:00 am »
I don't think there is any advantage in static linking except that it produces just one binary file. It may make the startup time a bit longer if you include big libraries.

Quote
Is the load/unload the main advantage? It could be used to unload libraries like LibRaw when they are not needed (no raw files to be processed).
There is the load/unload advantage. This can reduce the startup time and memory usage as you may not need to load all libraries. Another advantage is that you can look for the library file in the hard drive and load the one you prefer. I use that on Linux to get the best fit:
https://github.com/bgrabitmap/bgrabitmap/blob/master/bgrabitmap/linuxlib.pas
Conscience is the debugger of the mind

avra

  • Hero Member
  • *****
  • Posts: 2112
    • Additional info
Re: Question: Handling large image files
« Reply #17 on: December 22, 2020, 10:48:01 am »
...you can look for the library file in the hard drive and load the one you prefer. I use that on Linux to get the best fit:
https://github.com/bgrabitmap/bgrabitmap/blob/master/bgrabitmap/linuxlib.pas
Nice  8)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Timur Born

  • New Member
  • *
  • Posts: 16
Re: Question: Handling large image files
« Reply #18 on: December 22, 2020, 11:10:44 am »
Dummy question: Is the whole library compiled into the exe when it is statically linked, I thought that only the parts being used are?

I am not against dynamically linking, but I recently installed all MSVC redistributables via a single installer and it installed 24 different versions. Even with that I am sure one of the next installers is going to ask me to install some of it again.  >:D

The load/unload parts sure has its charm, depending on how taxing loads/unloads are during runtime.

One of the main reasons why you see me writing here instead of just enjoying user lala land is that I am really getting tired of all the unresponsiveness of desktop apps on expensive modern hardware. I keep throwing money at the problem, but still far too much time is spent on twiddling thumbs (watching animated circles/balls/"not responding") while resources are either unused or abused.

And then there is serious feature lack after decades of UI computing. Nomacs is fast, but you need to load multiple instances and half-manually sync them in order to compare images. And we get 3D in PDF, but there still is not a single Windows based PDF viewer that offers proper sorting of text search results (think files with hundreds/thousands of pages).

circular

  • Hero Member
  • *****
  • Posts: 3668
    • Personal webpage
Re: Question: Handling large image files
« Reply #19 on: December 22, 2020, 11:50:46 am »
Dummy question: Is the whole library compiled into the exe when it is statically linked, I thought that only the parts being used are?
I suppose if the linker can sort things out.

I am not against dynamically linking, but I recently installed all MSVC redistributables via a single installer and it installed 24 different versions. Even with that I am sure one of the next installers is going to ask me to install some of it again.  >:D
Yes, relying on installed things on Windows is not great either in my experience. I prefer to supply the Dll next to the Exe if I can.

Quote
The load/unload parts sure has its charm, depending on how taxing loads/unloads are during runtime.
The taxing happens anyway. Either on startup, when loading the image, or maybe when the application is idle.

Quote
One of the main reasons why you see me writing here instead of just enjoying user lala land is that I am really getting tired of all the unresponsiveness of desktop apps on expensive modern hardware. I keep throwing money at the problem, but still far too much time is spent on twiddling thumbs (watching animated circles/balls/"not responding") while resources are either unused or abused.
I am sure we are many here to agree on that. It would be great that companies spend more time optimizing things rather than make the brand new thing.
Conscience is the debugger of the mind

avra

  • Hero Member
  • *****
  • Posts: 2112
    • Additional info
Re: Question: Handling large image files
« Reply #20 on: December 22, 2020, 02:49:20 pm »
There is an interesting discussion how to speed up TIFF loading:
http://maptools-org.996276.n3.nabble.com/Fast-TIFF-Reading-on-Windows-td13424.html
Talk about using memory mapped files, adapting to file system block size, thread safe list of strips with each thread getting the next strip and reading it...

Quote
It is worth mentioning that the QT based Nomacs viewer is very fast at viewing these large files, maybe even faster than anything I tried yet. Even only using a single-thread opens the displays the 3 gb TIFF file within 3 seconds
Are you sure that whole 3GB of data are loaded in 3 seconds? That would render to 1GB per second which is very rare even with fastest M.2 SSDs. I would guess that for fast preview many tiff strips were skipped. For example, your preview window is 800x600 and your tiff image is 8000x6000. From that info and from info about tiff strips height in pixels, and from knowing the number of cpu cores available in your system, you can build a list of strips for multithreaded strips reading by prioritization of loading each n-th stip first to populate your preview window first, and then loading the rest of the strips when user thinks that all of it is already loaded. I am not aware of tiff image format details, so maybe you can also optimize loading of a single strip, by not loading all strip data at once.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 2112
    • Additional info
Re: Question: Handling large image files
« Reply #21 on: December 22, 2020, 03:07:11 pm »
I have found libtiff, libjpeg and zlib for Delphi with all headers already translated to pascal and all debug and release libraries 32bit *.obj files for windows:
https://web.archive.org/web/20070202123132/http://www.awaresystems.be/imaging/tiff/delphi/LibTiffDelphi_Full.zip

Might help you.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

circular

  • Hero Member
  • *****
  • Posts: 3668
    • Personal webpage
Re: Question: Handling large image files
« Reply #22 on: December 22, 2020, 05:35:03 pm »
Are you sure that whole 3GB of data are loaded in 3 seconds? That would render to 1GB per second which is very rare even with fastest M.2 SSDs. I would guess that for fast preview many tiff strips were skipped. For example, your preview window is 800x600 and your tiff image is 8000x6000. From that info and from info about tiff strips height in pixels, and from knowing the number of cpu cores available in your system, you can build a list of strips for multithreaded strips reading by prioritization of loading each n-th stip first to populate your preview window first, and then loading the rest of the strips when user thinks that all of it is already loaded. I am not aware of tiff image format details, so maybe you can also optimize loading of a single strip, by not loading all strip data at once.
Indeed, the TIFF structure may help to do that. It depends how it was saved. The file can organized as strips or as tiles and if that's the case, you can for example extract only the tiles that are being displayed (zoom) or skip strips (unzoomed).
Conscience is the debugger of the mind

Timur Born

  • New Member
  • *
  • Posts: 16
Re: Question: Handling large image files
« Reply #23 on: December 22, 2020, 06:11:38 pm »
My M.2 does over 2 gb/s on sequential reads of large blocks and once its in cache any waiting time remaining is processing time by the application opening the file. It's still possible that the file is opened in stripes, but if that helps faster loading then the better.

Thanks for the link! As far as I understand the 32-bit .obj files cannot be used in 64-bit applications (and would thus not be able to handle 3 gb files)?

« Last Edit: December 22, 2020, 06:17:45 pm by Timur Born »

circular

  • Hero Member
  • *****
  • Posts: 3668
    • Personal webpage
Re: Question: Handling large image files
« Reply #24 on: December 22, 2020, 06:29:56 pm »
Indeed you need object files with the same bit-ness
Conscience is the debugger of the mind

Timur Born

  • New Member
  • *
  • Posts: 16
Re: Question: Handling large image files
« Reply #25 on: December 22, 2020, 06:34:04 pm »
From that info and from info about tiff strips height in pixels, and from knowing the number of cpu cores available in your system, you can build a list of strips for multithreaded strips reading by prioritization of loading each n-th stip first to populate your preview window first, and then loading the rest of the strips when user thinks that all of it is already loaded. I am not aware of tiff image format details, so maybe you can also optimize loading of a single strip, by not loading all strip data at once.
Worth mentioning that I have not seen any image viewer load any image type multi-threaded (per single file). I may have seen 2 thread for TIFF once, but it's possible that one thread was just busy with the GUI (despite nothing but image loading happening). Nomacs at least uses multiple thread for thumbnail creation (one thread per file/thumbnail that is).

avra

  • Hero Member
  • *****
  • Posts: 2112
    • Additional info
Re: Question: Handling large image files
« Reply #26 on: December 23, 2020, 12:02:29 am »
As far as I understand the 32-bit .obj files cannot be used in 64-bit applications (and would thus not be able to handle 3 gb files)?
Right, but that are libtiff object files and header translation are from 2007. Nothing stops you to make new libtiff object files for both 32-bit and 64-bit. Remember that 3 GB limit is just a speculation because of 32-bit limits and assumptions that whole file is loaded into memory at once. It may be something else. Clever 32-bit application should be able to handle almost unlimited image size at cost of speed.

Worth mentioning that I have not seen any image viewer load any image type multi-threaded (per single file). I may have seen 2 thread for TIFF once, but it's possible that one thread was just busy with the GUI (despite nothing but image loading happening). Nomacs at least uses multiple thread for thumbnail creation (one thread per file/thumbnail that is).
Well, link I provided discusses possible speed improvements for tiff reading. One of them is multithreaded strips reading. It's up to you if you want to try it or not.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Timur Born

  • New Member
  • *
  • Posts: 16
Re: Question: Handling large image files
« Reply #27 on: December 23, 2020, 11:07:25 am »
Thanks again.  ;)

At first I thought that XnViewMP and IrfanView may be among those few viewers that do not try to load the whole image into memory, because they only consume 2.2 gb after decoding the 3 gb testfile. Turns out that my 3 gb testfile really only should be 2.2 gb of data, I will have to check that again. I currently use XnViewMP when I need to do image comparison in images too large for FastStone. It's good enough once images are loaded, but its browser view is slow when it reads in those large files specs using only a single thread, even without thumbnails being viewed.

Overall I think that reading the whole file(s) into memory is preferred for smooth operation without lags and judder. All software I tried seems to handle large images that way (with 32-bit ones failing once size is too large).

Concerning multi-threaded decoding: In the past I thought that single-threaded decoding might be a bottleneck even for uncompressed files, but 3 seconds for a 3 gb file are fast enough. Handling of (de)compression would likely benefit from multi-threading, at least for Deflate/ZIP, but maybe not so much for LZW.
« Last Edit: December 23, 2020, 11:36:19 am by Timur Born »

avra

  • Hero Member
  • *****
  • Posts: 2112
    • Additional info
Re: Question: Handling large image files
« Reply #28 on: December 23, 2020, 11:36:11 am »
Reading the whole file(s) into memory is absolutely preferred for smooth operation without lags and judder.
I was not saying that you should not load the whole file into memory. I was saying that you could first read every n-th strip (depending on preview window size) to show super fast preview to the user, and then read all the rest of the strips. If you wish you can also use multithreading (if I remember the linked discussion well - some developers said that it does speed up loading), but it is not a must. This is just a theoretical discussion. Final decision is fully yours.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Timur Born

  • New Member
  • *
  • Posts: 16
Re: Question: Handling large image files
« Reply #29 on: December 23, 2020, 11:37:11 am »
Yes, this sounds like a good idea, similar to how progressive JPEGs are decoded. Thanks for the clarification. English is not my native language, so sometimes I misunderstand stuff.

On fast systems as mine it might not make much of a difference, though, because 3 seconds. Decompression of compressed images might gain more from multi-threading. Opening the same large image in a compressed format (like Deflate TIFF) takes much longer on my PC.
« Last Edit: December 23, 2020, 11:40:18 am by Timur Born »

 

TinyPortal © 2005-2018