Recent

Author Topic: suggestions on how do handle a large array of boolean  (Read 1259 times)

mas steindorff

  • Hero Member
  • *****
  • Posts: 532
suggestions on how do handle a large array of boolean
« on: April 25, 2021, 01:39:49 am »
Hi all,
let's say I'm transferring a "file" from one machine to my local drive and I will be getting this file in blocks of a fixed size (1K each).  The transfer is more reliable if I request 1 block at a time but works much fastest but problematic if I request 5 to 10 blocks at a time.  I need to keep track of the good blocks and the ones that I need to request again.
FYI: A "file" can have up to 900,000 blocks that need to be transfer over. my plan is to make my 1st pass ASAP using the problematic transfer and then go back and get the ones that failed using the slower/more reliable transfer method.

my thinking is to setup a list that can grow to match the blocks of "file" size using
specialize TFPGList<boolean>
that I can then use as a todo list. 

Is this the best approach or is there a better (memory saving) object I should look into?

Mas
Thank you
windows 10 &11, Ubuntu 21+ - fpc 3.0.4, IDE 2.0 general releases

mas steindorff

  • Hero Member
  • *****
  • Posts: 532
Re: suggestions on how do handle a large array of boolean
« Reply #1 on: April 25, 2021, 06:32:13 am »
<lost last response>
the Tbits along with ASerge's TExBits extension is just the thing I was tiring to reinvent! it comes with all of the search functions I was trying to build.
thanks Jamie!

https://forum.lazarus.freepascal.org/index.php/topic,45770.msg324280.html#msg324280

and thanks ASerge
windows 10 &11, Ubuntu 21+ - fpc 3.0.4, IDE 2.0 general releases

egsuh

  • Hero Member
  • *****
  • Posts: 1273
Re: suggestions on how do handle a large array of boolean
« Reply #2 on: April 25, 2021, 02:59:09 pm »
What I write here is pure question.  AFAIK, even though read and write is done bytes by bytes, the physical operation is optimized by operating system using its own buffers. Is it really necessary to optimize more in many cases, and the improvements deserve such efforts? Of course there would be such cases. But my guess is that they are very rare. As I told at the first, this is pure question because I just don't know.

egsuh

  • Hero Member
  • *****
  • Posts: 1273
Re: suggestions on how do handle a large array of boolean
« Reply #3 on: April 26, 2021, 03:16:55 am »
Quote
I have no idea where the operating system comes into play here?
I'm talking about file transfer.

I thought file transfer only between PCs, in which case I/O speed may not be much different if done through OS --- this is my question. My guess is there is not large difference between reading 5 blocks at once and one by one unless the block size is coincide with default I/O buffer size of OS --- pure question.

But now I came to think about other devices, not PCs, in which case the programmer may have to take care of I/O directly.

 

TinyPortal © 2005-2018