Recent

Author Topic: suggestions on how do handle a large array of boolean  (Read 699 times)

mas steindorff

  • Sr. Member
  • ****
  • Posts: 455
suggestions on how do handle a large array of boolean
« on: April 25, 2021, 01:39:49 am »
Hi all,
let's say I'm transferring a "file" from one machine to my local drive and I will be getting this file in blocks of a fixed size (1K each).  The transfer is more reliable if I request 1 block at a time but works much fastest but problematic if I request 5 to 10 blocks at a time.  I need to keep track of the good blocks and the ones that I need to request again.
FYI: A "file" can have up to 900,000 blocks that need to be transfer over. my plan is to make my 1st pass ASAP using the problematic transfer and then go back and get the ones that failed using the slower/more reliable transfer method.

my thinking is to setup a list that can grow to match the blocks of "file" size using
specialize TFPGList<boolean>
that I can then use as a todo list. 

Is this the best approach or is there a better (memory saving) object I should look into?

Mas
Thank you
windows 7/10 - laz 2.0 / 1.2.6 general releases

jamie

  • Hero Member
  • *****
  • Posts: 4577
Re: suggestions on how do handle a large array of boolean
« Reply #1 on: April 25, 2021, 02:13:45 am »
I really don't think that is a good idea using a Generic for this, its going to cause memory frag...

You could use the Tbits class because you can get 8 bits or (Booleans) per byte so that saves some memory..

Or I guess you could just code a dynamic array of the number of Blocks the file has divided by 8 to condense it down..
The only true wisdom is knowing you know nothing

mas steindorff

  • Sr. Member
  • ****
  • Posts: 455
Re: suggestions on how do handle a large array of boolean
« Reply #2 on: April 25, 2021, 06:32:13 am »
<lost last response>
the Tbits along with ASerge's TExBits extension is just the thing I was tiring to reinvent! it comes with all of the search functions I was trying to build.
thanks Jamie!

https://forum.lazarus.freepascal.org/index.php/topic,45770.msg324280.html#msg324280

and thanks ASerge
windows 7/10 - laz 2.0 / 1.2.6 general releases

jamie

  • Hero Member
  • *****
  • Posts: 4577
Re: suggestions on how do handle a large array of boolean
« Reply #3 on: April 25, 2021, 02:35:17 pm »
I took a look at that code for the Tbits class and if you get serious in using it and notice a lag in performance then you should comment on it because I noticed it has lots of room for improvements and code condensing to speed it up..

 I made my own type of bit scanner way back and still use that, back in my early Delphi days. I don't know what the code structure looks like for the current Delphi implemented class but I had a good look at the Fpc implementation. My only comments to it is that it needs an overhaul if anyone seriously needs performance. On Top of that it seems to be using a const "Max" something that is not directly related to the class it appears and could one day get changed and thus effect this class a the same time.. Talk about QC.
The only true wisdom is knowing you know nothing

egsuh

  • Hero Member
  • *****
  • Posts: 741
Re: suggestions on how do handle a large array of boolean
« Reply #4 on: April 25, 2021, 02:59:09 pm »
What I write here is pure question.  AFAIK, even though read and write is done bytes by bytes, the physical operation is optimized by operating system using its own buffers. Is it really necessary to optimize more in many cases, and the improvements deserve such efforts? Of course there would be such cases. But my guess is that they are very rare. As I told at the first, this is pure question because I just don't know.

jamie

  • Hero Member
  • *****
  • Posts: 4577
Re: suggestions on how do handle a large array of boolean
« Reply #5 on: April 25, 2021, 05:17:09 pm »
What I write here is pure question.  AFAIK, even though read and write is done bytes by bytes, the physical operation is optimized by operating system using its own buffers. Is it really necessary to optimize more in many cases, and the improvements deserve such efforts? Of course there would be such cases. But my guess is that they are very rare. As I told at the first, this is pure question because I just don't know.

I have no idea where the operating system comes into play here? Maybe allocating memory? But how does still have anything to do with how its manipulated which is done locally within the compiled code, not in the OS.

 Also I need a correction on a statement I was foggy about earlier and that is the use of MASK constant which is a poor choice of use and basically not even needed, it should of been embedded within the class so not to be exposed for use elsewhere.

 Also, the naming of "TbitArray" was an unfortunate name to use since in other languages it is a CLASS of BitArray..
on top of that, its an array of a max range value that can never actually be used directly, that too should be hidden in local name space for the class that uses it and only the Pointer version to be exposed for global use.

 Finalizes the comments here because I have no more to say about this, I don't see where it's going to hurt anyone's code if some of these mishaps were corrected so that others can move forward with a more globally excepted naming of a TbitArray class., one that behaves much like the Tbits does but falls in common with other languages for their membership calls etc.
The only true wisdom is knowing you know nothing

egsuh

  • Hero Member
  • *****
  • Posts: 741
Re: suggestions on how do handle a large array of boolean
« Reply #6 on: April 26, 2021, 03:16:55 am »
Quote
I have no idea where the operating system comes into play here?
I'm talking about file transfer.

I thought file transfer only between PCs, in which case I/O speed may not be much different if done through OS --- this is my question. My guess is there is not large difference between reading 5 blocks at once and one by one unless the block size is coincide with default I/O buffer size of OS --- pure question.

But now I came to think about other devices, not PCs, in which case the programmer may have to take care of I/O directly.

 

TinyPortal © 2005-2018