Recent

Author Topic: What is the fastest way to make a copy of a TStringList  (Read 16930 times)

stoffman

  • Jr. Member
  • **
  • Posts: 67
What is the fastest way to make a copy of a TStringList
« on: July 16, 2017, 07:30:51 am »
Hey,

I'm working on a multi-threaded application and one of the threads needs to work with a _copy_ of TStringList (I don't wish to slowdown the main UI thread with locking)

So my question is what is the fastest way to make a copy of a TStringList? did anyone did a benchmark?

Thanks,
Yoni

Xor-el

  • Sr. Member
  • ****
  • Posts: 404
Re: What is the fastest way to make a copy of a TStringList
« Reply #1 on: July 16, 2017, 08:30:50 am »
I regularly use assign even though I have not benchmarked it.

Thaddy

  • Hero Member
  • *****
  • Posts: 14159
  • Probably until I exterminate Putin.
Re: What is the fastest way to make a copy of a TStringList
« Reply #2 on: July 16, 2017, 08:40:23 am »
That is also the correct answer Xor-el. A simple assignment from one to the other (:=) will also work.
Also
Quote
(I don't wish to slowdown the main UI thread with locking)
shows a complete lack of knowledge about threading and a thorough mis-understanding of threading requirements.
You can not avoid protecting the copy operation safely in a multi-threaded scenario. E.g. you *must* use a TCriticalSection or a TMultiReadExclusiveWriteSynchronizer or any other form of safe locking during copying. Note that locking isn't as bad as the name suggests...Memory copy operations are fast on most CPU's.
« Last Edit: July 16, 2017, 08:42:13 am by Thaddy »
Specialize a type, not a var.

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: What is the fastest way to make a copy of a TStringList
« Reply #3 on: July 16, 2017, 09:19:00 am »
actually assign is the slowest method of copying data between stringlists not by much but enough to give you a couple of seconds difference in 100 iterations with a 70MB text file in memory.
If you take a look on how assign is written you will find out that it copies a number of stringlist's properties and then call addstrings which then calls addobject which calls add which calls insert which calls insertitem well you the picture directly calling addstrings or even better set the capacity upfront to avoid multiple getmem call and then call add your self should avoid half of those calls and save a couple of miliseconds per call inside a measurement loop this might end up with a difference of 4~6 seconds on a 45 seconds run. But yeah not locking while copying bad bad bad practise. Just a small reminder the non locking wave of multi threading was born to avoid locking the motherboard's BUS not the access to the data.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Thaddy

  • Hero Member
  • *****
  • Posts: 14159
  • Probably until I exterminate Putin.
Re: What is the fastest way to make a copy of a TStringList
« Reply #4 on: July 16, 2017, 09:53:03 am »
actually assign is the slowest method of copying data between stringlists
Nope. Look at the code for assign and the setter method of the property. Exactly the same. If there is a slight speed difference, it's because not everything is inlined. (Same goes for Delphi) And in a multi-threaded application you *have to* lock the memory copy operation. <grumpy, you could have looked at the code yourself  >:D >:D>
« Last Edit: July 16, 2017, 09:57:27 am by Thaddy »
Specialize a type, not a var.

Blestan

  • Sr. Member
  • ****
  • Posts: 461
Re: What is the fastest way to make a copy of a TStringList
« Reply #5 on: July 16, 2017, 10:04:59 am »
you can use 2 string list and switch between with interlockedcompareexchange
Speak postscript or die!
Translate to pdf and live!

Thaddy

  • Hero Member
  • *****
  • Posts: 14159
  • Probably until I exterminate Putin.
Re: What is the fastest way to make a copy of a TStringList
« Reply #6 on: July 16, 2017, 10:13:29 am »
you can use 2 string list and switch between with interlockedcompareexchange
No, that would give a lock on the pointer, or even a string a piece... or it crashes at some point.
The thing is you have to lock the memory area, not just the pointer... <sigh>
If you would use your suggestion, that CAN use that, provided you, the programmer, REMEMBERS it. Which is highly unlikely in the case of almost everybody including you and OP and me, myself and I...

Interlocked functions are by definition locks anyway.....

OP should do what I wrote: use a Critical section. No further stupid replies plz. It's Sunday and I have a F1 race to watch..
« Last Edit: July 16, 2017, 10:15:22 am by Thaddy »
Specialize a type, not a var.

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: What is the fastest way to make a copy of a TStringList
« Reply #7 on: July 16, 2017, 10:15:03 am »
actually assign is the slowest method of copying data between stringlists
Nope. Look at the code for assign and the setter method of the property. Exactly the same. If there is a slight speed difference, it's because not everything is inlined. (Same goes for Delphi)
So you agree with call chain I described but you disagree with my results? Just to pin point where the nope refers to.


And in a multi-threaded application you *have to* lock the memory copy operation. <grumpy, you could have looked at the code yourself  >:D >:D >
OK so again you agree with me when I say
Quote
bad, bad, bad practice
As for looking at the code I've done one better the numbers I gave on my previous post are what I remember from a test I run a couple of months earlier. I could dig it up for you but then again you are capable enough to write your own.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Thaddy

  • Hero Member
  • *****
  • Posts: 14159
  • Probably until I exterminate Putin.
Re: What is the fastest way to make a copy of a TStringList
« Reply #8 on: July 16, 2017, 10:17:28 am »

I just wrote that any competent programmer could have a look at the code first... You did not. I had to ask you to do that.
Specialize a type, not a var.

Blestan

  • Sr. Member
  • ****
  • Posts: 461
Re: What is the fastest way to make a copy of a TStringList
« Reply #9 on: July 16, 2017, 10:29:37 am »
@thaddy:
1: F1 race is in the afternoon so you have plenty of time to deal with our stupid writings  O:-)
2: my propposal of icx was to avoid copiing between lists... not for a protection mechanism. for the copy itself.
3. strlist is defacto an pointer array so copy is not expensive and no string duplicatin is needed bexause strings use  copy on write semantic.
Speak postscript or die!
Translate to pdf and live!

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: What is the fastest way to make a copy of a TStringList
« Reply #10 on: July 16, 2017, 10:29:53 am »


I just wrote that any competent programmer could have a look at the code first... You did not. I had to ask you to do that.
yeah because remembering the exact call chain of stringlist is one of my super powers that I constantly use to conquer the world and did not had to look it up in the first place. Using the same tone as you do, get your head out of your ass and start reading carefully before posting nonsense or you will get ignored permanently
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

wp

  • Hero Member
  • *****
  • Posts: 11830
Re: What is the fastest way to make a copy of a TStringList
« Reply #11 on: July 16, 2017, 11:07:23 am »
A simple assignment from one to the other (:=) will also work.
The OP is talking of stringlists in general, not necessarily of stringlists as properties. For standalone stringlists this is wrong because the operator := here only copies the pointer to the stringlist, not the contents of the stringlist. Therefore, this program crashes at the end:
Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   Classes;
  7.  
  8. var
  9.   L1, L2: TStringList;
  10.   ch: Char;
  11. begin
  12.   L1 := TStringList.Create;
  13.   for ch:= 'A' to 'Z' do
  14.     L1.Add(ch);
  15.  
  16.   L2 := TStringList.Create;
  17.   L2 := L1;      
  18.   L2.Free;
  19.  
  20.   L1.Free;  // <--- crash here because L1 already has been destroyed in the line above.
  21.  
  22. end.

stoffman

  • Jr. Member
  • **
  • Posts: 67
Re: What is the fastest way to make a copy of a TStringList
« Reply #12 on: July 16, 2017, 11:44:19 am »
Thank you for your suggestions.

Regarding the multi-threading part copy is done on the main thread which then transfer it to the thread to do whatever needs to be done. While I understand it wasn't clear from the original message (and one could conclude whatever she/he likes) please @Thaddy use a better language next time.

 


taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: What is the fastest way to make a copy of a TStringList
« Reply #13 on: July 16, 2017, 11:52:38 am »
If you do not need the copied strings to remain in the original list you should listen to the other posts and simple create a new list to use as primary and pass the "filled" one for processing instead of coping data between lists.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

wp

  • Hero Member
  • *****
  • Posts: 11830
Re: What is the fastest way to make a copy of a TStringList
« Reply #14 on: July 16, 2017, 12:38:12 pm »
So my question is what is the fastest way to make a copy of a TStringList? did anyone did a benchmark?
There's one in the attachment. It demonstrates that a plain old "for" loop copying string by string is fastest, faster than Assign. It shows also that there is some speed advantage if the StringList Capacity is set to the known size of the source list (of course...).

 

TinyPortal © 2005-2018