How to develop Crc for TstringList? Can I keep 12M file inside StringList?

[d2010]

Hello..
My file copenttd.big, has size=copenttd.big 13,515,411=12Mo of ascii#13#10
My routine pix15_big_flistsvc, read from copenttd.big and send line by line, or each line to "SendMessage(ListHanbox, LB_ADDSTRING,"
 
C:Q1 =How to avoid 5*reiterator of callpas on samefile "pix15_big_flistsvc"?
    C:Q2= You search ??? with CTRL+F? Reloaded too much?

Code: [Select]

Eg. of Callpas
[code]
Program main;
Var bigGrinder:string='';
bigGrinder_oleacrc:double=0.0;
Begin
 dwg:=pix15_big_flistsvc(100,lisrt) +
    pix15_big_flistsvc(102,lisrt)   +
    pix15_big_flistsvc(132,lisrt)   +
    pix15_big_flistsvc(142,lisrt)   +
    pix15_big_flistsvc(152,lisrt)   +
End;

Function pix15_big_flistsvc(exe:word;ListHanbox:tHandle):integer;
Var cnt,nth,dxf,procenow,len1line,siz1step,sizenext,size3all,sizesuma:integer;
    bill:double;
Label vbCanNext;
Begin  nth:=00;
       dxf:=000;
       bill:=0.0;
        size3all:=php_filesize(ktoolfini_openttd32lst);
       with TStringList.Create do
Try
       Try
          java_index:=000;
          if (fileexists(ktoolfini_openttd32lst)) then
                   LoadFromFile(ktoolfini_openttd32lst) else
             SetText(orto_Painica2034cuminte71);

          cnt:=count-01;
          for nth:=00 to count-01 do
             Begin len1line:=length(Strings[nth]);
                   sizesuma:=sizesuma+len1line+02;
                   if (len1line<3) then goto vbCanNext;
                   strpcopy(@ddraw2.car[8],Strings[nth]);
                   dxf:=SendMessage(ListHanbox, LB_ADDSTRING, java_index,integer(@ddraw2.car[8]));
                   inc(java_index);
                   if (sizesuma<sizenext) then goto vbCanNext;
                   while (sizesuma>=sizenext) do
                      Begin adui_inc_digi23(@ddraw2.car[3],'z');
                            sizenext:=sizenext+siz1step;
                      End;
                    ddraw2.car[8+len1line]:='"';
                    ddraw2.car[9+len1line]:=#000;
                    SendMessage(StatWnd, SB_SETTEXT, 0,integer(@ddraw2.car[0]));
vbCanNext:
             End;//off("lb_addstringmy2008(100);")
finally
end;
finally
         free;
     End;
//off("la fisiere de 12Mb, incarcarea cu "SrcList.Items.LoadFromFile(ktoolfini_openttd32lst);" dureaza foarte mult")
   if (irunin.ini>=100) and ((irunin.ini-100)<cnt) then
   SendMessage(listHanBox,LB_SETCURSEL,irunin.ini-100,000);
End;
[/code]

 
My solution is calculate with pdouble()
              Begin
                   bigGrinder_oleacrc:=pDouble(@bigGrinder)^;???
                   bulgaria:='I am must100%sure not more Loading from, otherwise I keep the values in memory';
              End;

[jamie]

Does not look like you are getting many responses.

To be truthfully I cannot even figure out what you're trying to do that does not look like any CRC coding I've ever seen

[TRon]

I had a quick glance and am puzzled as well. I am probably going to ridicule myself with the following analysis: Code tries to load a file that has specific entries listed in textual form. It contains data-files that are stored within that same file and contain fields that list the checksum of these datafiles (md5 if my guess is correct). These checksums need to be verified.

Can you load 12 million+ lines in a stringlist ? Sure, in case there is enough memory. Is it the smartest solution ? I don't believe it is, especially not in case wanting to search the file (as I interpret was indicated). A virtual list or tree comes to mind.

I am especially puzzled by how the lines are added to the listbox using sendmessages (probably a windows thing ?).

[d2010]

Does not look like you are getting many responses.
To be truthfully I cannot even figure out what you're trying to do that does not look like any CRC coding I've ever seen

I try load, store the file in memory, and create CRC, all-three functions, inside OnceFunction.
 I replace  "bigGrinder_oleacrc:=pDouble(@bigGrinder)^;"  with MD5 of file.
Thank you.

[TRon]

I replace  "bigGrinder_oleacrc:=pDouble(@bigGrinder)^;"  with MD5 of file.

How would a 128-bit MD5 checksum fit into a double ?

Or were you actually referring to a CRC checksum (of either 8, 16, 32 or 64 bit) ?

fwiw, a checksum is not something to store into a variable of type double, unless I missed something.

BTW so that there is no further wasting time poking in the dark: you are trying to verify the checksum of grf files ?

[paule32]

I understand the OP as:
- he/she would store MD5 Checksum's

Then my answer is:
- use a Database like QSQLite
- then you can md5 Hashes, AND: encode/chipher the SQLite file (to make binary store)

[TRon]

FPC does come with package hash paule32 and as far as my understanding of the fileformats of ttd go, the checksum is used as verification. stored in another file. In which case user has no choice in the matter (other than to follow API/Guidelines).

[paule32]

so, how I understand the OP, he would store md5 Strings in a TStringList.
For 12,000,000 (twelfemillion) Items and 64 Byte for each, I would prefer Database File.
Because:
12,000,000 mul 64 Bytes = 768,000,000 Million Bytes

that means, that you need 768,000 MegaBytes
that means, that you need a Database that can handle minimum 1 GigaByte of Space.


Windows 32-Bit comes with 3,1 GigaByte of RAM

that means, that you have 2,1 GigaByte RAM for Applications
that means, that Windows comes with 1 GigaByte of System-Used RAM

that means, that you have minimal 1,1 of unused RAM

that means, that the logic is 1,1 GByte free RAM => 1 GByte costs for MD5 storage

that means, that you have 1.240.000 free GigaByte for the Application that should store the MD5 data.

that means, that you have: 1.240 - 732 = 508 MegaBytes of free RAM

that means, that you have round 9 Memory Pages a 64 K

that means, that I have seen only 4 Memory Paging in real World Applications

that means, that we on the Limt of 4 x 8 = 32 Bit CPU  32 - 1x Memory Page ( = 24 Bit

so, tell me: it is possible or not ?

[TRon]

so, how I understand the OP, he would store md5 Strings in a TStringList.

Interesting take on things. And in which case I am not disputing your calculations.

I rather think the checksum needs to be calculated over each entry in the stringlist (or over the bytes in the file, TS is not very clear on that) so iterating over each line and updating the checksum. In case of check-summing the file this can be done in a single pass.

Two different takes, two different approaches