Recent

Author Topic: Large or huge sets  (Read 10103 times)

Arioch

  • Sr. Member
  • ****
  • Posts: 421
Re: Large or huge sets
« Reply #45 on: September 14, 2022, 09:01:05 pm »
We just added them before Delphi did, thus they are not compatible in the way they are declared.

Same you did with generics, but then in Delphi Mode you added Delphi syntax for them

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Large or huge sets
« Reply #46 on: September 14, 2022, 09:26:10 pm »
Same you did with generics, but then in Delphi Mode you added Delphi syntax for them

(8 years later, so maybe that is not the best example  ;) )

Arioch

  • Sr. Member
  • ****
  • Posts: 421
Re: Large or huge sets
« Reply #47 on: September 14, 2022, 10:38:35 pm »

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Large or huge sets
« Reply #48 on: September 15, 2022, 09:27:16 am »
We just added them before Delphi did, thus they are not compatible in the way they are declared.

Same you did with generics, but then in Delphi Mode you added Delphi syntax for them

I'm not saying that we won't add support for Delphi's syntax just why it isn't supported yet.

okay, have your mega-ticket

https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/39902

Great job... you added a bug report for the FPC documentation to the Lazarus project.  ::)

Arioch

  • Sr. Member
  • ****
  • Posts: 421
Re: Large or huge sets
« Reply #49 on: September 15, 2022, 06:59:08 pm »
Great job... you added a bug report for the FPC documentation to the Lazarus project.  ::)

Indeed... do not race against clock in dead midnight :-(

But cannot tracker admins move the ticket to another, related tracker, like they can on StackOverflow?  Because i definitely can not, only copy-paste...

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Large or huge sets
« Reply #50 on: September 16, 2022, 10:12:26 am »
Copy paste Symmetric difference to your browser and presto, you have the reply in ~10 seconds.

It would be that very guesswork that i said is not enough.

Google does not describe this concept as applied to specific FPC types, does not give authoritative (meaning, FPC would be obliged to support it) FPC code examples.
If you google for "><" "symmetric difference" pascal, then this is the first link you get: https://wiki.freepascal.org/symmetric_difference. And it has FPC code example. The very next link is https://www.freepascal.org/docs-html/ref/refsu49.html.
« Last Edit: September 16, 2022, 10:17:04 am by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 14204
  • Probably until I exterminate Putin.
Re: Large or huge sets
« Reply #51 on: September 18, 2022, 10:40:47 am »
Side note:
It is bloody irritating that google search prefers wiki entries over real documentation. Our wiki is much worse than the real official documentation. Does not matter too much in this case, but it is annoying and may put people on the wrong foot. The wiki contains too much dubious entries.

Side note two: set >< syntax is indeed very old, why should it be confusing on inappropriate? It isn't.
I use sets a lot, the old hand that made that remark obviously does not know how to work with sets except for the bare minimum. Also note - huge - sets are what big data is all about and based on simple mathematical "Set theory"
« Last Edit: September 18, 2022, 10:45:38 am by Thaddy »
Specialize a type, not a var.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Large or huge sets
« Reply #52 on: September 18, 2022, 01:21:12 pm »
Side note:
It is bloody irritating that google search prefers wiki entries over real documentation. Our wiki is much worse than the real official documentation. Does not matter too much in this case, but it is annoying and may put people on the wrong foot. The wiki contains too much dubious entries.

Might help if it were available in PDF form via a standard protocol. It's a long time since I've seen Google indexing something only available via FTP.

I wish there were some way of depressing the ranking of a wiki page that hadn't been reviewed for an extended period. However the real problem is users who quite simply fail to understand the difference between fully-maintained version-specific documentation and hearsay from a random page suggested by Google.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Arioch

  • Sr. Member
  • ****
  • Posts: 421
Re: Large or huge sets
« Reply #53 on: September 18, 2022, 02:23:14 pm »
Now, official wiki at the official project TLD is hardly a
hearsay from a random page

When i - a newb and a stranger - was offered to just go and  edit LAZ wiki, instead of triggering talks in tracker - i freaked out

Now, you can put a banner in every page of wiki, saying "it is just draft and hearsay, try to find information in official doc first, here and there". This is at least under your control.

And Google probably operates in positive feedback loop. People see top search results - and click themm, and give links to them from forums/blogs. Google observes that behavior and upranks those pages, and this go rounds. Few years ago i was googling how to correctly compare floats in Pascal, so as to not write yet ad hoc retelling of ABC to yet another newb on Stack Overflow. And the top result was some UK FAQ site that bona fide suggested if FloatToSr(x) = FloatTostr(y) then...  They seem to take down that advice now, luckily.
« Last Edit: September 18, 2022, 02:25:27 pm by Arioch »

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: Large or huge sets
« Reply #54 on: September 18, 2022, 03:17:30 pm »
@Arioch

   Do you own a license of recent Delphi?
The only true wisdom is knowing you know nothing

Arioch

  • Sr. Member
  • ****
  • Posts: 421
Re: Large or huge sets
« Reply #55 on: September 18, 2022, 03:22:33 pm »
i prefer XE2 and can have it at work, i can have latest Community at home, but it sadly lacks sources

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Large or huge sets
« Reply #56 on: September 18, 2022, 04:02:22 pm »
Now, official wiki at the official project TLD is hardly a
hearsay from a random page

When i - a newb and a stranger - was offered to just go and  edit LAZ wiki, instead of triggering talks in tracker - i freaked out

I wasn't necessarily thinking of the wiki as hearsay etc., and I'd direct you to your later example about comparing floating point numbers.

However the fact remains that- at least in this project and community- the wiki is /not/ official documentation.

Quote
Now, you can put a banner in every page of wiki, saying "it is just draft and hearsay, try to find information in official doc first, here and there". This is at least under your control.

More importantly, and more relevant to my point, is that the wiki contains lots of stuff on "paths not taken": either discussion on relating to "approach A" when what Lazarus actually implemented was "approach B", or one-person subprojects where the person lost interest (possibly after demanding some precondition which he was told was quite simply not viable).

I share your feelings about Google being an echo chamber, but a particularly worrying thing is that when I watch people trying to use "The Interwebs" I see them go to the first page suggested by Google, read one screen, then backtrack and go to the second page. A particularly pernicious example is when they go to Wikipedia to look up e.g. a medical condition using their 'phone, and fail to realise that (a) coloured text is a link they can follow and (b) in the interest of brevity everything after the initial para has been compressed so they have to click on the section header before they see anything useful.

So they might start off being directed by Google to- say- the Wikipedia article on COVID, see nothing they recognise as useful or intelligible, backtrack and be given a Facebook Anti-vax page as the next suggestion.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

strmckr

  • Newbie
  • Posts: 6
Re: Large or huge sets
« Reply #57 on: September 20, 2022, 08:56:05 am »
Quote
What is "><" ? It is "symmetric difference".

yes its symmetrical difference from set theory

a >< b    would be the set of digits not found in both sets.

if
  a is [1..5]
  b is [ 5,6] 
    then the  >< of AB  is [1..4,6]

which is exactly the opposite of intersection
     a*b  = [5]

constructively the instructions are
( (A+B)  - (A*B) )   

web linked above by others seems to be in order....

i am a self taught coder:  using Free-pascal IDE compiler
  there is lots i don't know how to use  if you have a way that i can use or learn from  pls divulge or link with examples


i need a code that can make a data point of 9 values  each storing a set  terms out of a size of 46656 possibles 

 0-80 points each mark up with corresponding active sets of a list out of the 46656 sets.
   so that i can do the following set-wise operations  +,-, *, =, <>, ><,  and check the 0..80 points are not empty or not
 
 like regular sets can use :    a * b = [] ,  or   a*b <> []  or  a*b = C 

while comparing n points to each other.

currently I'm attempting to use thaddys examples with my mods,

 works but i run into issues that i cannot update the initial data points with new information / changes on successive runs {they just never seem  to empty or clear or even reset}

or successive runs triggers a memory limit fault and crashes everything.
 
« Last Edit: September 20, 2022, 09:20:34 am by strmckr »

Thaddy

  • Hero Member
  • *****
  • Posts: 14204
  • Probably until I exterminate Putin.
Re: Large or huge sets
« Reply #58 on: September 20, 2022, 09:50:51 pm »
I will look into that. What is the data size of the individual elements? 40K elements should not be a problem unless the data size itself is huge (multiple Gb) or exceeds max(prtUint) for your platform.
One solution is to use windowed access. Basically intermediate storage.
I was not aware about the modification issue.
Your program should not crash anyway, as an EOutOfMemory exception can still recover somewhat or exit gracefully, since it is preallocated.
Requires the use of exceptions, though. And I warned about creating a huge set with higher unsigned integer types like qword.

Those cases need to be resolved in a different way, but it can follow the same algorithm.
« Last Edit: September 20, 2022, 09:53:46 pm by Thaddy »
Specialize a type, not a var.

strmckr

  • Newbie
  • Posts: 6
Re: Large or huge sets
« Reply #59 on: September 21, 2022, 08:58:46 am »
Quote
And I warned about creating a huge set with higher unsigned integer types like
noted im just using:  Twordset

the data size of each set?
     N has size range of 1 - <600{max} depends on test case i'm checking
                      on a blank examination space a 46656 set is present for each N

the set range is
     [0..46655] 
 
 {each of these represents a 9 digit template where the 9 digits have a positional tag.}  -> not worried about this part

each: number 1-9 (n) 
where N is a set specifically of all the sets from the range that are applicable to N
     
 then   i have

[1..9][0..80] of set
   so that each N digit has 80 points  where each  0..80 is a set of x from N  that match 

code needs to be able to do

N(x) + N(x) = N 
{two cells match the full set of N}
   
   if i tried that directly using normal set syntax the system cant compute it at all so wrote a function for it.
 
testing code inserted into my per-working function that generated all 46656 sets, and a break down of N saved as its own  file.

Code: Pascal  [Select][+][-]
  1. // 46,656 potential  single digit grids.
  2.  
  3. procedure potential;
  4.  
  5. type
  6. hold = array of integer;
  7. base = array of numberset;
  8. digit = array of numberset;
  9.  
  10.  
  11. var
  12.    w1,w2,w3:TWordSet;  
  13.  
  14. setstuff: array [digits] of Twordset;
  15. Cellsetstuff: array [ digits,cell] of twordset;
  16.  
  17. locked:  array [digits] of numberset;
  18. deleted:  array [digits] of numberset;
  19.  
  20. stuff: array [digits] of numberset;
  21.  
  22. alist:array[digits] of word;
  23.  
  24. xn,w,p,n,n2,xn2,q,g,L,j,m,o,xn3,xn4,xn5:integer;
  25. output: text;
  26.  
  27. step: base;
  28. h: hold;
  29. z:digit;
  30.  
  31. begin
  32.  
  33. for n:=1 to 9 do
  34.  begin
  35.  setstuff[n]:= twordset.create;
  36.    for q in digitcell[n] do      
  37.     cellsetstuff[n,q]:=twordset.create;        
  38. end;
  39.  
  40.   for n:= 1 to 9 do
  41.     alist[n]:=0;
  42.        
  43. for N:= 1 to 9 do
  44. begin
  45. locked[n]:=[];
  46. deleted[n]:=[];
  47. stuff[n]:= [];
  48.  
  49. for xn:= 0 to 80 do
  50.  begin
  51.  
  52.   if s[xn] = [n]
  53.    then
  54.       locked[n]:= locked[n]+ [xn];
  55.  
  56.    if (s[xn] <> [n] ) and not( n in pm[xn])
  57.     then
  58.      deleted[n]:= deleted[n] + [xn];
  59.  
  60.   end;
  61.  end;
  62.  
  63. {startin cell}
  64.  
  65.   {delete the exsiting output if you want to rebuild it unhash this section}
  66. {assign(output,'C:\sudoku\pom\output.txt');
  67. erase(output);
  68. rewrite(output);
  69. close(output);  }
  70.  
  71. assign(output,'c:\sudoku\pom\exclusions.txt');
  72. erase(output);
  73. rewrite(output);
  74. close(output);
  75.  
  76. {smashes all prebuilt  txt files  of potential solutions for digits 1-9  }
  77.  for n:= 1 to 9 do
  78.   begin
  79.     case n of
  80.       1: assign(output,'C:\sudoku\pom\1.txt');
  81.       2: assign(output,'C:\sudoku\pom\2.txt');
  82.       3: assign(output,'C:\sudoku\pom\3.txt');
  83.       4: assign(output,'C:\sudoku\pom\4.txt');
  84.       5: assign(output,'C:\sudoku\pom\5.txt');
  85.       6: assign(output,'C:\sudoku\pom\6.txt');
  86.       7: assign(output,'C:\sudoku\pom\7.txt');
  87.       8: assign(output,'C:\sudoku\pom\8.txt');
  88.       9: assign(output,'C:\sudoku\pom\9.txt');
  89.      end;
  90.  
  91. erase(output);
  92. rewrite(output);
  93. close(output);
  94. end;
  95.  
  96. setlength(step,0);
  97. setlength(z,0);
  98. setlength(h,0);
  99.  
  100.  for xn:= 80 downto 72 do
  101.  
  102.   begin
  103.  
  104.   w:=0;    {step count}
  105.  
  106.   setlength(h,(w+1));  {set the array size to w}
  107.  
  108.   h[w]:=xn;        {starting point for first substep}
  109.  
  110.   setlength(step,(w+1));   {set the array size to w}
  111.  
  112.   step[w]:= [xn];  { keeps track of what cells are used at each step W }
  113.  
  114.   setlength(z,w+1);  {prevent occupy "new starting point" }
  115.  
  116.   z[w]:= peer[xn] + [xn]; {used cells  starting point}
  117.  
  118.    repeat
  119.  
  120.     for p:= h[w] downto (72-((w+1)*9)) do    {iteration of peers}
  121.  
  122.       if  p in ([0..80] - z[w] ) // added check used state
  123.        then
  124.          begin
  125.  
  126.           h[w]:=h[w]-1;    { advance the peer count for step w}
  127.  
  128.           inc(w);        {increase the step count}
  129.  
  130.           setlength(h,(w+1));
  131.           setlength(step,(w+1));     {increse the array size  to w}
  132.  
  133.           step[w]:= step[w-1] + [p] ;   {set the step cell active for the newly created step w}
  134.  
  135.           h[w]:= 71 - ((w)*9) ;  {set the new step w as 71 potential choice cells}
  136.  
  137.           setlength(z,w+1);  { increase size to w}
  138.  
  139.           z[w]:= z[w-1] +  peer[p] + [p]; {used cells  new  point}
  140.  
  141.           break;
  142.  
  143.         end
  144.        else
  145.           h[w]:=h[w]-1;  {if the above is false then advance the peer number}
  146.  
  147.  
  148. if w = 8
  149.   then
  150.    begin
  151.  
  152.  
  153. { generate the whole list to a specific file}
  154. {assign(output,'C:\sudoku\pom\output.txt');
  155. append(output);
  156. for n in step[w] do
  157.     write(output,n,' ');
  158.  
  159.     writeln(output);
  160.  
  161.     close(output); }
  162.  
  163.  for n:= 1 to 9 do
  164.   begin
  165.  
  166.     case n of
  167.       1: assign(output,'C:\sudoku\pom\1.txt');
  168.  
  169.       2: assign(output,'C:\sudoku\pom\2.txt');
  170.  
  171.       3:  assign(output,'C:\sudoku\pom\3.txt');
  172.  
  173.       4: assign(output,'C:\sudoku\pom\4.txt');
  174.  
  175.       5: assign(output,'C:\sudoku\pom\5.txt');
  176.  
  177.       6: assign(output,'C:\sudoku\pom\6.txt');
  178.  
  179.       7: assign(output,'C:\sudoku\pom\7.txt');
  180.  
  181.       8:  assign(output,'C:\sudoku\pom\8.txt');
  182.  
  183.       9:  assign(output,'C:\sudoku\pom\9.txt');
  184.        end;
  185.  
  186.   if ( step[w]  * locked[n] = locked[n] )
  187.   and ( step[w] * deleted[n] = [] )
  188.    then
  189.      begin
  190.          inc(alist[n]);
  191.        append(output);
  192.  
  193.        for q in (step[w] - locked[n])  do
  194.              begin
  195.           write(output, q,' ');
  196.                  
  197.                   include2(cellsetstuff[n,q],alist[n]);
  198.                   include2(setstuff[n],alist[n]);
  199.                   end;
  200.  
  201.         writeln(output);
  202.         close(output);
  203.  
  204.         stuff[n]:= stuff[n] + (step[w] - locked[n]);
  205.  
  206.      end;
  207.  
  208.  
  209. end; { N choice}
  210.  
  211.  end;   {w=8}
  212.  
  213.  
  214.     if ((h[w] < 0 )  and (w > 0 ))
  215.       or (w=8)
  216.       or ( ( [0..80] - z[w] = [] ) and (W < 8) and (w > 0) )
  217.        or ( (h[w] < (72 - ( (w+1)*9) ) )  and (w > 0)  )
  218.  
  219.     {the following resets the step to the previous state}
  220.      then
  221.       repeat
  222.       begin
  223.        w:=(w-1);
  224.        setlength(h,(w+1));
  225.        h[w]:= h[w];
  226.        setlength(step,(w+1));
  227.        setlength(z,w+1);
  228.  
  229.         end;
  230.        until   ( w = 0 ) or (h[w] > ((71 - (w+1)*9))  )
  231.  
  232.     until ((w = 0) and (h[w] < 0) ) or  ( ( w = 0) and (h[w] < (72 -( (w+1)*9) ) ) )
  233.  
  234.  end;
  235.  {size printing of all sets}
  236.  for n:= 1 to 9 do
  237.  begin
  238.  gotoxy(2,60+n);
  239.   write(n,':=  ',alist[n]);
  240.   end;
  241.  
  242.  {size 1}
  243.   for n:= 1 to 9 do
  244.  if  (stuff[n] <> [])
  245.  and (stuff[n] *  (digitcell[n]  -  (locked[n] + deleted[n]) ) = stuff[n])
  246.  and (  ((digitcell[n]  -  (locked[n] + deleted[n]) ) - stuff[n])  <> [] )
  247.   then
  248.     begin
  249.      assign(output,'C:\sudoku\pom\exclusions.txt');
  250.      append(output);
  251.         write(output,n, ' @: ');
  252.                         // cell has no templates out of all of them.    
  253.         for  xn  in  ((digitcell[n]  -  (locked[n] + deleted[n])) - stuff[n])do
  254.           begin
  255.             write(output,xn,' ');
  256.                     covered2[n]:=covered2[n]+[xn];
  257.                     active:=true;
  258.                    end;
  259.                    
  260.                    //cell is exclusivly in all sets
  261.                   for xn in digitcell[n] do  
  262.                    if equality(cellsetstuff[n,xn],setstuff[n])
  263.            then  
  264.               begin
  265.                              active:=true;
  266.                  covered[xn]:=covered[xn] + (pm[xn]-[n]);
  267.                                  write(output,'*',xn,'<>( ');
  268.                                 for o in pm[xn]-[n] do
  269.                                  write(output,o,' ');
  270.                                   write(output,') ')
  271.                 end;                                   
  272.                    
  273.                    
  274.            writeln(output);
  275.            close(output);
  276.                    
  277.                    
  278.      end; {size 1}
  279.  
  280.  {size 2}
  281.  for n in [1..9] do
  282.     for xn in digitcell[n] do
  283.            for xn2 in digitcell[n]*[(xn+1)..80] do    
  284.           begin
  285.        w1:=twordset.create;
  286.            
  287.        w1:= cellsetstuff[n,xn] + cellsetstuff[n,xn2];
  288.  
  289.          if  setstuff[n]<=w1
  290.            then
  291.                     begin                        
  292.                           for q in [1..9]-[n] do
  293.                            begin
  294.                  for g in setstuff[q] do
  295.                   if [xn,xn2] * digitcell[q] = [xn,xn2]
  296.                                    then                                  
  297.                                     if (g in (cellsetstuff[q,xn]))
  298.                                      and (g in (cellsetstuff[q,xn2]))
  299.                                       then
  300.                                           begin                                                          
  301.                         w2:=twordset.create;
  302.                                                
  303.                                            for L in digitcell[q] do
  304.                                              if (g in cellsetstuff[q,l])
  305.                                                    then
  306.                                                      begin
  307.                                                      exclude2(cellsetstuff[q,L],g);    
  308.                                                          include2(w2,l);
  309.                                                          end;
  310.  
  311.  
  312.         assign(output,'C:\sudoku\pom\exclusions.txt');
  313.      append(output);
  314.           write(output,'(',n,')',xn,',',xn2,':(');
  315.             for m in setstuff[n] do write(output,' ',m); write(output,' ) & ');
  316.          write(output,'(',q,')',xn,',',xn2,':(');
  317.                 for o in (setstuff[q]) do write(output,' ',o); write(output,' ) ');
  318.               write(output,'RT: ', G,' @ Digit: ',q, ' =>> <> ');
  319.                          
  320.                           exclude2(setstuff[q],g);                       
  321.                                                    
  322.                                                  for L in digitcell[q] do
  323.                                                     begin
  324.                                                            {no templates left for cells}
  325.                               w3:= setstuff[q] - cellsetstuff[q,l];                              
  326.                                                            if equality(w3,cellsetstuff[q,l])
  327.                                                               then
  328.                                                                     begin
  329.                                                                       write(output,L,' ');
  330.                                                                           active:=true;
  331.                                                                           covered2[q]:=covered2[q] + [L];
  332.                                                                           end;
  333.                                                                 {digits are locked to all sets}          
  334.                                                                 if equality(cellsetstuff[q,l],setstuff[q])
  335.                                    then  
  336.                                      begin
  337.                                                                          active:=true;
  338.                                       covered[l]:=covered[l] + (pm[l]-[q]);
  339.                                                                           write(output,'*',L,'<>( ');
  340.                                                                            for o in pm[l]-[q] do
  341.                                                                              write(output,o,' ');
  342.                                                                                  write(output,') ')
  343.                                      end;                                                                        
  344.                                                          end;
  345.                                                        
  346.                                                        
  347.                                                          
  348.                   writeln(output);
  349.            close(output);      
  350.                    
  351.                    include2(setstuff[q],g);
  352.                    for L in w2 do
  353.                    begin
  354.                                 include2(cellsetstuff[q,L],g); 
  355.                            end;
  356.                    w2.free;
  357.                          break;                                        
  358.  
  359.                                           end;                                     
  360.                                         end;                                     
  361.                  end;
  362.                                  w1.free;
  363.                                  
  364.             end;        {size 2}                 
  365.  
  366. end;

 when i run this code, then reload the stat : and run again
i get
EacessViolation: access Violation
or
  some random memory limit error when i remove the "break" from the above code.

this code for me: is something I've wanted to add to my program for years but lacking large sets has hindered my progress on it.
« Last Edit: September 21, 2022, 09:18:20 am by strmckr »

 

TinyPortal © 2005-2018