Recent

Author Topic: Sorting and Counting  (Read 8365 times)

julkas

  • Sr. Member
  • ****
  • Posts: 424
  • KISS principle / Lazarus 2.0.0 / FPC 3.0.4
Re: Sorting and Counting
« Reply #135 on: November 10, 2019, 07:19:42 pm »
We can't compare Python (interpreted Lang) with Pascal.
Yes, we can since Python relies so heavily on library code compiled in other languages. Python is just glue. Much more so than pure scripting.
(You can also use FPC to write and add Python libraries)
And indeed:
I use Python and I like Python... :P :o
PYTHON rocks.
PYTHON is great.
PYTHON forever.
PYTHON νῦν και ἀεὶ.
« Last Edit: November 10, 2019, 07:38:38 pm by julkas »
procedure mulu64(a, b: QWORD; out clo, chi: QWORD); assembler;
asm
  mov rax, a
  mov rdx, b
  mul rdx
  mov [clo], rax
  mov [chi], rdx
end;

Thaddy

  • Hero Member
  • *****
  • Posts: 9273
Re: Sorting and Counting
« Reply #136 on: November 10, 2019, 07:29:25 pm »
Well, as it seems it needs some FPC compiled sort libraries,,,,, 8-)
also related to equus asinus.

mpknap

  • Jr. Member
  • **
  • Posts: 92
Re: Sorting and Counting
« Reply #137 on: November 10, 2019, 08:11:23 pm »
Ultimately, I use this algorithm, it is clear to me and because I have a problem installing the LG package ....
30MB file counted and sorted is 6 minutes;)

I think TStringList is eminently suitable for this task.
Here's an alternative solution, which may use less resources.
Code: Pascal  [Select]
  1. unit mainSortCount;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.    Classes, SysUtils, Forms, StdCtrls;
  9.  
  10. type
  11.    TForm1 = class(TForm)
  12.      Memo1: TMemo;
  13.      procedure FormCreate(Sender: TObject);
  14.    end;
  15.  
  16. var
  17.    Form1: TForm1;
  18.  
  19.   procedure SortCount(const anInFile: String; out aList: TStringList);
  20.  
  21.   procedure ShowListInMemo(constref aList: TStringList; aMemo: TMemo);
  22.  
  23. implementation
  24.  
  25. {$R *.lfm}
  26.  
  27. { TForm1 }
  28.  
  29. procedure SortCount(const anInFile: String; out aList: TStringList);
  30. const
  31.   one = PtrUInt(1);
  32. var
  33.   textf: TextFile;
  34.   s: String;
  35.   idx: Integer;
  36.  
  37.   function GetSuccObj(anIntObj: TObject): TObject;
  38.   var
  39.     i: PtrUInt absolute anIntObj;
  40.   begin
  41.     Inc(i);
  42.     Exit(anIntObj);
  43.   end;
  44.  
  45. begin
  46.   Assert(FileExists(anInFile), 'cannot find file "'+anInFile+'"');
  47.   aList := TStringList.Create;
  48.   aList.Duplicates := dupError;
  49.   aList.Sorted := True;
  50.   AssignFile(textf, anInFile);
  51.   try
  52.     Reset(textf);
  53.     while not EOF(textf) do
  54.       begin
  55.         ReadLn(textf, s);
  56.         s := Trim(s);
  57.         idx := aList.IndexOf(s);
  58.         case idx of
  59.           -1: aList.AddObject(s, TObject(one));
  60.           else
  61.             aList.Objects[idx] := GetSuccObj(aList.Objects[idx]);
  62.         end;
  63.       end;
  64.   finally
  65.     CloseFile(textf);
  66.   end;
  67. end;
  68.  
  69. procedure ShowListInMemo(constref aList: TStringList; aMemo: TMemo);
  70. var
  71.   i: Integer;
  72. begin
  73.   if Assigned(aList) and Assigned(aMemo) then
  74.     begin
  75.       aMemo.Clear;
  76.       for i := 0 to aList.Count-1 do
  77.         aMemo.Lines.Add('%s - %d', [aList[i], PtrUInt(aList.Objects[i])]);
  78.     end;
  79. end;
  80.  
  81. procedure TForm1.FormCreate(Sender: TObject);
  82. var
  83.   sl: TStringList;
  84. begin
  85.   SortCount('infile.txt', sl);
  86.   try
  87.     ShowListInMemo(sl, Memo1);
  88.     Memo1.Lines.SaveToFile('outfile.txt');
  89.   finally
  90.     sl.Free;
  91.   end;
  92. end;
  93.  
  94. end.


As for Python, that was just curiosity, but thank you.

In general, the problem is a little different. Unixtime is one variable of a certain record. It's about displaying records and their numbers with the same Unixtime. After sorting, I don't know how to refer to the other variables in the record, but I don't want to ask for more :)


Well, unless it's interesting for you;)

440bx

  • Hero Member
  • *****
  • Posts: 1258
Re: Sorting and Counting
« Reply #138 on: November 10, 2019, 08:20:42 pm »
After sorting, I don't know how to refer to the other variables in the record, but I don't want to ask for more :)
Question for you, do you want your program to run on multiple platforms (e.g, Linux, Windows, other) or is Windows only acceptable ?

ETA: if Windows-only is acceptable then, another question: are the records in the file fixed length or variable length ? 
« Last Edit: November 10, 2019, 08:24:41 pm by 440bx »
using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

avk

  • Full Member
  • ***
  • Posts: 162
    • my self-education project
Re: Sorting and Counting
« Reply #139 on: November 11, 2019, 06:13:33 am »
... I have a problem installing the LG package ....

What kind of problem?

Thaddy

  • Hero Member
  • *****
  • Posts: 9273
Re: Sorting and Counting
« Reply #140 on: November 11, 2019, 08:54:20 am »
What kind of install? Just make sure it is in your path.
also related to equus asinus.

mpknap

  • Jr. Member
  • **
  • Posts: 92
Re: Sorting and Counting
« Reply #141 on: November 11, 2019, 09:28:59 am »
After sorting, I don't know how to refer to the other variables in the record, but I don't want to ask for more :)
Question for you, do you want your program to run on multiple platforms (e.g, Linux, Windows, other) or is Windows only acceptable ?

ETA: if Windows-only is acceptable then, another question: are the records in the file fixed length or variable length ?

I work in Windows 10.
At the beginning I asked about sorting Unix times alone, because I thought I could do it myself. Therefore, as an example I gave a different TXT file format.
The base is originally TXT in the format:


1570485826087           - UnixTime
0,0                                -lat
0,0                                -lon
Patrycja Maca┼éa         -user Name
16584                           -User Number
ekonomik1                    -Team Name
1581                             -Team Number
                                     - Free line as separator
1570485826087
0,0
0,0
Gargastw├│r
17943
kosmos
1243

1570485909840
41,6548226860975
12,523491502443182
Emanuele Maria Latorre
4125
Divulgazione Libera
829

1570485929612
50,63142735
19,63182841
Przemek
137
Przemek
148

1570485941031
42,30451192
-71,22277853
Asaf
15190
Israel
2610


Sample file in the attachment, the smallest due to forum restrictions.

These are the times of detection received by users' smartphones in the CREDOSCIENCE project.

I am interested in obtaining such information :
- how many detections are in the same second / minute / hour. (I already got it thanks to your help)
- If there are several in the same second / minute / hour, display which users and their other data as number, coordinates, Team name etc ..

As I wrote, this is not my job, I'm not a programmer, it's fun for me in my free time. Thank you for your interest :)


What kind of install? Just make sure it is in your path.
I will try as you write. I installed the Lgenerics.LPK file, there was an error and I gave up.


440bx

  • Hero Member
  • *****
  • Posts: 1258
Re: Sorting and Counting
« Reply #142 on: November 11, 2019, 03:10:10 pm »

1570485826087           - UnixTime
0,0                     -lat
0,0                     -lon
Patrycja Maca┼éa        -user Name
16584                   -User Number
ekonomik1               -Team Name
1581                    -Team Number
                        - Free line as separator
1570485826087
0,0
0,0
Gargastw├│r
17943
kosmos
1243

<many more>

1567365497795
0,0
0,0
J.C.K.
6037

1

1567367488139
0,0
0,0
J.C.K.
6037

1

Just to make sure I understand the structure of your file and its records.  It looks like every record in the file consists of seven (7) fields (one per line) and that, sometimes, a field may be blank.  I want to confirm that every record is always 7 fields (though one or more, except the unixtime, may be blank), is this correct ? 


I am interested in obtaining such information :
- how many detections are in the same second / minute / hour. (I already got it thanks to your help)
Mostly with the help of the other contributors in this thread since my implementation doesn't even use the appropriate collating sequence.

- If there are several in the same second / minute / hour, display which users and their other data as number, coordinates, Team name etc ..
Piece of pie.  I got several things to do today so I won't commit to a solid timeframe but, I'll give you something soon.

using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

mpknap

  • Jr. Member
  • **
  • Posts: 92
Re: Sorting and Counting
« Reply #143 on: November 11, 2019, 03:42:51 pm »


[/font][/size]
Just to make sure I understand the structure of your file and its records.  It looks like every record in the file consists of seven (7) fields (one per line) and that, sometimes, a field may be blank.  I want to confirm that every record is always 7 fields (though one or more, except the unixtime, may be blank), is this correct ? 

Yes. One record is 7 lines. The eighth line is the separator between records.

 

Piece of pie.  I got several things to do today so I won't commit to a solid timeframe but, I'll give you something soon.
Thanks .

mpknap

  • Jr. Member
  • **
  • Posts: 92
Re: Sorting and Counting
« Reply #144 on: November 11, 2019, 08:32:44 pm »
I will tell you what it is needed for.
Detections in smartphones are the impact of cosmic ray particles on the phone's camera (photons, electrons, muons). They are mainly single, not regular impacts on the camera matrix.
Scientists, creators of the CREDO project, are looking for the so-called showers, i.e. rain, hitting many phones at one time, or a short time interval.

Scientists are also looking for links between detection frequencies and other phenomena, with an earthquake, with medicine, and solar wind. I also want to put these data on top of each other. maybe someday I will compare with the results of scientists, maybe I will be the first, maybe not, but I will definitely not be bored with such tasks :)
I "play" such searches on my own.

Anyone can join the project. Data are widely available. They mainly operate on Python.

Currently, the biggest problem is the elimination of false detections. They are mainly committed by new users or those who want to be better in the ranking. For this you need specialists in image analysis.

I don't want you to think that I'm using you, what I do is on my own, for my own curiosity.

440bx

  • Hero Member
  • *****
  • Posts: 1258
Re: Sorting and Counting
« Reply #145 on: November 12, 2019, 02:23:06 am »
I don't want you to think that I'm using you, what I do is on my own, for my own curiosity.
Don't worry about that. 

Detections in smartphones are the impact of cosmic ray particles on the phone's camera (photons, electrons, muons).
<snip>
Scientists are also looking for links between detection frequencies and other phenomena,
Experiments like that have the potential to yield surprising results.  Many scientific discoveries resulted from unexpected side effects.  Some that come to mind are microwave ovens, the scientists were constantly having headaches, that's how they figured out that the microwaves were frying their brains (not kidding... though, saying "frying" is a bit of an exaggeration.)

One my favorite unexpected side effects is related to the smartphones you mention.  You may have noticed that cell phones/smart phones no longer have an antenna that has to be pulled out of the phone in order to get reception (old cell phones used to have an antenna.) The reason there _seems_ to be no antenna goes back all the way to Georg Cantor, a mathematician who studied infinities, his work influenced Benoit Mandelbrot, who defined and studied the behavior of fractals, subsequently it was found, apparently mostly by accident, that fractal antennas had a significantly greater wave reception spectrum (due, in part, to their being partially dimensional.)

Without fractals antennas, smartphones would need about half a dozen different antennas protruding out of them in order to capture the spectrum of waves they need to implement all their nifty functions.

I didn't have any time to work of the program today but... I haven't forgotten.
using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

avk

  • Full Member
  • ***
  • Posts: 162
    • my self-education project
Re: Sorting and Counting
« Reply #146 on: November 12, 2019, 05:56:56 am »
@mpknap, did you mean something like that?

mpknap

  • Jr. Member
  • **
  • Posts: 92
Re: Sorting and Counting
« Reply #147 on: November 12, 2019, 07:09:00 am »
@mpknap, did you mean something like that?
that's exactly how I would see it. I have to test. I want to transfer this data to Google map.

avk

  • Full Member
  • ***
  • Posts: 162
    • my self-education project
Re: Sorting and Counting
« Reply #148 on: November 12, 2019, 07:27:12 am »
What is essential:
Your test file contained a BOM, I just deleted it manually. Of course, you can load the data file in another way.
The project uses TVirtualStringTree, you must make sure that you have the virtualtreeview package installed.
And of course, the project uses LGenerics. :)
« Last Edit: November 12, 2019, 07:29:28 am by avk »

440bx

  • Hero Member
  • *****
  • Posts: 1258
Re: Sorting and Counting
« Reply #149 on: November 12, 2019, 04:45:14 pm »
@mpknap

Avk's program is about as good as it gets.  I guess you're all set.

@avk

Nice!  thank you.
using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.