Recent

Author Topic: Gathering anonymous usage data for an application  (Read 395 times)

Gizmo

  • Hero Member
  • *****
  • Posts: 688
Gathering anonymous usage data for an application
« on: July 20, 2019, 09:26:48 am »
Hi

I have a desktop application for Linux, Windows and OSX that after several years now has a healthy download count etc. But other than the download counts I have no way of determining usage patterns. e.g repeat use, what features a user uses it for mostly, what volume of work they did with it etc etc.

I realise there are ethics around data usage but that is a separate matter (making it obvious, getting a tick box agreement etc). What I need to know is the technical methods of achieving it. I know tools like Firefox etc do this but I don't know how it is done. I know for Android and iOS there are ways to do it in a more built-in fashion, but my question relates to application development with Lazarus for the desktop.

So my question is a) is there a package that facilitates this kind of functionality and b) if not, how are others achieving it? I'd like to collect probably a GUID of machine name (to generate uniqueness), frequency of repeat use by the same machine or person, perhaps the IP address, time of use, duration of session use. Any pointers would be appreciated.

Thanks

(PS I did ask this question several years ago but I now cannot find the thread! So I am sorry to ask again)
Lazarus 2.0.4 and fpc 3.0.4 - Linux Mint 19 LTS, Windows 10 64 and Mac OSX Catlina
Useful Page to remember : http://wiki.freepascal.org/Cross_compiling#From_Linux_x64_to_Linux_i386

trev

  • Sr. Member
  • ****
  • Posts: 259
  • Former Delphi 7 and Delphi 10.2 User
Re: Gathering anonymous usage data for an application
« Reply #1 on: July 20, 2019, 10:12:23 am »
I avoid any "phoning home" without consent by simply having a manual "Check web for update" menu item. I also do not collect any data other than the entry in the web server log file which only reveals the date, time, IP address, and file fetched. I used to have one update file per OS, but in the end I settled for just one across all OS (Windows, macOS, FreeBSD, Linux) for simpler admin. I know how many of each OS version are downloaded from the web logs anyway.

Once you start collecting more information there's various different jurisdictional privacy law issues which I'd rather avoid. While many can be avoided by seeking informed consent which requires you to make crystal clear how everything that is being collected will be used, how long the data will be kept etc and optimally making each item allowed/disallowed individually so as not to have users simply disallow everything should they object to any one item.

I might add that uniquely identifying individual machines with a GUID is pretty extreme and would probably lead me to denying any data collection if it was not optional. For example, doing so allows you to track an individual's IP addresses which may no longer be accepted as anonymous data collection.

Just my 2c worth as a retired legal eagle.
o Lazarus v2.1.0 r61775, FPC v3.3.1 r42640, macOS 10.14.6 (with sup update), Xcode 10.3
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.0 (Parallels VM)
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 18.04 (Parallels VM)

lucamar

  • Hero Member
  • *****
  • Posts: 2148
Re: Gathering anonymous usage data for an application
« Reply #2 on: July 20, 2019, 11:13:47 am »
As for the technical part, just use any http client lib (fpweb should suffice) and send the data in a POST request to your server. It's no different than uploading a file or sending a form.

Just make sure your server is secure against attemps to "spoof" or break the data-collection endpoint.

Above all, don't let your application behaviour depend on whether it can send data to you: if it can and the user accepted the "spying", good; if it can't (e.g. cause your server is down), let it keep going without a peep. When an application whines to me: "I can't send 'statistics': your internet is down or something!", that application gets stored in /dev/null
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.4/2.0.6  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

Thaddy

  • Hero Member
  • *****
  • Posts: 9303
Re: Gathering anonymous usage data for an application
« Reply #3 on: July 20, 2019, 03:14:00 pm »
Usually - with consent - just log locally and send the logs on start-up (previous log) or shut-down (current log) if there is an internet connection available. Your usage data lags a bit, but you can easily parse it server side. and for statistics you don't need real-time data. Use https or any certificate based ssl connection. The certificate can be a resource in that case.
« Last Edit: July 20, 2019, 03:18:16 pm by Thaddy »
also related to equus asinus.

Xor-el

  • Sr. Member
  • ****
  • Posts: 384
Re: Gathering anonymous usage data for an application
« Reply #4 on: July 20, 2019, 03:36:12 pm »
Also do note that if you want to log stuffs in a HTTPS server, you need to ship you client binary with OpenSSL binaries for the SSL connections especially since you are targeting cross platform.

Thaddy

  • Hero Member
  • *****
  • Posts: 9303
Re: Gathering anonymous usage data for an application
« Reply #5 on: July 20, 2019, 03:53:41 pm »
And you also need to timestamp the logs to prevent processing duplicates serverside.
also related to equus asinus.