Recent

Author Topic: Folder locations on linux/ubuntu  (Read 6703 times)

qdos

  • Full Member
  • ***
  • Posts: 112
Folder locations on linux/ubuntu
« on: September 27, 2014, 01:42:36 am »
Updated: This should have been posted to the operative-system/Linux section. I apologize for posting in in general.

This is something I asked about a couple of years back, which caused a great deal of debate. I published code for OSX and Windows on my website, but I never got around to adding linux support publicly, but gave it to a member of the laz team. Sadly I dont recall who :(

The great argument was, as now, that it could not be done easily. But I did present code that worked fine in 90% of cases, with fallback mechanisms in case no clear-cut API could be used.

So, second time around - how exactly do i get the special folders under linux?

e.g:
Documents
Pictures
Movies
Music
User Folder

I have included what I came up with back then, but I was under the impression that my code was included (in some form) in the RTL? Please help me out with the function names. Are there standard functions for getting "special folders" on all operative systems?

Well, here is what I wrote a couple of years back. The StrPasW stuff should most likely be deleted and replaced with standard functions -- but they work and get the job done.

Code: [Select]
unit jlshellpaths;

  {$mode objfpc}{$H+}

  interface

  uses
  sysutils, classes;

  (* Special folders *)
  Function  JL_Shell_GetCurrentUserName:WideString;
  Function  JL_Shell_GetHomeFolder:WideString;
  Function  JL_Shell_GetMyDocumentsFolder:WideString;
  Function  JL_Shell_GetMyPicturesFolder:WideString;
  Function  JL_Shell_GetMyMusicFolder:WideString;
  Function  JL_Shell_GetMyVideosFolder:WideString;
  Function  JL_Shell_GetTempFolder:WideString;

  Function  StrPasW(Const Data):WideString;overload;
  Function  StrPasW(Data:system.PWideChar):WideString;overload;
  Procedure FillCharW(Const Data;aLen:Integer;const Value:system.WideChar);

  implementation

  {$HINTS OFF}

  {$IFDEF WINDOWS}
  uses windows, ShlObj;

  function GetSpecialFolder(const CSIDL:integer):WideString;
  var
    FBuffer:  Packed array [1..MAX_PATH] of WideChar;
  begin
    {$WARNINGS OFF}
    FillCharW(FBuffer[1],MAX_PATH,#0);
    If (SHGetSpecialFolderPathW(0, @FBuffer[1], CSIDL, false)) then
    result:=StrPasW(FBuffer[1]) else
    result:='';
    {$WARNINGS ON}
  end;
  {$ENDIF}

  {$IFDEF DARWIN}
  uses Macosall;

  Const
  CTL_BUFFERLENGTH = 1024;

  {$DEFINE SHELL_SUPPORT_FALLBACK}

  Function GetSpecialFolder(Const sysFolderType:Cardinal):WideString;
  var
    FBuffer: PChar;
    FRef: FSRef;
  Begin
    result:='';

    (* Allocate Temporary buffer *)
    try
      FBuffer:=Allocmem(CTL_BUFFERLENGTH);
    except
      (* Not much we can do here. If we cant even allocate
         a memory buffer then Exit is the only viable option *)
      on exception do
      exit;
    end;

    try
      (* Flush our buffers *)
      Fillchar(FBuffer^,CTL_BUFFERLENGTH,#0);
      Fillchar(FRef,Sizeof(FRef),#0);

      (* Attempt to get a hold of the system folder in question *)
      if FSFIndFolder(kUserDomain,sysFolderType,false,FRef)=0 then
      Begin
        (* Convert from pathreference to UTF-8 string *)
        if FSRefMakePath(FRef,FBuffer,1024)=0 then
        result:=UTF8ToAnsi(StrPas(FBuffer)); //Should be DecodeUTF8 if not used visually
      end;
    finally
      (* Release our buffer *)
      FreeMem(FBuffer);
    end;
  end;
  {$ENDIF}

  (* METHOD:      FillCharW ()
     DESCRIPTION: Fill a region of memory using Widechar (Word = 2 bytes).
     NOTES:       Length parameter defines the number of "words" to fill,
                  not bytes or chars. To get the bytesize, multiply by 2.
  *)
  Procedure FillCharW(Const Data;aLen:Integer;const Value:system.WideChar);
  var
    FAddr: PWideChar;
  Begin
    if aLen>0 then
    Begin
      FAddr:=@Data;
      if FAddr<>NIL then
      Begin
        try
          While aLen>0 do
          Begin
            FAddr^:=Value;
            dec(aLen);
            inc(FAddr);
          end;
        except
          on exception do
          Raise;
        end;
      end;
    end;
  end;

  (* METHOD:      StrPasW ()
     DESCRIPTION: Extract a NULL terminated widestring from memory buffer.
     NOTES:       See ordinary StrPas() documentation
  *)
  Function StrPasW(Const Data):WideString;
  var
    FAddr: PWideChar;
  Begin
    result:='';
    FAddr:=@Data;
    if FAddr<>NIl then
    result:=StrPasW(FAddr);
  end;

  (* METHOD:      StrPasW ()
     DESCRIPTION: Extract a NULL terminated widestring from memory buffer.
     NOTES:       See ordinary StrPas() documentation
  *)
  Function StrPasW(Data:system.PWideChar):WideString;
  Begin
    result:='';
    If Data<>NIl then
    Begin
      try
        if Data^<>#0000 then
        Begin
          repeat
            Result:=Result + Data^;
            inc(Data);
          Until Data^=#0000;
        end;
      except
        on exception do
        Raise;
      end;
    end;
  end;

  Function JL_Shell_GetTempFolder:WideString;
  {$IFDEF WINDOWS}
  var
    FBuffer: Packed array[1..MAX_PATH] of WideChar;
    FLen:Integer;
  {$ENDIF}
  Begin
    {$IFDEF WINDOWS}
    FLen:=GetTempPathW(MAX_PATH, @FBuffer);
    If FLen>0 then
    result:=StrPasW(FBuffer[1]) else
    result:=sysutils.GetEnvironmentVariable('TEMP');
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=GetSpecialFolder(kUserSpecificTmpFolderType);
    {$IFDEF SHELL_SUPPORT_FALLBACK}
    If length(result)<1 then
    result:='/tmp';
    {$ENDIF}
    {$ENDIF}
  end;

  Function JL_SHELL_GetCurrentUserName:WideString;
  {$IFDEF WINDOWS}
  Var
    FBuffer: Packed array[1..MAX_PATH] of WideChar;
    UserNameLen : Dword;
  {$ENDIF}
  Begin
    {$IFDEF WINDOWS}
    UserNameLen:=MAX_PATH;
    If GetUserNameW(@FBuffer[1], UserNameLen) Then
    Result := StrPasW(FBuffer[1]) Else
    Result := sysutils.GetEnvironmentVariable('USERNAME');
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=sysutils.GetEnvironmentVariable('USER');
    {$ENDIF}
  end;

  Function JL_SHELL_GetHomeFolder:WideString;
  Begin
    {$IFDEF WINDOWS}
    result:=GetSpecialFolder(CSIDL_PERSONAL);
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=GetSpecialFolder(kCurrentUserFolderType);
    {$IFDEF SHELL_SUPPORT_FALLBACK}
    If length(result)<1 then
    result:=sysutils.GetEnvironmentVariable('HOME');
    {$ENDIF}
    {$ENDIF}
  end;

  Function JL_Shell_GetMyDocumentsFolder:WideString;
  Begin
    {$IFDEF WINDOWS}
    result:=JL_SHELL_GetHomeFolder;
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=GetSpecialFolder(kDocumentsFolderType);
    {$IFDEF SHELL_SUPPORT_FALLBACK}
    if length(result)<1 then
    result:=IncludeTrailingPathDelimiter(JL_SHELL_GetHomeFolder) + 'Documents';
    {$ENDIF}
    {$ENDIF}
  end;

  Function JL_Shell_GetMyVideosFolder:WideString;
  Begin
    {$IFDEF WINDOWS}
    result:=GetSpecialFolder(CSIDL_MYVIDEO);
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=GetSpecialFolder(kMovieDocumentsFolderType);
    {$IFDEF SHELL_SUPPORT_FALLBACK}
    if length(Result)<1 then
    result:=IncludeTrailingPathDelimiter(JL_SHELL_GetHomeFolder) + 'Movies';
    {$ENDIF}
    {$ENDIF}
  end;

  Function JL_Shell_GetMyPicturesFolder:WideString;
  Begin
    {$IFDEF WINDOWS}
    result:=GetSpecialFolder(CSIDL_MYPICTURES);
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=GetSpecialFolder(kPictureDocumentsFolderType);
    {$IFDEF SHELL_SUPPORT_FALLBACK}
    if length(result)<1 then
    result:=IncludeTrailingPathDelimiter(JL_SHELL_GetHomeFolder) + 'Pictures';
    {$ENDIF}
    {$ENDIF}
  end;

  Function JL_Shell_GetMyMusicFolder:WideString;
  Begin
    {$IFDEF WINDOWS}
    result:=GetSpecialFolder(CSIDL_MYMUSIC);
    {$ENDIF}

    {$IFDEF DARWIN}
    result:=GetSpecialFolder(kMusicDocumentsFolderType);
    {$IFDEF SHELL_SUPPORT_FALLBACK}
    if length(result)<1 then
    result:=IncludeTrailingPathDelimiter(JL_SHELL_GetHomeFolder) + 'Music';
    {$ENDIF}
    {$ENDIF}
  end;

  end.
« Last Edit: October 02, 2014, 05:28:23 am by qdos »

eric

  • Sr. Member
  • ****
  • Posts: 267
Re: Folder locations on linux/ubuntu
« Reply #1 on: September 27, 2014, 08:41:14 am »
One way to get the user's home directory in Linux is:

HomePath := GetEnvironmentVariable('HOME');

This will nearly always be /home/<username>

The other directories aren't standardised, but they will normally be subdirectories under the user's home directory, e.g.

/home/<username>/Documents
/home/<username>/Pictures
... etc.

Not all of the directories you mention will necessarily exist at all.

You can't make any assumptions here, so you'll have to find other ways of handling the issue.

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Folder locations on linux/ubuntu
« Reply #2 on: September 27, 2014, 09:27:37 am »
I don't think there is 100% fool-proof cross-platform answer to all your needs yet (happy to be proved wrong).
But the fpc RTL provides at least GetAppConfigDir, GetAppConfigFile and GetUserDir, and the Lazarus FileUtils unit provides UTF8 implementations for these. All these work for Linux, Mac, BSD ...

Rails

  • Guest
Re: Folder locations on linux/ubuntu
« Reply #3 on: September 27, 2014, 02:51:17 pm »
Take a look at the Filesystem Hierarchy Standard (FHS) at  http://www.linuxfoundation.org/collaborate/workgroups/lsb/fhs. Most distributions adhere to this to some degree. Note that there are no current standards for subdirectories under /home/user.

qdos

  • Full Member
  • ***
  • Posts: 112
Re: Folder locations on linux/ubuntu
« Reply #4 on: October 02, 2014, 05:03:00 am »
Take a look at the Filesystem Hierarchy Standard (FHS) at  http://www.linuxfoundation.org/collaborate/workgroups/lsb/fhs. Most distributions adhere to this to some degree. Note that there are no current standards for subdirectories under /home/user.

Yes I know. Thats why I spent a few days writing linux code for it, and giving it to a member here for inclusion. I never published it, and i cant find the finalized units anymore (this was a couple of years back after all). I tested the code on Ubuntu, Fedora and what I recall was Mint. Sadly my mac which hosted the code died a couple of years ago, so the library is lost.

A bit sad to see that it was never adopted. I even wrote a driver system for it, which allowed you to enumerate all disks/IO devices on all systems, and also extract direct size info, block and cluster size etc... The last i did with that code was add block-level read/write. Although the windows version would require a custom driver to access files in use (optional, the high level interface was more important).

I was going to expand it with a more "unified file-access api", a proposal for the lazarus crew for adoption. Using it could be made very simple:

procedure openFile;
var
  mFile: TFilesysObject;
Begin
  if FileAPI.Documents.getFileObj("myDoc.txt") then
  Begin
    try
      with mFile.Stream do
      Begin
        try
          //Do something
        finally
          free;
        end;
      end;
    finally
      mFile.free;
    end;
  end;
end;

* FileAPI would be automatically created on app-start
* Several mapped functions, like "documents" above which is actually also a filesystem object
* Full implementation of ByteRage buffers (see my google code repo)

Working with files across all platforms would have been uniform, fast and easy to use. Memory mapped files would be reduced to a "one liner" and doing complex operations, like insering 1 gigabyte into the middle of a 20gigabyte file, likewise a single command (google code -> byterage). Byterage buffers are excellent replacements for streams, and perfect for fast and platform independent data manipulation.

Well, if I re-write it who do I send it to? Who is actually in the RTL design team for proposals?

Also, found some info. Remember doing this as a fallback is ENV("home") failed. You can read out the home-folder from the user account database:

http://pubs.opengroup.org/onlinepubs/007904975/functions/getpwuid.html
« Last Edit: October 02, 2014, 05:14:49 am by qdos »

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Re: Folder locations on linux/ubuntu
« Reply #5 on: October 02, 2014, 10:04:25 am »
Well, if I re-write it who do I send it to? Who is actually in the RTL design team for proposals?
See the freepascal.org site, development page IIRC.
Summary: please post on the fpc-devel mailing list to discuss this kind of thing.
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Folder locations on linux/ubuntu
« Reply #6 on: October 02, 2014, 10:48:38 am »
Full implementation of ByteRage buffers (see my google code repo)

well I was curious enough to take a quick look on your ByteRage page, what I do not understand is what is so different from the streams, you page actually describes the tstream based mechanism to the letter and there was nothing there that would interest me to drop tstream in favor of your ragebyte library so what it is so different? given that fact that there are today a number of streams descentands including memory streams, resource streams, tcp streams and even custom format streams that represent a single file in a file system in a file stream. So what do you really offer more than the streams other than the insert functionality which can be implemented on top of streams as well?

Just curious though so fill free to ignore me if you don't have the time or will to answer. Better spend your free ime convincing the rtl team for a better design than me.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Folder locations on linux/ubuntu
« Reply #7 on: October 02, 2014, 10:52:11 am »
I have included what I came up with back then, but I was under the impression that my code was included (in some form) in the RTL? Please help me out with the function names. Are there standard functions for getting "special folders" on all operative systems?

Well before moving it to a general include, that is the first thing to research. At least the Tier 1 platforms (various *nixes including OS X and Windows) should be reasonably supported.

And afaik *nix doesn't really provide these things via api calls, and might also not have the localization features Windows has (from Vista on, the English names are always available and the internationalized ones are symlinks).

I think the desktop package (Gnome/KDE/whatever) generally makes assumptions about these, and there is a package from opendesktop called "xdg" that tries to harmonize those a bit.  (xdg-user - dir DOWNLOAD should return download).

There are several issues to deal with:
  • Requires executing programs, and this should be done lazily (only once for each directory on first query) , and it shouldn't increase minimal requirements. A dependence on e.g. TProcess rules out sysutils as location, requiring a separate unit in a package
  • Only desktop systems (and users initialized for desktop use) support this. Reasonable defaults and propagating back errors if impossible.  This might mean that using calls suitable for windows might not be the best form, since generally it will probably have the if getspecialdirfunction(var s:string) then .. form.
  • Not all dirs are available on all systems. Generally a minimal set should be available portable, but all should be available via an OS specific interface (since we live in a real world)
  • Which encoding to use ? UTF8 is awkward on Windows, UTF16 is alien on *nix

I checked my GUI less Debian install, and it has neither directories nor xdg-user by default installed.
Quote
Well, here is what I wrote a couple of years back. The StrPasW stuff should most likely be deleted and replaced with standard functions -- but they work and get the job done.

IMHO it tries to expand the Windows minorly to OS X without really taking a step back and reflecting how a truely crossplatform solution would look like. This includes the usage of two byte string types (which on Linux requires special libraries and units, undesirable)

But that doesn't mean it is useless. The good solutions are iterated several times, and usually base on already validated OS specific interfaces.

« Last Edit: October 02, 2014, 10:54:28 am by marcov »

bylaardt

  • Sr. Member
  • ****
  • Posts: 309
Re: Folder locations on linux/ubuntu
« Reply #8 on: October 02, 2014, 03:58:08 pm »
the fast way to get de "special" directory names IMHO, is get the names from this file:/$home/.config/user-dirs.dirs.
as marcov says, this file is create from the xdg-user-dirs program. they are required in most of distros.
if this file not exists, then xdg-user-dirs is not instaled, and those directories aren't defines.

if this file not exists, then xdg-user-dirs is not installed, and those directories aren't defined. "regula stabilis except est"


qdos

  • Full Member
  • ***
  • Posts: 112
Re: Folder locations on linux/ubuntu
« Reply #9 on: October 02, 2014, 04:41:53 pm »
well I was curious enough to take a quick look on your ByteRage page, what I do not understand is what is so different from the streams, you page actually describes the tstream based mechanism to the letter

First of all I never proposed that we kick out streams in favour of my buffers, far from it, they complement each other.

If you take a look at the code you will find that it has several advantages over vanilla streams:

** Data view mechanism: You can view/read/write to the buffers using various aux classes (same as C#, C++ and JavaScript). So the same buffer can be accessed as byte, word, longword and even bits. Each "view" targets the same buffer, but regards the buffer content differently. Writing your own view for custom datatypes is simple and elegant.

** Capabilities awareness: Buffers provide info on their caps (e.g: write allowed, read not allowed, scale allowed)

** Faster than streams, quite substancially since seek has no place. Int64 based offsets all the way

** Provides better functions and operations:
  --Insert (which really means insert, not overwrite)
  --Remove (which again removes X number of bytes from anywhere in a buffer)
  --ZLib support, pack a buffer on the fly

** Easy to expand. A memory mapped file version is trivial. A http buffer likewise so (loading data from the web using the http part + offset API).
 
** Record classes: binary persistence simplified

** Owned and foreign buffers, simplify access to buffers not allocated by you

For instance, pixelrage, which is a low-level graphics library - uses buffers to deal with all in-memory operations. It doesnt even need a device context, it does everything in memory.
But by altering the buffer type to disk, operations can be done there (all operations, like line, circle, clip, fill, gradient etc. etc). This is quite handy for services which deals with en-mass manipulation (I wrote a huge service for Hydro, a norwegian oil company, and with their security context you couldnt not even create a device context. I was forced to improvise and created pixelrage + byterage)

NOTE: Byterage should not be considered "complete", but rather providing rock solid routines as building blocks for larger systems.
It also ships with a TStream adapter, so you can mix and match buffers with streams.

I actually used the buffer classes to write my own DB engine. And since it uses buffers it works everywhere buffers can go (same benefits as streams, but faster and more flexible).

What is easiest to write:

mBuffer.Pack;

or.. (pseudo):

mPacker:=TCompressionStream.Create(mTarget);
try
  mPacker.read(mSource,mSource.Size);
finally
  mPacker.free;
end;

As always -- you can chose to ignore subtle differences and advantages of a fast and light buffer system over streams, thats up to you. No one is forcing anyone.
To be frank, the more options FPC/Laz has the better.
« Last Edit: October 02, 2014, 04:44:17 pm by qdos »

qdos

  • Full Member
  • ***
  • Posts: 112
Re: Folder locations on linux/ubuntu
« Reply #10 on: October 02, 2014, 04:50:45 pm »

Well before moving it to a general include, that is the first thing to research. At least the Tier 1 platforms (various *nixes including OS X and Windows) should be reasonably supported.

And afaik *nix doesn't really provide these things via api calls, and might also not have the localization features Windows has (from Vista on, the English names are always available and the internationalized ones are symlinks).

From what I remember from the last time i dabbled with this, Unix and Linux also have translation services. So if the english name for a folder is "Documents", it will show up in Norwegian as "Dokumenter", but the *REAL* folder on disk will be the english version. This led to quite some confusion and debate until i posted the pictures of it here.

OS X provides API calls for it (see code inclued in this post), but you are right that this may not be the case for other distros.

But translation services taken into account, i think its fair to say that locating documents, pictures, movies etc.. is a matter of starting with the user-folder, no matter what the locale is set to -- the real folders will have english names (unless its a very exotic form of linux perhaps)

 

TinyPortal © 2005-2018