Recent

Author Topic: Project Layout/Management  (Read 7801 times)

guest60499

  • Guest
Project Layout/Management
« on: July 22, 2018, 09:33:13 am »
    I'm trying to create some cross platform units for making use of low level OS functionality. I have not done much, but the structure can be viewed at
https://gitlab.com/LibFreePascal. There may be some checkout issues as I set the submodules to use SSH.

I keep running into problems that make it hard to work.
  • Units are lowercased by default, meaning I have to go back and rename them.
  • Units must have their full namespace in the name, making the directory structure awkward.
  • Managing a complex hierarchy is slow from within Lazarus.
  • Console output is not displayed.

These are not the worst problems, I guess. With the third I may be doing something wrong. The second is typically solved by having a flat hierarchy with the units in it, but I find this hard to navigate from both GUI and command line as the names are long and repetitive. The first has been discussed before but no solution seemed to be forthcoming.

Can anyone describe their workflow for large codebases? How do I deal with the fourth item?[/list]

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Project Layout/Management
« Reply #1 on: July 22, 2018, 10:21:30 am »
Units are lowercased by default, meaning I have to go back and rename them.
There is a "Naming" pane in Environment options.
Select "No automatic renaming" for the Save As - auto rename ... option.

Quote
Units must have their full namespace in the name, making the directory structure awkward.
No, in Pascal a directory structure does not follow the unit names. In Java it does.

Quote
Managing a complex hierarchy is slow from within Lazarus.
Using the Project Inspector? Explain more please.

Quote
Console output is not displayed.
In what situation?
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

wp

  • Hero Member
  • *****
  • Posts: 11858
Re: Project Layout/Management
« Reply #2 on: July 22, 2018, 11:04:22 am »
Units are lowercased by default, meaning I have to go back and rename them.
Since you aim at a cross-platform unit you are calling for trouble if you do this. Lazarus distinguishes between the unit name (which can be any case given by the name specified in the first line) and the unit file name. To avoid issues finding the units in case-sensitive OSs Lazarus offers the possibility to lower-case the filenames automatically. For example, the lcl unit ComCtrls has "unit ComCtrls" in its first line and is saved as "comctrls.pp". Therefore, you can mention it as ComCtrls, comctrls, Comctrls, or even cOmCtRlS in the uses clause, but Lazarus will still find the unit file because it searches for the lower-case filename if it does not find the file in the exact case used in the uses clause.

guest60499

  • Guest
Re: Project Layout/Management
« Reply #3 on: July 23, 2018, 06:22:28 am »
Units are lowercased by default, meaning I have to go back and rename them.
There is a "Naming" pane in Environment options.
Select "No automatic renaming" for the Save As - auto rename ... option.
Thank you.

Quote
Units must have their full namespace in the name, making the directory structure awkward.
No, in Pascal a directory structure does not follow the unit names. In Java it does.
I'm not sure I understand you. I mean that I must name a file "System.Time.Timer.pas" to make its unit be "System.Time.Timer." I do not think the directory structure should necessarily coincide with any concept of namespace, but I am surprised there is no way to keep from having rather long file names, and fake namespace names that are slightly misleading.

Quote
Managing a complex hierarchy is slow from within Lazarus.
Using the Project Inspector? Explain more please.
Yes. I want to have a main library project which runs tests. This project is tracking all of the units in subrepositories that are for coherent sets of libraries, e.g. System.Time. These subrepos can have subrepos of their own, as is the case of System.Time.Timer as there will be a lot of platform specific code.

Now, part of the difficulty I experience has to do with the multiple repositories, so I am trying to not include that. I think this setup best long term in lieu of a large single repo but I may change it back. Part of my justification for this is I actually expect the library to be distributed as a set of unit files - so, obviously, what is the point of including the test code in the repo?

It's my understanding it won't be compiled or placed into the generated executable unless it were used. Perhaps I shouldn't care.

Quote
Console output is not displayed.
In what situation?

I made a normal application but see no setting to enable console output. I'm later going to need to turn it into an FPCUnit application, but for testing I need console output as a GUI is harder to set up and kind of pointless.


Since you aim at a cross-platform unit you are calling for trouble if you do this. Lazarus distinguishes between the unit name (which can be any case given by the name specified in the first line) and the unit file name. To avoid issues finding the units in case-sensitive OSs Lazarus offers the possibility to lower-case the filenames automatically. For example, the lcl unit ComCtrls has "unit ComCtrls" in its first line and is saved as "comctrls.pp". Therefore, you can mention it as ComCtrls, comctrls, Comctrls, or even cOmCtRlS in the uses clause, but Lazarus will still find the unit file because it searches for the lower-case filename if it does not find the file in the exact case used in the uses clause.
I've heard this justification before but it makes no sense. Lazarus could simply read the file name from disk and then lowercase it, as it could with any string as part of a uses clause. I think it does this already?


If anyone has a better project layout I would like to hear it. Just put everything in a flat repository?

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Project Layout/Management
« Reply #4 on: July 23, 2018, 01:23:36 pm »
I'm not sure I understand you. I mean that I must name a file "System.Time.Timer.pas" to make its unit be "System.Time.Timer." I do not think the directory structure should necessarily coincide with any concept of namespace, but I am surprised there is no way to keep from having rather long file names, and fake namespace names that are slightly misleading.
The dots in "System.Time.Timer.pas" are part of the file name. They form a namespace but it has nothing to do with directory structures.
In Java a similar source file Timer.java would be in directory System/Time/. In Java the namespace is built using a directory structure.
I also don't understand you. Why you write about directory structures here? Why would you need fake namespace names that are slightly misleading?

For the project management issues, you could try the project group.

Quote
I made a normal application but see no setting to enable console output. I'm later going to need to turn it into an FPCUnit application, but for testing I need console output as a GUI is harder to set up and kind of pointless.
On Linux I get console output also from a GUI app if I start it from the console.
For Windows there is a setting "Win32 gui application" in project options. It affects the console.

Quote
I've heard this justification before but it makes no sense. Lazarus could simply read the file name from disk and then lowercase it, as it could with any string as part of a uses clause. I think it does this already?
Read what file name from disk? We are talking about case sensitive file systems here. It means almost all file systems except for FAT and NTFS used by Windows.
IIRC the strategy is to first search for the exact UnitName with .pas or .pp appended.
If not found then a lowercase unitname with .pas or .pp appended is searched.
If you match the spelling between UnitName and file name exactly, then everything works.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

guest60499

  • Guest
Re: Project Layout/Management
« Reply #5 on: July 24, 2018, 06:22:22 pm »
I'm not sure I understand you. I mean that I must name a file "System.Time.Timer.pas" to make its unit be "System.Time.Timer." I do not think the directory structure should necessarily coincide with any concept of namespace, but I am surprised there is no way to keep from having rather long file names, and fake namespace names that are slightly misleading.
The dots in "System.Time.Timer.pas" are part of the file name. They form a namespace but it has nothing to do with directory structures.
In Java a similar source file Timer.java would be in directory System/Time/. In Java the namespace is built using a directory structure.
I also don't understand you. Why you write about directory structures here? Why would you need fake namespace names that are slightly misleading?

For the project management issues, you could try the project group.
I know. However it is FPC with the fake namespace names. The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them. As I was told last time I brought this up it is impossible to just type "uses Timer" although Delphi properly implements namespaces and searching. The naming restrictions may still be in place.

This shouldn't require an additional keyword. Should I declare a unit as A.B.C.D and import A.B.C then all units under it would be imported as well. Additionally unit D in namespace A.B.C need not be named A.B.C.D.pas.

As this is not possible with FPC (to my knowledge) it effectively does not have namespaces.

Quote
I made a normal application but see no setting to enable console output. I'm later going to need to turn it into an FPCUnit application, but for testing I need console output as a GUI is harder to set up and kind of pointless.
On Linux I get console output also from a GUI app if I start it from the console.
For Windows there is a setting "Win32 gui application" in project options. It affects the console.
I can start the application in a terminal and get text output but when running from the GUI I see no way to launch the application in a terminal.

Quote
I've heard this justification before but it makes no sense. Lazarus could simply read the file name from disk and then lowercase it, as it could with any string as part of a uses clause. I think it does this already?
Read what file name from disk? We are talking about case sensitive file systems here. It means almost all file systems except for FAT and NTFS used by Windows.
IIRC the strategy is to first search for the exact UnitName with .pas or .pp appended.
If not found then a lowercase unitname with .pas or .pp appended is searched.
If you match the spelling between UnitName and file name exactly, then everything works.
You're not making any sense. If the case of the file on disk is unknown then read the file name from disk and then lowercase it. The file can then be mapped back to the on-disk name by lowercasing the unit name in the uses clause. There is no reason to enforce a lowercase-only policy.

The lowercase-only policy has made a lot of FPC code extremely ugly as developers are too lazy to fix the casing of units that are autocompleted by the IDE. It also requires me to fix names manually to keep a uniform coding style in my projects.

wp

  • Hero Member
  • *****
  • Posts: 11858
Re: Project Layout/Management
« Reply #6 on: July 24, 2018, 06:44:05 pm »
You're not making any sense. If the case of the file on disk is unknown then read the file name from disk and then lowercase it. The file can then be mapped back to the on-disk name by lowercasing the unit name in the uses clause. There is no reason to enforce a lowercase-only policy.
The problem is that FPC does not read the file system in the first place - there can be any files which do not belong to the project. It's the other way round: FPC sees a unit in the uses clause, e.g. Stdctrls. It looks in the project's lpi or package's lpi file for the filename of this unit - let's assume it is the same. Then it seeks the known directories for a file named "Stdctrls.pas" or "Stdctrls.pp". If it does not find anything the search is repeated with the lower-case names "stdctrls.pas" and "stdctrls.pp". Therefore, all units are initially saved in lower-case (if allowed by the setting in the options as mentioned in reply #1).

« Last Edit: July 24, 2018, 06:45:56 pm by wp »

guest60499

  • Guest
Re: Project Layout/Management
« Reply #7 on: July 24, 2018, 08:22:19 pm »
Okay. I guess I would need to create that patch then? The current way is silly and causes problems.

All you are doing is describing what currently is, not why what is is the correct option.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Project Layout/Management
« Reply #8 on: July 24, 2018, 09:23:16 pm »
The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them. As I was told last time I brought this up it is impossible to just type "uses Timer" although Delphi properly implements namespaces and searching. The naming restrictions may still be in place.
Ok, I did not know namespaces in FPC lack features found in Delphi. I must confess I have not used namespaces really, just done some simple tests.
I believe FPC aims to have a Delphi compatible namespace syntax later.

Quote
If the case of the file on disk is unknown then read the file name from disk and then lowercase it.
Do you notice the striking contradiction in your own sentence? How would you read the file name from disk if it is unknown?

Quote
...
There is no reason to enforce a lowercase-only policy.

The lowercase-only policy has made a lot of FPC code extremely ugly as developers are too lazy to fix the casing of units that are autocompleted by the IDE. It also requires me to fix names manually to keep a uniform coding style in my projects.
Lowercase-only is not enforced. There is an option for it as you learned.
Maybe you now confuse lowercase unit names with lowercase file names. FPC libs indeed have some sloppy lowercase unit names although most of them are CamelCase. All unit names in Lazarus sources are CamelCase IIRC.
However lowercase file name is the right way to go. Pascal is a case-insensitive language. A unit name in uses section can be typed as "Stdctrls" or "sTdcTrLs" or with any other casing. If the compiler gives a "unit not found" error then it can be considered buggy.
However if you deside to always use one CamelCase spelling for a unit name then you can use the same CamelCase spelling for the associated file name. It is a very good decision an there is an option supporting it.

Okay. I guess I would need to create that patch then? The current way is silly and causes problems.
What patch?

Quote
All you are doing is describing what currently is, not why what is is the correct option.
I am a little disappointed that you refuse to understand the reasons.
Maybe you still confuse unit names and file names. They are 2 different things. Ideally unit names are CamelCase and file names are lowercase.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Project Layout/Management
« Reply #9 on: July 24, 2018, 09:43:32 pm »
  • Units are lowercased by default, meaning I have to go back and rename them.

Just leave them lowercase.

Quote
  • Units must have their full namespace in the name, making the directory structure awkward.

Directory structure is irrelevant for namespaces.
Quote
  • Managing a complex hierarchy is slow from within Lazarus.

Don't know

Quote
I know. However it is FPC with the fake namespace names. The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them.

It's a namespace, not necessarily a classpath structure. As said namespaces and directories are unrelated.

Quote
As I was told last time I brought this up it is impossible to just type "uses Timer" although Delphi properly implements namespaces and searching. The naming restrictions may still be in place.

Pass -FNSystem.Time or so (but that is trunk only)

guest60499

  • Guest
Re: Project Layout/Management
« Reply #10 on: July 24, 2018, 10:55:20 pm »
The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them. As I was told last time I brought this up it is impossible to just type "uses Timer" although Delphi properly implements namespaces and searching. The naming restrictions may still be in place.
Ok, I did not know namespaces in FPC lack features found in Delphi. I must confess I have not used namespaces really, just done some simple tests.
I believe FPC aims to have a Delphi compatible namespace syntax later.
This is good.

Quote
If the case of the file on disk is unknown then read the file name from disk and then lowercase it.
Do you notice the striking contradiction in your own sentence? How would you read the file name from disk if it is unknown?
Enumerate files in a directory. All file systems must support this. How else would the operating system know their names?

Quote
...
There is no reason to enforce a lowercase-only policy.

The lowercase-only policy has made a lot of FPC code extremely ugly as developers are too lazy to fix the casing of units that are autocompleted by the IDE. It also requires me to fix names manually to keep a uniform coding style in my projects.
Lowercase-only is not enforced. There is an option for it as you learned.
Maybe you now confuse lowercase unit names with lowercase file names. FPC libs indeed have some sloppy lowercase unit names although most of them are CamelCase. All unit names in Lazarus sources are CamelCase IIRC.
However lowercase file name is the right way to go. Pascal is a case-insensitive language. A unit name in uses section can be typed as "Stdctrls" or "sTdcTrLs" or with any other casing. If the compiler gives a "unit not found" error then it can be considered buggy.
However if you deside to always use one CamelCase spelling for a unit name then you can use the same CamelCase spelling for the associated file name. It is a very good decision an there is an option supporting it.
I think I have been missing this. If the unit name is cased properly the autocompletion will pick up on it?

Properly implementing case insensitivity does not force the storage of a specific case. It converts the cases to something uniform in memory. This is how Windows works - it's not like all of your files are lowercase, are they? Yet you can still access them regardless of the case you give later.

Okay. I guess I would need to create that patch then? The current way is silly and causes problems.
What patch?
One I would need to find time to write, apparently.

Quote
All you are doing is describing what currently is, not why what is is the correct option.
I am a little disappointed that you refuse to understand the reasons.
Maybe you still confuse unit names and file names. They are 2 different things. Ideally unit names are CamelCase and file names are lowercase.
The problem is there is no point. Storing the filenames as lowercase is unnecessary.



  • Units are lowercased by default, meaning I have to go back and rename them.

Just leave them lowercase.
But then you end up with unreadable filenames like synedittextdyntabexpander.pas. If I could work with things only via the editor maybe I would agree.

Quote
  • Units must have their full namespace in the name, making the directory structure awkward.

Directory structure is irrelevant for namespaces.
Directories still have to exist. Whether I put all files in one flat hierarchy or try to butcher in some nested directories there is still some kind of structure, even if the compiler ignores it.

Quote
I know. However it is FPC with the fake namespace names. The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them.

It's a namespace, not necessarily a classpath structure. As said namespaces and directories are unrelated.
You're missing the point of having a separate word. C has a flat namespace. C++ has a namespace keyword that can introduce nesting.

So no, it is not a namespace. The top level, flat namespace has a member A.B.C.D, but that namespace exists by default.

Quote
As I was told last time I brought this up it is impossible to just type "uses Timer" although Delphi properly implements namespaces and searching. The naming restrictions may still be in place.

Pass -FNSystem.Time or so (but that is trunk only)
This is good but not necessarily great. Are there any plans to add e.g. "uses System.Time.*"? Perhaps there is a better syntax?

I am glad it exists though, I will install trunk. If this feature is in trunk then namespaces are maybe halfway supported.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Project Layout/Management
« Reply #11 on: July 25, 2018, 10:04:17 am »
Just leave them lowercase.
But then you end up with unreadable filenames like synedittextdyntabexpander.pas. If I could work with things only via the editor maybe I would agree.

Or use Windows only. But in general lowercase is recommended, and optimizes searching.

Directory structure is irrelevant for namespaces.
Directories still have to exist. Whether I put all files in one flat hierarchy or try to butcher in some nested directories there is still some kind of structure, even if the compiler ignores it.


You can use directories all you like, but there is no correlation between namespaces and directories.

Quote from: guest link=topic=41976.msg292518#msg292518  date=1532465720
I know. However it is FPC with the fake namespace names. The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them.


Fake namespace is an undefined term. These are namespaces, and the toplevel namespace is an unit. It is not Java that doesn't know units.

Quote from: marcov link=topic=41976.msg292515#msg292515  date=1532461412
It's a namespace, not necessarily a classpath structure.

This is crucial to remember, and also that the core namespace is the unit. Hierarchy is only meant to disambiguate units.

Quote
You're missing the point of having a separate word. C has a flat namespace. C++ has a namespace keyword that can introduce nesting.

C++ has no units, which are the corner stone of namespaces on Pascal, even before hierarchical extension.

Similarly, java/C# chose mapping hierarchy to a filesystem structure and have a one class per file limitation that Pascal doesn't suffer

Different requirements, different solutions.

Quote
So no, it is not a namespace. The top level, flat namespace has a member A.B.C.D, but that namespace exists by default.

It is a namespace. It maybe is not a namespace as you are used to, but it still is a namespace. Identifiers can be used multiple times in different places in the hierarchy.

Pascal was always quite free with that (because of the unit system), only the unit names itself couldn't be duplicated. The hierarchical system is a workaround for that.

One can argue if it is elegant (I think it is medium quality at best), but it works and it solves some of the old problems at the expensive of renaming the units to a dotted form.

Quote
Pass -FNSystem.Time or so (but that is trunk only)
This is good but not necessarily great. Are there any plans to add e.g. "uses System.Time.*"? Perhaps there is a better syntax?

There are no plans like that.

The unit and the unit system remain the core of namespace control, and the hierarchy is only a minor extension.  The core meaning of hierarchical namespaces in Delphi/Pascal are only meant to disambiguate packages that reuse unit identifiers.

The deepest level of the namespace is always an unit, and an unit can already contain multiple classes, so wildcards a la Java to include multiple classes at the deepest level are not needed. Also consider that Pascal allows  type aliases to reexport identifiers (so system.timer.a  can reexport a type from system.timer.b, making it unnecessary to USES system.timer.b implicitely)

To me it sounds you are trying a one class/symbol per unit system. The system is not meant for that, and you are trying to imitate a foreign design concept on Delphi/Pascal.

It is smarter to design your package to Pascal's rules.

Quote
I am glad it exists though, I will install trunk. If this feature is in trunk then namespaces are maybe halfway supported.

Keep in mind what the namespace system is used for, and avoid overuse. Forget what you know about namespaces from other languages, those serve a different purpose, and are because they don't have a proper unit system.

Work with the unit system instead against it.

« Last Edit: July 25, 2018, 11:18:40 am by marcov »

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: Project Layout/Management
« Reply #12 on: July 25, 2018, 11:03:37 am »
If you really want to publish your sourcecode, you'd better follow Marco's advice and explanation.
Specialize a type, not a var.

guest60499

  • Guest
Re: Project Layout/Management
« Reply #13 on: July 27, 2018, 03:23:53 am »
FreePascal does not have hierarchical namespaces. It has a single flat namespace which can have members with periods in the name.

There is no reason to lowercase file names. You can still find a file in a search path if you do not know how it is cased: you list all files, lowercase the names in memory, and compare them to your target string after lowercasing it.

Is there something about this either of you do not understand?

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Project Layout/Management
« Reply #14 on: July 27, 2018, 07:31:53 am »
I'm not sure I understand you. I mean that I must name a file "System.Time.Timer.pas" to make its unit be "System.Time.Timer." I do not think the directory structure should necessarily coincide with any concept of namespace, but I am surprised there is no way to keep from having rather long file names, and fake namespace names that are slightly misleading.
The dots in "System.Time.Timer.pas" are part of the file name. They form a namespace but it has nothing to do with directory structures.
In Java a similar source file Timer.java would be in directory System/Time/. In Java the namespace is built using a directory structure.
I also don't understand you. Why you write about directory structures here? Why would you need fake namespace names that are slightly misleading?

For the project management issues, you could try the project group.
I know. However it is FPC with the fake namespace names. The file naming system as it exists now is not really a "namespace." Everything is still a flat hierarchy with names that have periods in them. As I was told last time I brought this up it is impossible to just type "uses Timer" although Delphi properly implements namespaces and searching. The naming restrictions may still be in place.

Citing Delphi's "Namespace" documentation is misleading, cause it's for Delphi.Net even though it's not explicitely mentioned there. You can see this in Delphi Tokyo's documentation where at the end it mentions "assembly" which is .Net only. What non-.Net Delphi implements and FPC supports as well are so called Unit Scope Names and as Marco said trunk even supports the option to pass unit scope name prefixes using the -FN<x> parameter.

There is no reason to lowercase file names. You can still find a file in a search path if you do not know how it is cased: you list all files, lowercase the names in memory, and compare them to your target string after lowercasing it.

Yes, there is a reason to do it the way FPC does it currently: case sensitive file systems can have files with the same lower case name in the same location. E.g. there could be a StdCtrls and a stdctrls in the same directory. By lower casing the file name it's not clear (for the user) which file is picked especially as the order is depending on the OS. By searching the file in a specific casing order (as-is, lower case, upper case) this is handled in a documented and cross platform way.
Please note that this can also happen on Windows. E.g. if you use a case sensitive file system (like Ext for which there are drivers) or with network file systems.

 

TinyPortal © 2005-2018