Recent

Author Topic: Issues publishing a project (No subdirectories incl., no non-Pas. files incl.?)  (Read 6975 times)

GetMem

  • Hero Member
  • *****
  • Posts: 3256
@wp
Here is a short description:
1. Current status in Lazarus trunk
 a.) Project files are completely ignored. Basically the IDE does a recursive search in the main project directory, files are published according to the include/exclude filters. If by chance a project file extension mach one of the include filter the project file gets published. This is a complete nonsense in my opinion. Every single project file must be published regardless of the filter settings, otherwise the project won't build. Filters are useful for other non-project files like: ini, image, text etc... files. 
 b.) Subfolders are ignored
 c.) Other project directories(not subfolders) are ignored

2. My latest patch fix the above mentioned issues, the only drawback is that every single file is copied in one common "published" folder. By the way, it makes sense if you think about it. Lets say you have two project folders d:\Folder1\ and d:\Folder2\(I would never keep my files this way, but apparently is not so uncommon), and you wish to publish it to c:\published\. It makes no sense to keep the folder structure intact since the relative paths are changed anyways and must be updated in project.lpi. As far as I can see this is the main disagreement between different users. Perhaps I should add an option: "Keep directory structure intact", if checked files would be copied in separate folders, if not a everything goes to a single folder.


« Last Edit: August 10, 2018, 04:12:06 pm by GetMem »

BrainChemistry

  • Jr. Member
  • **
  • Posts: 65
@GetMem

Feedback on the version in #28:

A) For test.zip from #16
- I tried three different cases:
1) Include filter checked, exclude filter not
2) Include filter and exclude filter checked
3) Include filter checked but with empty edit field, exclude filter not checked

Alle three trials gave the same faulty result. The files of the main dir. have been copied as shown in the attached zip to the folder sub1 and sub2. The incl. und excl. filters seem to be completly ignored.

B) For my own project:
Some strange different behaviour was examined. In this case, also a unit is in a sub directory. Instead of copying all files from main dir. into this sub dir. (as for the test case above), in this case I didn't get a sub dir. created at all, but the unit from the sub dir. is copied to the main unit in the published project. - This could be the the desired behaviour, still the include/exclude filters are completly ignored.

As for the discussion what the goal for publishing projects is:

I guess there are two different ideas on how to use the published projects.
1) As tudi_x stated several times, he wants to use the published project feature to just quickly transfer it to somebody, so that somebody can easily compile the project. No folder structure in the way, just the plain files and the other one will figure out how to do it with his machine.
2) For me (and wp?), the goal is more like to have the project ready to be published in like a github account. So the folder structure would be highly beneficial, because this is the way how the published project folder could directly be used as github source folder, too. 

Technically, there is only one problem, that is, the case mentioned several times, if there are files in folders which are no sub dirs. of the main dir. of the project. A simple solution to this would be to copy all files which reside in a folder which is no sub dir. of the main dir. into the main folder (hence ignoring folder structure for these files), still keep the folder structure from the main dir. on intact.

I fear, the only way to have both ways possible is an option as GetMem suggested.




JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3554
  • I like bugs.
I did a major revamp for the publish feature and dialog in r58785. Please test.
Fundamental changes were needed. Then respecting the existing code too much would lead to ugly hacks. So, fundamental changes I did.
Now the main function is to copy the project / package member files. Additional filters can be used to copy extra files but hopefully they will not be needed much. Form files (.lfm) and project main resource file (.res) are copied automatically even if they are not part of the project / package. That should cover most files then.

There is only one enable-CheckBox for the filters. The idea is that an empty filter means "not used" or "no effect". I realized I did not test that one but now I am too lazy and tired to test more...

I scratched my head for a while to get the directory structures right. A typical structure is like:
Code: [Select]
SoftWare
  ﹂ MyProject
        -- ReadMe.txt
      ﹂ MainDir
           -- Project1.lpr
           -- Project1.lpi
      ﹂ AnotherDir
           -- SomeSrc.pas
From Lazarus POV the project directory is "MainDir" because it contains the projects main source and info files. Logically it is directory "MyProject" because all sources are collected under it in subdirectories.
So, I built a logic to traverse project/package files and move the effective top directory "up" always when a relative '../' reference was found in front of a file name. It works with infinite number of directory levels.
Then the files are copied using the new top directory as a reference point. Perfect, even if I say it myself!  :)

The "Command after" was used only for compressing the result but it was not very cross-platform.
I removed it and added a CheckBox for compressing finally the directory. It is not implemented yet. May I ask somebody to please implement it and provide a patch, or commit it directly if having SVN rights. There is a Zipper in FPC's libs and example code somewhere. Should be easy.
There are also other ToDo items in the code. Please take a look.

I removed the CheckBox for binary files. The compiled project executable should never be copied. This feature is about publishing sources. Are there other binaries to consider?

I also removed 2 CheckBoxes for projects which made no sense in my opinion. They only added confusion.

What do you think? Is the feature set right? Was it OK to remove "Command after" and others?
Actual bugs can be reported either here or in the report which I keep open for a while.
  https://bugs.freepascal.org/view.php?id=34102
Anybody who is interested in this feature must get the SVN trunk version of Lazarus. There is no excuse especially because it is so easy to do.
« Last Edit: August 28, 2018, 09:04:26 am by JuhaManninen »

GetMem

  • Hero Member
  • *****
  • Posts: 3256
@Juha
In my opinion is much better then before. Thank you for putting all the pieces together. One feature is still missing and although I don't need it, other people explicitly asked in this thread many times before. Please take a look at the following directory structure:
Quote
    SoftWare
      ﹂ MyProject
            -- ReadMe.txt
          ﹂ MainDir
               -- Project1.lpr
               -- Project1.lpi
          ﹂ AnotherDir
               -- SomeSrc.pas
      ﹂ MyOtherProjectDir
          -- SomeOtherSrc.pas

MyOtherProjectDir is not exported. The question is where MyOtherProjectDir folder should go? Inside the publish folder, at the same level as Maindir.

PS: I will do the zipping part at a later time.


JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3554
  • I like bugs.
MyOtherProjectDir is not exported.
Did you test it? It should be exported if SomeOtherSrc.pas is part of the project and its path is relative.
Actually the file's path can be saved anyhow. In procedure AdjustTopDir I create a relative path :
  RelPath := ExtractRelativePath(FTopDir, AFilename);
and check for '../' in front of it.

Quote
The question is where MyOtherProjectDir folder should go? Inside the publish folder, at the same level as Maindir.
Why at the same level? Now the directory structure is retained.
« Last Edit: August 28, 2018, 09:09:52 am by JuhaManninen »

GetMem

  • Hero Member
  • *****
  • Posts: 3256
Quote
Did you test it?
Yes. If everything is inside a big directory the publish feature works correctly. The paths are retained, the project files are exported. I was considering more extreme cases like this:
Folder1: d:\ProjectFolder1\
Folder2: d:\ProjectFolder2\
Alternatively you can add subfolders too. If you wish to publish to d:\published\ for example, the feature will fail. Even in theory I don't know how the publish should work in this particular case, where the two folder should be copied?


JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3554
  • I like bugs.
I was considering more extreme cases like this:
Folder1: d:\ProjectFolder1\
Folder2: d:\ProjectFolder2\
Alternatively you can add subfolders too. If you wish to publish to d:\published\ for example, the feature will fail. Even in theory I don't know how the publish should work in this particular case, where the two folder should be copied?
Their common parent is d:\ root. Thus the new logic should copy them to
 d:\published\ProjectFolder1\
and
 d:\published\ProjectFolder2\
Why is my code not doing that? Maybe ExtractRelativePath() gets confused because of the d:\ drive letter. If so, it should be fixed.
Can you please debug the code to see what is going on. Add more DebugLn calls to procedure AdjustTopDir for example. I currently can develop for Windows only with a virtual machine and it is a big pain.

Your case is not really extreme because the directories have a common parent. This would be extreme:
Folder1: c:\ProjectFolder1\
Folder2: d:\ProjectFolder2\
but if somebody makes a project spreading over 2 Windows drives then he must "pay the consequences" and deliver his project by other means.

GetMem

  • Hero Member
  • *****
  • Posts: 3256
Quote
Can you please debug the code to see what is going on. Add more DebugLn calls to procedure AdjustTopDir for example. I currently can develop for Windows only with a virtual machine and it is a big pain.
In unit publishprojectdlg.pas, method AdjustTopDir instead of:
Code: Pascal  [Select]
  1. '../'
you need:
Code: Pascal  [Select]
  1. AppendPathDel('..')
or
Code: Pascal  [Select]
  1. '..' + DirectorySeparator;

« Last Edit: August 28, 2018, 09:04:02 pm by GetMem »

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3554
  • I like bugs.
Oops, right. I remembered the paths were switched at that point to use '/' as direcory separator but they were not.
It means the code did not work at all on Windows. Now I have tested there and was able to publish a test project.
Please test with r58798.

GetMem

  • Hero Member
  • *****
  • Posts: 3256
Tested with the latest revision, works fine now, both d:\ProjectFolder1\ and d:\ProjectFolder2\ are copied to the publish folder(subfolders too). However now two new directories appear along the main folders(as in the attached image). Folders "backup" and "lib" are originally part of ProjectFolder1. I saw a comment in the source: "Store project's BackupDir and UnitOutputDir for filtering files." I think the two directories are intentionally created, but the question is why? Both lib and backup are redundant for publishing purposes and should be ignored or I'm I missing something?

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3554
  • I like bugs.
Both lib and backup are redundant for publishing purposes and should be ignored or I'm I missing something?
You missed the ToDo comment in TPublishProjectOptions. :)
Code: Pascal  [Select]
  1. function TPublishProjectOptions.FileCanBePublished(const AFilename: string): boolean;
  2. begin
  3.   Result:=inherited FileCanBePublished(AFilename);
  4.   // ToDo: filter also based on FBackupDir and FUnitOutputDir
  5. end;
Actually I used your code from the publish.patch to separate those directory names.

I am not sure what is the best way to filter them. The ToDo comment may now be in a wrong place. Maybe procedure TPublisher.DoFileFound is a better place.
In addition to overridden procedure DoDirectoryFound there is also DoDirectoryEnter available which could be useful.
Could you please study and implement the filtering for those libs. I will probably not touch the ToDo items for at least a week.

Hmmm... how did the old code handle those directories?
BTW, why do you use the filters for your project? Are there some files which do not get copied otherwise?
« Last Edit: August 29, 2018, 08:19:14 pm by JuhaManninen »

GetMem

  • Hero Member
  • *****
  • Posts: 3256
Quote
You missed the ToDo comment in TPublishProjectOptions. :)
Indeed.  :)

Quote
Could you please study and implement the filtering for those libs.
Yes, but only next week. In my opinion we should ignore those folders completely. The publish feature is about creating a "clean" version of the project, only source code, no binaries, backup files, etc...

Quote
Hmmm... how did the old code handle those directories?
The old code ignored every subdirectory, so those folders were never published.

Quote
BTW, why do you use the filters for your project? Are there some files which do not get copied otherwise?
Until now I never used the publish project feature :), but I will in the future, especially when the compressing part is ready. In my opinion the Exclude filter is no longer needed, since we exclude everything, except project files. The include filter is useful if the user wish to publish other non-project files like images, resources, etc...



JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3554
  • I like bugs.
In my opinion we should ignore those folders completely. The publish feature is about creating a "clean" version of the project, only source code, no binaries, backup files, etc...
Yes of course. I was only thinking how to best inspect if a certain file is under the output/backup directory. Now we extract "lib" or "backup", it is better to get the whole path and check if a file's path matches.

Quote
The old code ignored every subdirectory, so those folders were never published.
Ah yes. It was very limited.

Quote
In my opinion the Exclude filter is no longer needed, since we exclude everything, except project files. The include filter is useful if the user wish to publish other non-project files like images, resources, etc...
I guess in the old design either exclude or include filter was used at one time. It did not make much sense to use them both, unless the wildcard filters somehow overlapped. (?)
I removed the exclude filter in r58804 and simplified the GUI still more. I also renamed variables and the whole dialog.
I did not change the functionality otherwise, I will have a short pause with that now.

GetMem

  • Hero Member
  • *****
  • Posts: 3256
@Juha
Looks/works great. I will implement the remaining functionality next week, also we need a lot of testing, hopefully other will join in. Regarding the filter text: "Use filter for extra files", I would change it to "Use filter to include extra files".

GetMem

  • Hero Member
  • *****
  • Posts: 3256
I implemented the remaining functionalities(r. 58851):

1. The possibility to compress the destination folder
2. Update relative paths in the published lpi
3. Always preserve directory structure. In theory it can handle weird directory structures too, like: c:\projectfolder1; d:\projectfolder2; d:\somefolder\projectfolder3; etc...
Please test.