project1.lpi and project1.lps are empty

[anna]

Related bug : https://bugs.freepascal.org/view.php?id=35460
Lazarus: lazarus-1.6.4-fpc-3.0.2-win32.exe (MD5 = 91466e5072595f38d3cff2f691d5f2c2 )

My question is. Where is the code responsible for the time comparison in the sources? Lazarus 1.6.4 makes this time comparison of project.lpi during the project-save process.

E.g. in debugger I see such ASCII lines as:
"ProjectOptions/i18n/EnableI18N/Value"
"ProjectOptions/PublishOptions/"
,--near virtual address  00719B78. But I can't see these lines in sources.

Code: [Select]

00719B78  |.  E8 D313F0FF   CALL lazarus.0061AF50
>0061AF7E  |.  FF93 DC000000 CALL DWORD PTR DS:[EBX+DC]
>>009431F8  |.  E8 D312D6FF   CALL lazarus.006A44D0
>>009431FD  |.  84C0          TEST AL,AL
>>009431FF  |.  75 28         JNZ SHORT lazarus.00943229
>>00943201  |.  8B45 F0       MOV EAX,DWORD PTR SS:[EBP-10]
>>00943204  |.  E8 A70CD6FF   CALL lazarus.006A3EB0
Loaded module I:\lazarus\lazarus.exe
009431F8 -- is the time comparison call.
00943204 -- is writing 2nd project1.lpi call

First Lazarus 1.6.4 creates 1st project.lpi, computes time. Then (after unit1.pas saving) Lazarus moves project.lpi to backup folder, and creates 2nd project.lpi, calculates its time, makes comparison of times and write 2nd project.lpi only when times are different.

[Martin_fr]

No one will know that from the assembler.

How did you get to that asm? If you compile lazarus with debug info, the debugger should tell you where the code is in the source.

The 2 string constants are in packager\packagedefs.pas
procedure TLazPackage.SaveToXMLConfig(XMLConfig: TXMLConfig; const Path: string);
Line 2950...

And also in the related Load procedure.

[anna]

It seems 00719B78 is line

Code: [Select]

FXMLConfig.Flush;from I:\lazarus\ide\project.pp.
But I can't find time comparison inside. When I load lazarus.lpi in Lazarus, and set a breakpoint in project.pp , the breakpoint doesn't work. Why??

[lucamar]

When I load lazarus.lpi in Lazarus, and set a breakpoint in project.pp , the breakpoint doesn't work. Why??

Because it's not built(not just compiled, built) with debug info? (Just a guess)

[Martin_fr]

Does it have blue dots in the gutter? Assuming you use the IDE debugger.

To build the IDE with and the package LazUtils (which contains the function) with debug info, go to
  Tools > Configure Build Lazarus
add -gw to the options.

In the Package LazUtils make sure that the Opitons for the package (package dialog > options) under "custom options" contains $(IDEBuildOptions)

[anna]

Does it have blue dots in the gutter?

No. So I assume Lazarus doesn't use it.

I do not understand what the rest of the manipulations mean. E.g. I have 2 Lazarus installed : Lazarus 1.6.4 and Lazarus 2.0.2. I'm trying to load Lazarus 1.6.4 in Lazarus 2.0.2. In what lazarus must I set -gw option ? -- If in Lazarus 1.6.4 then it changes nothing.

As for the LazUtils package, I don’t see how to enter to the package options.

[Martin_fr]

Best to debug each Lazarus in itself.

If you want to debug 1.6.4, then you only use 1.6.4

- Add -gw to the Tools > ConfigureBuildLaz.
- Open the package LazUtils (package menu > open package: look inside the 1.6.4 install)
- The package window has an option button, and it opens a window, which as a "custom options" section)
  In that section is a big memo: $(IDEBuildOptions)

Rebuild 1.6.4

In  the running 1.6.4 open the ide/lazarus.lpi from the 1.6.4 install, then press run.
Now it should be debugging.

If you make changes to any source file, always rebuild via Tools > Build Laz.

---
Same for 2.x. Just use the 2.x IDE on the 2.x install folder.

[anna]

Thank you very much. Now blue dots appear.
The time comparison is in I:\lazarus\components\codetools\codetoolsconfig.pas

Code: [Select]

procedure TCodeBufXMLConfig.WriteXMLFile(ADoc: TXMLDocument;
  const AFileName: String); 
...
         if Buf.FileOnDiskIsEqual then exit; 
...


Code: [Select]

function TCodeBuffer.FileOnDiskIsEqual: boolean;
begin
  if IsVirtual then
    exit(true);
  if IsDeleted then
    exit(not FileExistsCached(Filename));
  if (not LoadDateValid)
  or Modified or (FFileChangeStep<>ChangeStep)
  or (not FileExistsCached(Filename))
  or (FileDateOnDisk<>LoadDate)
  then
    exit(false);
  Result:=true;
end;
The difference is that Lazarus 1.6.4 sets LoadDateValid=true, but Lazarus 2.0.2 LoadDateValid=false.
So if LoadDateValid=false is hardcoded , the bug is non-reproducible on Lazarus 2.0.2. I need some time to investigate.

[Martin_fr]

I have tried lazarus-2.0.2-fpc-3.0.4-win32.exe on Virtual PC with guest OS WinXP SP3 and it seems the bug is fixed.

Ok, I am with Juha: If the issue is fixed, then what is the problem?
There are no fixes going to be made for 1.6.x.  Unless you want to patch your local version yourself.

As I see the bug occurs when I'm clicking very quickly on Save button of the Save dialog.
If I wait a few seconds before unit1.pas saving, the files are not empty.
I can't reproduce the bug in OllyDBG, because a breakpoint on WriteFile function makes some delay in execution and the bug disappears.

There is some code in Idle or QueueAsync. It updates some cache of available files. (Checks for changes on disk too).
So you may want to see if you can find that, and if it has anything to do with the issue.

Mind that is just a very far guess... May be entirely off...

[JuhaManninen]

@Anna, why exactly you want to use Lazarus 1.6.4?
Is there a regression bug in 2.0.x that prevents you from updating? We should fix that instead.

[anna]

why exactly you want to use Lazarus 1.6.4?

The main reason not to update is consumption of RAM. I have 2 GB RAM  and haven't a possibility of  upgrade. I turned off the paging file, so if a using memory reaches 1,7 GB (video-adapter consumes ~300 MB) some processes are just randomly terminated by OS.
I've installed Lazarus 2.0.2.
Lazarus 2.0.2, depending on the project, consumes 500 MB and  more. When I used Lazarus 2.0.2 in debug IDE mode gdb.exe was consuming 1GB.


Returning to the topic, I did not find where 1.6.4 and 2.0.2 sets the LoadDateValid flag.
1.6.4 makes LoadDateValid = true after Buf.Save of 1st project.lpi. When it calls WriteXMLFile to save 2nd project.lpi,  the flag is still true. And if a user didn't check ShowHint property the flag is set to false by Buf.LoadFromStream(ms).
Lazarus 2.0.2 has the flag false at WriteXMLFile call always.

Code: [Select]

"I:\lazarus\components\codetools\codetoolsconfig.pas"
procedure TCodeBufXMLConfig.WriteXMLFile(ADoc: TXMLDocument;
  const AFileName: String);
var
  Buf: TCodeBuffer;
  ms: TMemoryStream;
  Cache: TCodeCache;
begin
  Cache:=GetCache;
  if Cache<>nil then begin
    Buf:=nil;
    if (not fKeepFileAttributes) or (not FileExistsCached(AFileName)) then
      Buf:=Cache.CreateFile(AFilename)
    else
      Buf:=Cache.LoadFile(AFilename);
    if Buf<>nil then begin
      fKeepFileAttributes:=true;
      ms:=TMemoryStream.Create;
      try
        Laz2_XMLWrite.WriteXMLFile(ADoc, ms, WriteFlags);
        ms.Position:=0;
        Buf.LoadFromStream(ms);
        if Buf.FileOnDiskIsEqual then exit;
        //debugln(['TCodeBufXMLConfig.WriteXMLFile writing ',AFileName,' ...']);
        if Buf.Save then exit; // success
      finally
        ms.Free;
      end;
    end;
  end;
  // try default (this will create the normal exceptions)
  inherited WriteXMLFile(ADoc, AFileName);
end;
But I still think that the date comparison (FileOnDiskIsEqual procedure) can cause a problem, because it leaves a possibility if comparing times are  < 1 sec, there will be no writing. Maybe it should be so according to logic, I'm not sure.

Also if a project never was saved , then a making the backup is weird.

[JuhaManninen]

The main reason not to update is consumption of RAM. I have 2 GB RAM  and haven't a possibility of  upgrade. I turned off the paging file, so if a using memory reaches 1,7 GB (video-adapter consumes ~300 MB) some processes just randomly terminated by OS.
I've installed Lazarus 2.0.2.
Lazarus 2.0.2, depending on the project, consumes 500 MB and  more. When I used Lazarus 2.0.2 in debug IDE mode gdb.exe was consuming 1GB.

That sounds very strange. Lazarus 2.0.2 should not consume much more than 1.6.4. The new HiDPI icon support may consume a little more but that is marginal.
You can make it lighter by removing IDE packages which are built into the "BigIDE" but not needed necessarily, like TurboPowerIProDsgn and OnlinePackageManager. The latter in a new addition in BigIDE.
Building IDE with debug info without optimization increases the binary size.
I have an IDE with all debug info + BigIDE packages. When browsing a big project (lazarus.lpi) it eats some 200MB memory. See my screenshot. The OS is Manjaro Linux + KDE, Lazarus built with GTK2 bindings.
Compilation takes more memory because it starts an external compiler but 1.6.4 does the same thing.
Your screenshot shows 2 lazarus instances, one eating 530MB and one 105MB. Why?

To reduce memory consumption you can:
1. Build a slim IDE by removing unused IDE packages.
2. Build an optimized IDE when you don't need to debug the IDE itself.
3. Dump Windows and install a desktop Linux into your computer. Yes I am serious with this. 1,7 GB user memory is too little for Windows especially when it hogs more and becomes slower over time. Install MX Linux (or Manjaro) and it feels like a super-computer.

You can do all possible tasks using a modern desktop Linux, except for playing new 3-D games, but your old machine prevents that anyway.
I even used MS O365 apps recently from my Linux box. They work well with both Firefox and Falkon browsers.

Once you start to build Lazarus more often, you may also want to get sources directly from the SVN fixes branch:
 http://wiki.freepascal.org/Getting_Lazarus#Fixes_branch_of_latest_Lazarus_release

[Martin_fr]

Maybe a question of what packages you have installed.
I managed to hit 450 MB, but for that I had to rebuild Lazarus itself (which is a large project), including smart linking and full debug info (which both add to mem usage).

Of course that is without debugging. Debugging will need some memory.

I have not tested any of the below. They may save nothing, or just a very few bytes. They may save more.

If you look at our sourceforge page, in each of the Windows sections (32/64) there is an "Alternative GDB" folder. The newer gdb are often faster. I have not compared them for memory usage.
But whether you use 2.x or 1.6, you may want to download them, and compare them.

You also may want to set "Reset debugger after each run" in Tools > Options > Debugger. That may release some memory.

Try "stabs" vs "dwarf with sets" (on 64 bit, it is always dwarf, stabs does not work / on 32bit you can choose)
dwarf gives the better results. But I do not know which uses more mem.
If you do, ensure you change it for packages too. They have there own options. (Search "Additions and overrides" on the wiki)

Better yet, if you do not step into packages (like lcl) during debug, then consider turning off debug info for those packages. That should save some mem. (debugging and compiling)
To do that, open each package, go to its options, disable debugging, and under "custom options" remove $(IDEBuildOptions)

You can save some more mem (1.6 and 2.x) with regards to debugging

- Turn off the debug history. The window (view > debug windows > history) has a power button.
  Unfortunately it does not stick.
   Edit  lazarus\debugger\historydlg.pp   and change the state of the power button in the object inspector.
   And edit lazarus\debugger\debugger.pp  around line 2765
       constructor TSnapshotManager.Create;
          FActive := True;  // change to false
  I think that should stop it.

- If you eval a big lot of watches, the debugger keeps some cache. Disabling it may slow down the debugger, but can save some mem.
  lazarus\components\lazdebuggergdbmi\gdbmidebugger.pp
  Around line 11550  comment out this line:
  FTheDebugger.FTypeRequestCache.Add(ContextThreadId, ContextStackFrame, AReq^);
 
You may have to search for the lines. The numbers are svn trunk. But you should be able to find the code, even in 1.6. It may be a few 100 lines above or below.

If you do test 2.0.x again. Try fpdebug.
I have not tested it for mem usage. But it does not need gdb at all. Of course it will need more mem in the IDE. Not sure if the grant total is better.

Note that FpDebug only works with "dwarf" (dwarf with sets).

Also fpdebug is still beta. It still has some small hiccups. But on windows it should be use-able.

If your apps have large structures (big arrays, etc), in 2.0 you can set various memory limits for gdb (beyond those, gdb will not eval any watch)

In 2.0, there are some new Highlights for the editor (such as begin end blocks having a vertical colored line).
Turn them off, and restart the IDE.

Tools > Option > Editor > Markup and Matches
  Turn off "Outline (global)

In 1.6 and 2.0

You also may turn off "Keyword brackets", or deselect some of the keywords you do not need.

You may want to turn off folding ... > Editor > Code Folding
Or turn off all highlight completely (Icon on the "colors" page).

Turn off "Highlight occurrences off word under caret"

And ... > Editor > General > Miscellaneous : Turn off "trim spaces" 


Not sure how much that will save....
As I said try it

Delete (backup) the folder  (will affect code completion/navigation // compilation will still work)
 lazarus\fpc\3.0.4\source\packages
and do menu Tools > Rescan fpc directory
restart IDE

Play with using / not using "smart linking"

Smart linking needs a ton of memory to link your exe.
Though without it the resulting exe will be bigger. (and more data for the debugger)