* * *

Author Topic: An optimisation level better =0 hangs computer.  (Read 1775 times)

han

  • New member
  • *
  • Posts: 15
An optimisation level better =0 hangs computer.
« on: August 19, 2017, 01:07:41 am »
If have pretty large application. As soon I set in Lazarus the optimize level better then "no optimisation"=0,   the computer hangs/freezes after some time and only a hard reset will solve it.

I occurs  both in MS Windows (Lazarus 1.8,  FPC 3.0.2) and Linux (Lazarus 1.6.2,  FPC 3.0.0) environment. In the past,  I didn't have this problem. Any idea how to resolve this or how to find the problem?

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5809
Re: An optimisation level better =0 hangs computer.
« Reply #1 on: August 19, 2017, 01:18:37 am »
In case of vague bugs, you basically go after all things that can give you potential error sources:

- check warnings.
- turn on all checks (-CRriot )
- since you can test on Linux, try to run through valgrind (compiled with -gv). It runs slow, but checks integrity of heap allocations regularly, so that when it gives an error you know you are at, or at least pretty close to the origin of the bug.

.. and try to fix them all.

Thaddy

  • Hero Member
  • *****
  • Posts: 4629
Re: An optimisation level better =0 hangs computer.
« Reply #2 on: August 19, 2017, 09:48:12 am »
Do not only resolve (not just check) all warnings, but after you have done that compile with -glh first. (Run from a terminal window) That will reveal any memory leaks.
If something stops working after some time it is usually a memory issue.
After that, try to use the optimization you want and test again.
After that, follow up on the rest of Marco's advice.
"Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive."

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5809
Re: An optimisation level better =0 hangs computer.
« Reply #3 on: August 19, 2017, 11:44:46 am »
Do not only resolve (not just check) all warnings, but after you have done that compile with -glh first. (Run from a terminal window) That will reveal any memory leaks.
If something stops working after some time it is usually a memory issue.
After that, try to use the optimization you want and test again.
After that, follow up on the rest of Marco's advice.

Valgrind already catches that, so that is double work. Testing with -glh can be easier to get to work though.

han

  • New member
  • *
  • Posts: 15
Re: An optimisation level better =0 hangs computer.
« Reply #4 on: August 19, 2017, 11:23:32 pm »
Thanks for the hints.

I tested the compiler options but did not find any important error. Note that the problem occurs in compiling, not in the executing of the program. Linux is running in a virtual machine and in some cases it compiles fine but sometimes I get following errors:

- Failed to execute ppcx64.exe
- out of memory.

Not always. In some case it compiles and runs normally.

So in Windows, I started task manager. In some case it now compiles and works but in some cases PPCX64.exe eats all my memory, 2.7gbyte of 4gbytes, win10 64 bit system and the computer becomes very slow and then finally freezes.

So I have the impression FPC  runs out of control and eats all the memory.  The application HNSKY is about 40.000 lines long.

So any suggestion what to do next?

Han

Later, I tried the 32 bit compiler in a Win10 64 bit environment . Here Lazarus just stops compiling with error message "Fatal, no memory left" In task manager I see ppcxi386 using a lot of memory.
 
 

« Last Edit: August 19, 2017, 11:42:07 pm by han »

Akira1364

  • Sr. Member
  • ****
  • Posts: 322
Re: An optimisation level better =0 hangs computer.
« Reply #5 on: August 20, 2017, 12:42:13 am »
What sort of optimization flags do you typically build the compiler (and the RTL) with? It may very well have something to do with that. I've certainly noticed FPC using more memory (and taking longer) when linking against files that were built with a lot of debug information in the past.

Regarding the compiler executable specifically, I've found that the overall speed of compilation tends to be noticeably faster and that it crashes far less often when I build it with at least -O2 and no debug information or symbols (-g- -Xs), as well as the relevant CPU-specific optimizations (which in my case amounts to -CfAVX2 -CpCOREAVX2 -OpCOREAVX2.)
« Last Edit: August 20, 2017, 12:46:40 am by Akira1364 »

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5809
Re: An optimisation level better =0 hangs computer.
« Reply #6 on: August 20, 2017, 08:39:22 am »
Using switch -gtt to initialize variables to random values can be very useful too btw.

han

  • New member
  • *
  • Posts: 15
Re: An optimisation level better =0 hangs computer.
« Reply #7 on: August 20, 2017, 09:28:10 am »
Attached the .lpi file (zipped) with the compiler settings. Almost all settings are off. I exclusively use Lazarus. Any setting to change? Speed of final application is important.



Thaddy

  • Hero Member
  • *****
  • Posts: 4629
Re: An optimisation level better =0 hangs computer.
« Reply #8 on: August 20, 2017, 11:51:31 am »
Attached the .lpi file (zipped) with the compiler settings. Almost all settings are off. I exclusively use Lazarus. Any setting to change? Speed of final application is important.
Well that does not tell me too much that would help.
Does your application use a lot of recursion or local procedures? (you mention speed..) That can trick the compiler in recursive optimizations and would explain why the compiler itself runs out of resources trying to optimize those..
Especially if the code itself is not well written.
Set some hard limitations in recursive routines and local procedures and try again.

If you have simple code to reproduce it, please give us the code.
I can trigger the compiler into this behavior, but only with deliberately flawed code: either with recursive allocation or with recursive local routines.
For noobs: simplified explanation: yes in the application code, w/o executing, but during compilation the compiler builds a tree. When this tree is exhausted, compilation fails for lack of resources. This can be caused by even simple optimizations that are applied on endless recursion. Usually that is the programmers mistake and not the compiler.
« Last Edit: August 20, 2017, 11:53:46 am by Thaddy »
"Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive."

josh

  • Sr. Member
  • ****
  • Posts: 428
Re: An optimisation level better =0 hangs computer.
« Reply #9 on: August 20, 2017, 12:18:53 pm »
Hi
From original post I assume it was working in previous version of lazarus and fpc?
Have you tried going back to this version and trying from there?
If it still works fine in previous version, then check the documentation on the changes that have been implemented since your working version.
If it fails in previous version that was working, then possibly some change to your code has caused an issue.

Another one to watch is, that any 3rd party components your using is fully up to date and 100% supports your New Lazarus and FPC version.

I have had issues when moving from laz 1.3 and fpc 2.7.1 to trunk , and most of the issues were due to the changes made; that were documented in the changes; if your previous version used a lot of STRING variables; read carefully the changes made for "Unicode ??" your old code may have some difficulty, changing to shortstring may help.
I am still converting my old code which does not take too long, the time is in the testing and stress testing the modified code across paltform and os varients.

Do you have a lot of fixed sized arrays defined? if so have you tried creating these as Dyanmic; else you could runout of memory very quickly.

Just some thoughts that may help.

Josh
Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

Cyrax

  • Hero Member
  • *****
  • Posts: 542
Re: An optimisation level better =0 hangs computer.
« Reply #10 on: August 20, 2017, 12:31:25 pm »
Thanks for the hints.

I tested the compiler options but did not find any important error. Note that the problem occurs in compiling, not in the executing of the program. Linux is running in a virtual machine and in some cases it compiles fine but sometimes I get following errors:

- Failed to execute ppcx64.exe
- out of memory.

Not always. In some case it compiles and runs normally.

So in Windows, I started task manager. In some case it now compiles and works but in some cases PPCX64.exe eats all my memory, 2.7gbyte of 4gbytes, win10 64 bit system and the computer becomes very slow and then finally freezes.

So I have the impression FPC  runs out of control and eats all the memory.  The application HNSKY is about 40.000 lines long.

So any suggestion what to do next?

Han

Later, I tried the 32 bit compiler in a Win10 64 bit environment . Here Lazarus just stops compiling with error message "Fatal, no memory left" In task manager I see ppcxi386 using a lot of memory.

Make sure that you have set up enough swap space.

Thaddy

  • Hero Member
  • *****
  • Posts: 4629
Re: An optimisation level better =0 hangs computer.
« Reply #11 on: August 20, 2017, 12:58:38 pm »
Make sure that you have set up enough swap space.
Well... The compiler (not his own program) fails on its own recursion during optimization , so let's say put the swap space on infinite..... >:( >:( >:D >:D

Anyway, in the case of my contrived code to reproduce this, it would be mitigated by {$R+} in his code, so the compiler will see that an ever increasing counter has an actual limit and should not start from zero again, indefinitely. He has sourcecode errors. He did not give us sourcecode.
« Last Edit: August 20, 2017, 01:11:49 pm by Thaddy »
"Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive."

han

  • New member
  • *
  • Posts: 15
Re: An optimisation level better =0 hangs computer.
« Reply #12 on: August 21, 2017, 12:39:41 am »
Thanks for the hints. I will try them tomorrow. The 40.000 lines astronomy freeware application has a long history and  uses a lot of complicated math and string & data manipulations. A part of the code is published.  A small part are third party copyrighted routines.

I noticed compiling seems to run out of control at the same phase as visible in the three screenshots. I will investigate if I can tweak sometihing to fix it. I will also try going back to older compilers and code.

The weirdest thing that is seems to behave randomly. Sometimes it compiles sometimes not in Linux. At least the operating system stays now functional in a Linux virtual machine and I386 allowing some experiments and more testing. In win64 is just crashes the operating system.

han

  • New member
  • *
  • Posts: 15
Re: An optimisation level better =0 hangs computer.
« Reply #13 on: August 21, 2017, 12:42:27 pm »
Located the problem!  :)

By placing in the units the compiler directives  {$OPTIMIZATION OFF} {$OPTIMIZATION ON} the procedure causing the problem has been located.

It contains only has a long case statement with calculations. I have placed it in a test application and the problem is now reproducible. See attached zip file. If the target is 64 bit it crashes my win10 so be aware. It contains an  {$OPTIMIZATION OFF}  If you change it to  {$OPTIMIZATION ON} the ppcx will eat all the memory.  If the target is i386 WIN10 64bit stays healthy and compiler end with "out of memory".

Will now investigate further.

Later:
Found it is caused by case range. Reducing range from +99999999 to +9999999  fixes the problem:

Causes the compiler to consume all memory:
case year of
  3000..+99999999:begin
                             end;
end;{case}



Compiles without problems:
case year of
  3000..+9999999:begin
                           end;
end;{case}


So it is fixed but why does it happen?

Han

« Last Edit: August 21, 2017, 02:52:29 pm by han »

Cyrax

  • Hero Member
  • *****
  • Posts: 542
Re: An optimisation level better =0 hangs computer.
« Reply #14 on: August 21, 2017, 02:53:20 pm »
Please report your findings in the bugtracker. Also, if you can test if FPC trunk behaves similarly.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus