"Demo" project size for possible bug(s)

[440bx]

Hello,

This question is because I want to make sure I don't unwittingly do/expect something unreasonable.

I have experienced some "deficiencies" in the current version of Lazarus (v3.2) as well as the trunk version (very recent version, just yesterday's.)  The problem is, when I tried to replicate those "deficiencies" in a simple/short program, they are not there anymore or they are different.   This is a problem by itself.

I am currently working on a project, which I intended to contribute once finished, that is roughly six (6) thousand lines of code and has all the "deficiencies" that are getting in my way but, that's a "big" little project just to point out a few minor deficiencies.

My question is: should I attach that project along with the deficiencies I've observed or should I go about it a different way ? (the 7zip file is 333Kb and it includes a 64bit version of the Zydis (a disassembler) dll created with Visual Studio, i.e, requires Windows.)

NOTE: the project isn't finished but it's somewhat usable/useful in its current state.  It's about 80% done.  Anyway, I will make the "full version" available when it's done.

[AlexTP]

You may post the bugreport (bugtracker) with the src only, and give the GitHub link, or e.g. OneDrive link, to Assembly DLLs. This way several bugreports from you will share the single DLLs.

[TRon]

My question is: should I attach that project along with the deficiencies I've observed or should I go about it a different way ? (the 7zip file is 333Kb and it includes a 64bit version of the Zydis (a disassembler) dll created with Visual Studio, i.e, requires Windows.)

Did you limited your project with adding only windows headers for the library  ?

[440bx]

Did you limited your project with adding only windows headers for the library  ?

Yes, the headers are only for Windows but, there are more headers than there would normally be because I use my own Windows headers. I don't use the windows, messages, etc units, I use my own and I had to include those that were necessary to compile the project. That's 139K uncompressed and maybe 10K compressed (all text with lots of whitespace, 7zip squeezes that stuff really well.)  That's nothing compared to the full size of all the headers, about 17MB, it takes time to extract from there to create a small custom version just for that test program (that's not a problem, it's already done because I intended to share the final result anyway.)  In that project, what takes a fair amount of space is the dll, its 224K compressed, a little over 1MB uncompressed.

[Martin_fr]

First of all, it would be good to know where about in the IDE the deficiencies are. Different maintainers may have different preferences as to what they find easier to deal with.

In general however, there are several implication to "minimal", and then it comes down to try and match as many of them as possible.

1) The project itself. 

2) The dependencies.
That can in some cases be even more important. The less dependencies the better. Zero should be the goal. Even if you have to copy some units from a dependency.

2a)  Dependencies that need to be installed into the IDE.  Really, really try to get rid of those, please.

Minimal example is usually not about the download size. But the work the extra code causes when tracing the issue.

Example, if I need to debug codetools, because some declaration breaks it, then codetools will scan your entire code, a lookup from the point that fails, may iterate any identifier in your code, and all that needs to be watched in the debugger. => So if there are 20 fields in a class that don't relate to the issue, then that are 20 fields for which debugging has to follow codetools.

Of course it then depends on how easily you can reproduce the issue. If you need to test for an hour each time you removed a field, then that isn't practical.

If the issues with certain settings of the IDE, then try to start of with a default config for the IDE, and see if you can find the minimum set of config changes. That can help getting an initial idea where the issue may be.

If it takes steps to reproduce (including making config changes not present in a downloadable config file), then try find a small set of such steps.


But ultimately, there isn't a hard-limit to what is a minimal example. Sometimes a bug really needs large amounts of code. Or at least a small example can't be found without actually knowing what causes the issues.

[TRon]

Yes, the headers are only for Windows but, there are more headers than there would normally be because I use my own Windows headers. I don't use the windows, messages, etc units, I use my own and I had to include those that were necessary to compile the project. That's 139K uncompressed and maybe 10K compressed (all text with lots of whitespace, 7zip squeezes that stuff really well.)  That's nothing compared to the full size of all the headers, about 17MB, it takes time to extract from there to create a small custom version just for that test program (that's not a problem, it's already done because I intended to share the final result anyway.)  In that project, what takes a fair amount of space is the dll, its 224K compressed, a little over 1MB uncompressed.

Ah, thanks for clarifying 440bx.

That is unfortunate because that leaves out a simple test to check if the issues you face are perhaps platform related. E.g. As long as the project (the library in this case) is well maintained it should not have to be a problem to build the library for a given/supported platform.

[440bx]

First of all, it would be good to know where about in the IDE the deficiencies are.

The two main problems I've been experiencing very often are:

1. attempting to jump to a function declaration jumps to a place that is not remotely related to the function it was supposed to jump to.  (I've tried replicating the problem in a smaller program but, the problem usually goes away.)

2. similar to the above and possibly related, when hovering over a constant identifier, e.g, const SOMECONSTANT = 'constanttext', sometimes there is no popup window at all, sometimes there is a popup window but it is grey (in spite of the TP design package being installed) and sometimes it is yellow as expected.  The behavior depends on the identifier hovered onto and I have not been able to discern a pattern that causes a specific behavior.  Of course, if there is no popup then attempting to jump to the constant definition is futile.    I've also tried, unsuccessfully, to replicate this problem in a smaller program.

One thing I've noticed is that, for what it cannot locate correctly, it seems to always jump to the same place (totally unrelated to where it should have jumped.)

Now that I'm typing this stuff, I'm thinking that maybe what I should do is finish the project.  Once it's done, rewrite it a little bit at a time testing every behavior I've noticed in the development of the original and post those smaller "snapshots" that exhibit the problems I came across while coding the finished version.  The real problem right now is that I'm coding and coding and by the time I notice a problem in Lazarus, I have no idea of which one of the many changes/additions I made may have triggered it.  Rewritting from scratch focusing on the appearance of those problems would likely yield useful information.

Just FYI, I normally have very few to no dependencies.  The only dependency I can think of is that TP design package that gives the nice yellow hints.  Other than that, everything I use is out of the box and my own code (which I would supply of course.)

The problem with what I've got at this time is that it's about 6K lines of code (that's the actual program), 25,500 total source lines (reported by FPC.)   Depending on the nature of the bug, that many lines of code may be a real hindrance.

[440bx]

Ah, thanks for clarifying 440bx.

You're welcome.  I understand that, the project being Windows-specific leaves out the possibility of finding out if the problems are platform specific or not.

[Josh]

Have you tried rescanning the  sources, i had a similar issue where jump to dfinition, jump to proc went 'off its head', and this seemed to fix it.

[Martin_fr]

1. attempting to jump to a function declaration jumps to a place that is not remotely related to the function it was supposed to jump to.  (I've tried replicating the problem in a smaller program but, the problem usually goes away.)

2. similar to the above and possibly related, when hovering over a constant identifier, e.g, const SOMECONSTANT = 'constanttext', sometimes there is no popup window at all, sometimes there is a popup window but it is grey (in spite of the TP design package being installed) and sometimes it is yellow as expected.  The behavior depends on the identifier hovered onto and I have not been able to discern a pattern that causes a specific behavior.  Of course, if there is no popup then attempting to jump to the constant definition is futile.    I've also tried, unsuccessfully, to replicate this problem in a smaller program.

Those may be the same thing... or not.

The big question is, can it be reproduced, with a freshly started IDE? I.e, is it always reproducible?

Also, when it happens, check the "messages window" => is there a message from codetools, claiming there is an error in the source ? (even if the source is fine).

If codetool believes there is an error in that source, then it may jump there (there is a setting to turn that off, but then it will just print the error and do nothing else).
And if codetool believes there is an error, it may not be able to provide the hint either.


If that is the case, the workaround (without restarting the IDE) is:

Edit the code one or two lines above the claimed error, and introduce a real error there. Attempt to "Jump to declaration" from the word after the edit you done. (will fail). Undo the edit, and do the "jump to declaration" from the word after the edit again.
After that codetool should be fine again.

Background: Something goes wrong with a cache in codetool. Editing the code in the described way clears that cache.

If that is the issue that you have
- It is really old, and so far evaded any attempt of locating it.
- If you have a case that can always reproduce it => great. (include your IDE config, just in case).

[440bx]

@Josh,

Thank you for the suggestion.  I tried it, made no difference.  Definitely was worth a shot.

The big question is, can it be reproduced, with a freshly started IDE? I.e, is it always reproducible?

At least the answer to that is definitive and it is a definite _yes_, always reproducible and, in the particular case I'm going to describe, the behavior is always exactly the same.  I will refer to the attached screenshots.

Step 1. the first screenshot, ProcJump_00 shows exactly what I'm doing.  Right click on "OutputSegments" and select the first menu option "Find Declaration of ..."

The second screenshot, ProcJump_01, shows where I land.  It also shows there is a codetools error.    I followed your suggestion, I added some junk on line 4335 (the line above where codetools complains about an error), recompiled, got an (expected) error from FPC, removed the junk to cause the error to go away, recompiled, no FPC errors, attempted "step 1." again and landed exactly in the same place.

The third screenshot, shows where I landed where I started from and where I should have landed.  It shows the two places are more than a thousand lines apart.

I performed the test multiple times, some times with a freshly started IDE and a freshly and successfully recompiled program.  The result was _exactly_ the same.  codetools gave the same error and jumped to the same place.  This problem is as repeatable as repeatable gets: exactly the same way, nothing changes.

One interesting thing, refer to the ProcJump_01 screenshot.  Attempting to go where "LABEL_CONVERT" is defined/declared works as expected and interestingly enough, hovering over that identifier results in a little yellow popup showing information about it.    OTH, hovering over "ZYDIS_INSTRUCTION_ENCODING_XOP" (or any of those) does not show a popup window and attempting to go to where the identifier is declared causes the caret to jump back to line 4336.  It always jumps to that line, that's also always repeatable.

Also, when it happens, check the "messages window" => is there a message from codetools, claiming there is an error in the source ? (even if the source is fine).

Yes, the ProcJump_01 screenshot shows the error.

If codetool believes there is an error in that source, then it may jump there (there is a setting to turn that off, but then it will just print the error and do nothing else).
And if codetool believes there is an error, it may not be able to provide the hint either.

That's exactly what is happening.

If that is the case, the workaround (without restarting the IDE) is:

Edit the code one or two lines above the claimed error, and introduce a real error there. Attempt to "Jump to declaration" from the word after the edit you done. (will fail). Undo the edit, and do the "jump to declaration" from the word after the edit again.
After that codetool should be fine again.

Tried that more than once, it did _not_ make the problem go away.

If that is the issue that you have
- It is really old, and so far evaded any attempt of locating it.
- If you have a case that can always reproduce it => great. (include your IDE config, just in case).

It definitely sounds like the issue I am experiencing.  As far as IDE config, what files would you like me to include ?

Just for completeness I was going to check how v3.2 behaved.  When I first started v3.2 the second editor window was in the wrong place and with the wrong dimensions, I moved the editor window and resized it. Once that done, I quit for it to save the settings and restarted it.  To my surprise all source editor windows (both of them) were gone.  No editor windows.  I selected "View->Source Editor" and got what you see in screenshot ProcJump_04_lv32 (lv32 = Lazarus v3.2).  That was a surprise and now that's all I get.  I can quit and restart Lazarus and I get NO editor window until I "View->Source Editor" and then I get that blank window.  That's repeatable too but, I don't know if the original cause is repeatable because I cannot repeat that part.

NOTE: I just noticed that I skipped number "02".  IOW, there is no "ProcJump_02".

Also, with v3.2, if I open a different project then I get the "Source Editor" window as expected but, when I re-open that project, the editor window is gone again until I do "View->Source Editor" and get a blank (no text) window.  Fortunately, that problem does _not_ exist in v3.99.  V3.99 shows both editor windows correctly.

[Thaddy]

I have observed a similar issue with reasonably large code with many include files. The IDE seems to loose track which makes it hard to debug. (In my case KOL, which is 60K+ lines, that can give me a headache.)
Can it be that there is a mismatch between how the IDE keeps track of the includes, the inclusions themselves are OK to the compiler, and the presentation layer, which uses a different parser from the compiler?
IOW the IDE can not match the compiler when there are many includes. (and/or many conditionals)
maybe the IDE or CodeTools has a fixed limitation on its include stacks? Or its conditional stack?

That's rethorical, btw.

The number of lines parsed are not correct with include files taken into account. These lines are not counted or not correctly counted.
In the case of KOL it can be several 1000's of lines off and points to code that is completely unrelated.
I would look at the include mechanism and the line count. An editor like Geany handles them correctly, since it relies only on what the compiler tells it.

[440bx]

IOW the IDE can not match the compiler ...

I think that is the root of the problem.  I am amazed at how well codetools does given that it is, codewise, a totally different parser for the dialects FPC supports, most of which lack a formal definition.  The cause or causes for the mismatches between the two parsers is anyone's guess at this time (and, given the lack of formal language definitions, likely at anytime.)

An editor like Geany handles them correctly, since it relies only on what the compiler tells it.

I believe that is the right way of going about it but, unless I am mistaken that means Geany depends on debug information being produced by the compiler which, if that is correct, is also problematic.

This thing reminds me of the interview with Anders Hejlsberg where he talks about the C# compiler and how its parser is designed to "assist" the VS IDE. 

Ideally, scanning, parsing, code generation and linking should be _completely_ separate compiler steps (for those which include linking) but the reality is, at least for FPC, is that the first three and likely the rest, cannot be performed independently which means, it would probably be difficult to have "parsing only" information from FPC that the Lazarus IDE could use (it would eliminate the need for codetools since the compiler would become the "codetools" facility.  For that to be genuinely useful would likely require to have FPC function as a library too in order to avoid having to do IPC (which even when really well implemented, is on the slow side, unless FPC can be made into a service/daemon.)

[440bx]

Figured out the source of the problem! and have a reasonably small program where it is repeatable and testable   That should be useful to smack that annoying bug.

As usual, refer to the screenshots.

The first screenshot shows the popup working.  This is how things should be.  The popup won't show in the attached project without modifying the source code.

The second screenshot is with the caret hovering over "APPNAME" on line 152 and no popup showing indicating there is a problem.  Attempting to jump to the definition of APPNAME will go to a place which is not where APPNAME is defined.

The third screenshot shows the culprits that cause the problem.

Do the following:

1. comment out "enum" on line 54 ( // works fine for that).  Now that "enum" is commented out, go back to line 152 and attempt to jump to the definition of APPNAME.  It will work!!  Uncomment "enum" and it will no longer work.  Do that as many times as you like.  Whenever "enum" is commented out, jumping to the definition of APPNAME works, whenever uncommented the jump fails.

2. with "enum" uncommented, comment out line 56, the definition of the "Static" variable.  with "Static" commented out, jumping to APPNAME works in spite of "enum" not being commented out.  Uncomment "Static" and the jump fails again.

3. with "enum" uncommented, ensure "Static" is uncommented, the jump to APPNAME should (and does) fail.  Now change the name of the "Static" variable to "StaticX" (or some other name of your chosing) and, jumping to the definition of APPNAME works again!.

In summary:

Codetools dislikes something about the combination of an enumeration followed by the word "static".  IOW, there are cases where codetools treats the word "Static" as being special and one of those cases is when it is preceded by an enumeration.

PS: I modified the project I used to report the grey popup problem and kept its name but, the attached project has nothing to do with the color of the popup.

[440bx]

Another interesting "factoid" about that bug is that changing the name from "Static" to "&Static" makes the problem go away even though a regular (non-reserved word) identifier starting with "&" is the same as one that starts without the "&", i.e, functionally &identifier = identifier.  The difference is only for the scanner to consider reserved words to just be regular identifiers.

IOW, to codetools, "&Static" is _not_ the same as "Static" and that is a problem because, at least in that context, they are functionally the same (the scanner does not need the "&" to scan it and define it correctly for the parser.)