Recent

Author Topic: Checking Speed of SynEdit Highlighters  (Read 105087 times)

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Checking Speed of SynEdit Highlighters
« Reply #90 on: October 29, 2013, 02:32:06 pm »
Morton gave me some ideas.
I've hacked up a keyword match function that appears to be 3 to 4 times faster than the current synedit implementation, at least for bulk Pascal code as used in the attached timed comparison which locates files at Windows locations. (Is there a cross-platform routine to locate installed Lazarus directories like $(LazarusDir) that works outside the IDE?)
I've not tried 'profiling' before, so the test app may be flawed, nevertheless initial results look promising.
I've used sets (basically an array of booleans at the bit level) to reject non-keywords, combined with an over-simple hash to match possible keywords which are confirmed by comparing two actual characters.

Edit
I've removed the faulty attachment. See later message.
« Last Edit: October 29, 2013, 10:54:12 pm by howardpc »

Morton

  • Full Member
  • ***
  • Posts: 111
Re: Checking Speed of SynEdit Highlighters
« Reply #91 on: October 29, 2013, 05:59:52 pm »
@howardpc
I can not load the project.
Lazarus 1.0.12 Linux 64Bit.

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Checking Speed of SynEdit Highlighters
« Reply #92 on: October 29, 2013, 06:55:46 pm »
You can extract the zip OK?
On Linux you'll have to replace the hard-coded file locations which are correct only for Windows.
I just picked several large source files at random really.
Or add a TOpenDialog and locate a pas/pp file yourself to feed to the comparison routine as you did in your own example.

Edson

  • Hero Member
  • *****
  • Posts: 1301
Re: Checking Speed of SynEdit Highlighters
« Reply #93 on: October 29, 2013, 07:07:09 pm »
I can compile on Windows, but it give me error on executing:
Lazarus 2.2.6 - FPC 3.2.2 - x86_64-win64 on Windows 10

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Checking Speed of SynEdit Highlighters
« Reply #94 on: October 29, 2013, 09:48:14 pm »
Sorry, previous project was developed on 64-bit non-release Lazarus version and it seems incompatible with release version.
I don't know why.
Delete the first download, and try the attached update, which I hope will not only compile but run for you.
« Last Edit: October 29, 2013, 09:50:35 pm by howardpc »

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Checking Speed of SynEdit Highlighters
« Reply #95 on: October 30, 2013, 12:01:05 am »
If it did not work for you, then change the hard-coded path  ;D
main_test.pp - line: 75

Code: Text  [Select][+][-]
  1. procedure TForm1.LoadAndParseTestFiles;
  2. const Path = 'c:\lazarus\ide\';
  3.  

Thanks Howardpc for sharing your code. Did not check it, yet.

Edson

  • Hero Member
  • *****
  • Posts: 1301
Re: Checking Speed of SynEdit Highlighters
« Reply #96 on: October 30, 2013, 02:02:39 am »
It compile and run correctly for me, now.

Good job howardpc.

At first sight it seem to be very fast. I like the algorithm focus on rejects.

But if you use TSynPasSyn.IsKeyword() of "SynHighlighterPas", it doesn't use the tipical hash-algorithm. It use a simple "Find" on a "TStringList". It seem to be a extra utility for searching Keywords.

Would you compare with TSynPasSyn.IdentProc()? It is used for the highlighter, for searching KeyWords.

I don't know what Martin thinks.
Lazarus 2.2.6 - FPC 3.2.2 - x86_64-win64 on Windows 10

sam707

  • Guest
Re: Checking Speed of SynEdit Highlighters
« Reply #97 on: October 30, 2013, 02:32:27 am »
I like the algorithm focus on rejects.

Great ! ;) TY

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: Checking Speed of SynEdit Highlighters
« Reply #98 on: October 30, 2013, 02:33:18 am »
Well interesting.

But the compare is useless. SInce you are not comparing with what the highligther does in a normal scan.

(
And if you did, as I wrote before, the current hash, has an implementation flaw. If a hash is matched it should first compare with the potential word, and then call the sub. Yet it calls the sub and compares there. Thats wasted time.
So for a fair compare, you need to rewrite the hash in the HL first.
)

Also You compare words, that already know the len. But the HL does not have that luxury.

If the HL would know the wordlen upfront, it could use that.
But parsing the text into words is part of the time a HL needs to spent.

---------------
I am not going to see, if I can find a word that fails it (since that would be a brute force search... / and for the few pascal keywords it will likely not exist.)

But in a highlighter with user configured keywords, it will be possible to have a set of keywords, that allow a none keyword to be matched.
After all: Each of the individual checks is just a partial check.



howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: Checking Speed of SynEdit Highlighters
« Reply #99 on: October 30, 2013, 08:27:59 am »
But the compare is useless. SInce you are not comparing with what the highligther does in a normal scan.
...
So for a fair compare, you need to rewrite the hash in the HL first.
Point taken. I'll look more carefully at the HL implementation to try to understand what it actually has to do, and see if I can offer any measureable performance improvement in that.

Two things I noticed in a quick scan of the relevant units. The keyword 'on' is listed correctly as a TP word, but then repeated in the Delphi list. The current docs (presumably the source of this) also have this same duplication, but it is clearly a typo.
Since FPC allows '@' as the first character of an identifier, I also wonder if TValidStringChars and the protected GetIdentChars method need to be updated.


Morton

  • Full Member
  • ***
  • Posts: 111
Re: Checking Speed of SynEdit Highlighters
« Reply #100 on: October 30, 2013, 10:02:31 am »
But if you use TSynPasSyn.IsKeyword() of "SynHighlighterPas", it doesn't use the tipical hash-algorithm. It use a simple "Find" on a "TStringList".

A hashed list like my mwkeylist.pas for MatcherTest would do it more than 20 times faster on average.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: Checking Speed of SynEdit Highlighters
« Reply #101 on: October 30, 2013, 10:41:50 am »

Two things I noticed in a quick scan of the relevant units. The keyword 'on' is listed correctly as a TP word, but then repeated in the Delphi list. The current docs (presumably the source of this) also have this same duplication, but it is clearly a typo.
Would you point me to more details? e.g. The documentation. The doc is fpc? So best to discuss on the fpc-mail list?

Quote
Since FPC allows '@' as the first character of an identifier, I also wonder if TValidStringChars and the protected GetIdentChars method need to be updated.

Does it? "@" is an operator. It can appear in front of an identifier. (With or without a space: OnClick := @   Foo; )



However the "&" is not yet handled
Code: [Select]
var
   &type: pchar;
type should not highlight.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: Checking Speed of SynEdit Highlighters
« Reply #102 on: October 30, 2013, 11:29:00 am »
Ok, after all, I took a look myself. Using a Prefix tree.

I have not compared it against the original hash. So no idea what is better.

Only compared against the code by HowardPC
On my system:
HowardPC  1295
Prefix  795

And the Prefix is a full compare.

Morton

  • Full Member
  • ***
  • Posts: 111
Re: Checking Speed of SynEdit Highlighters
« Reply #103 on: October 30, 2013, 11:56:40 am »
Only compared against the code by HowardPC
On my system:
HowardPC  1295
Prefix  795

The matcher with a full compare of the matches by a hashed list 14.6 million times per second.
The matcher with  just checking the matches by the hashes  in the hashed list 15.6 million times per second.
The matcher checket  by a Tstringlist 0.7 million times per second.

The hasches are unique enough to omit a full compare, and because of this omitting will save only 7%-10%.

A  hashed list with sublists would be much faster.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: Checking Speed of SynEdit Highlighters
« Reply #104 on: October 30, 2013, 12:03:04 pm »
Unfortunately the figures of HowardPC's test are in a different unit...

So I can not compare that...

 

TinyPortal © 2005-2018