Internet Tools

[BeniBela]

I made a new version, of my internet tools package..
It should contain everything you need to process web pages:
In details:

• Wrapper around wininet and synapse, which inherit a common class, so you can switch between both libraries
• a simple html parser to read all tags and text elements (a little bit like sax)
• a tree parser that will build a tree structure from it (definitely not  dom)
• an almost standard-compliant implementation of XPath 2 to run simple queries on that tree
• a template based html processor, that will match a template against the html document and store the result of several queries in different variables (similar idea like xsl, but better for pascal) 
• finally an auto-update class

It's there: http://benibela.de/sources_de.html#internettools

new stuff is on the last page of the thread!

[Leledumbo]

I'll look into it, might be useful for web apps.

[BeniBela]

A new version:

Common commands in the html templates can now be abbreviated  (e.g. you can use <a>{link:=@href}</a>* to read all links instead of <template:loop><a><template:read var="link" source="@href"/></a></template:loop> )

And the XPath 2 part is now mostly Standard compatible

[User137]

This is kind of interesting, but do you know if anyone has wrapped sockets with Synapse? Managing TCP and/or UDP sockets and their threads, for fast communication between applications though internet.

[BeniBela]

This is kind of interesting, but do you know if anyone has wrapped sockets with Synapse? Managing TCP and/or UDP sockets and their threads, for fast communication between applications though internet.

For P2P or own client<->own server? No idea...

(and my internet tools are not really intended for connecting to something; more for processing the html once it is downloaded)

[picstart]

The downloaded code has very many dependencies on the file structure the author is using. Files are missing
Ex. progressdialog.pas(69,4) Fatal: Can't open include file "progressdialog.lrs".
 

[BeniBela]

Files are missing
Ex. progressdialog.pas(69,4) Fatal: Can't open include file "progressdialog.lrs".
 

No, I believe that file is automatically generated by Lazarus, when you compile.

At least it is, if you open progressdialog.lfm and save it.

[picstart]

Still too many dependencies it won't compile without the lrs file and saving the lfm does not help.
What is unpleasant about lazarus
With lazarus the exe is reasonably portable to other folders on a PC and even to other devices via a cross compile EX winCE. But that's where frustration free portability ends. The IDE environment is very quirky with dependencies outside the folder the IDE runs in. This means any outside dependency Ex.users\userxx\appdata\local\lazarus must be backed up and deleted and a clean install done just to avoid being tripped up by hidden dependencies. Even after doing all that there is no certainty some vestige dependency still persists short of installing on a new PC that never ever had lazarus installed. This is further compounded when others seek to supply ide compatible source code ... if their hidden dependencies are different than others hidden dependencies then issues arise.
 Anyway its free software so I shouldn't be critical.
Here are the errors the compile is generating

Hint: Start of reading config file C:\lazarus\fpc\2.6.0\bin\i386-win32\fpc.cfg
Hint: End of reading config file C:\lazarus\fpc\2.6.0\bin\i386-win32\fpc.cfg
Free Pascal Compiler version 2.6.0 [2012/03/14] for i386
Copyright (c) 1993-2011 by Florian Klaempfl and others
Target OS: Win32 for i386
Compiling autoupdatetest.lpr
Compiling autoupdateexu.pas
Compiling autoupdate.pas
Compiling progressdialog.pas
progressdialog.pas(26,17) Warning: An inherited method is hidden by "constructor TProgressBarDialog.create(AnsiString,AnsiString,LongInt="100");"
progressdialog.pas(28,29) Hint: Parameter "sender" not used
progressdialog.pas(69,4) Fatal: Can't open include file "progressdialog.lrs"

[BeniBela]

it won't compile without the lrs file and saving the lfm does not help.

It only creates it, if the form or unit is changed.
Resizing it slightly or adding a space to the corresponding pas does the trick


But now I also have put my lrs in the archive

What is unpleasant about lazarus
With lazarus the exe is reasonably portable to other folders on a PC and even to other devices via a cross compile EX winCE. But that's where frustration free portability ends. The IDE environment is very quirky with dependencies outside the folder the IDE runs in. This means any outside dependency Ex.users\userxx\appdata\local\lazarus must be backed up and deleted and a clean install done just to avoid being tripped up by hidden dependencies. Even after doing all that there is no certainty some vestige dependency still persists short of installing on a new PC that never ever had lazarus installed. This is further compounded when others seek to supply ide compatible source code ... if their hidden dependencies are different than others hidden dependencies then issues arise.

Things like that is why I don't use Lazarus to manage the dependencies, but simply put all paths in fpc.cfg

[picstart]

BeniBeta, It is very nice that you are attentive to these issues. The lrs is now resolved but it then allowed the opportunity for another issue to surface.
FYI Compiler errors with

w32internetaccess.pas(118,40) Error: Incompatible type for arg no. 1: Got "LongWord", expected "LPDWORD"

Thank you 

[BeniBela]

Interesting, fpc has got a wininet header. When did that happens?


But it is incompatible to the Delphi wininet header, and I used that, so you need to download the Delphi wininet header (e.g. you can google for "unit wininet" inurl:wininet.pas) to compile the wininet part on Windows.

[BeniBela]

Xidel:

I made a command line tool with the Internet Tools, which you can use to extract web page data without actually writing a program.

E.g. print the latest stable fpc release:

Code: [Select]

xidel http://freepascal.org/download.var -e //b[1]
Or the title/authors/reply-count of all forum posts of all first pages of all main forums here:

Code: [Select]

xidel "http://forum.lazarus.freepascal.org/index.php?action=forum" -f "css('td.info a.subject')" -e "<TABLE class='table_grid'><TR><TD><A>{title := text()}</A><A>{author := .}</A></TD><TD>{replies := filter(text(), '(.*) Replies', 1)}</TD></TR>*</TABLE>?"

Or including child boards:

Code: [Select]

xidel "http://forum.lazarus.freepascal.org/index.php?action=forum" -f "css('td.info a.subject')"  -e "<table class='table_list'><tr><a>{_follow:=@href}</a></tr>*</table>?<TABLE class='table_grid'><TR><TD><A>{title := text()}</A><A>{author := .}</A></TD><TD>{replies := filter(text(), '(.*) Replies', 1)}</TD></TR>*</TABLE>?"

[BeniBela]

Now it also supports all of XQuery (except xml schemas, static type checking and error codes)

And all function return now ref-counted interfaces, so you don't have to free anything.

Together, you can e.g. calculate all primes with:

Code: [Select]

  uses simpleinternet;
 [...]
  var v: IXQValue;
 [...]
  for v in process('',
    'xquery version "1.0";'                                +
    'declare function local:isprime($p){'                  +
    '  every $i in 2 to $p - 1 satisfies ($p mod $i != 0)' +
    '};'                                                   +
    'for $i in 2 to 30 where local:isprime($i) return $i') do
    writeln(v.toString);



[BeniBela]

I added JSONiq, now you can use JSON within XQuery:
E.g. to get the tenth element of an three objects {prop1: {prop2: {prop3: [..]..}..}..}

Code: [Select]

uses xquery_json;
writeln(TXQueryEngine.evaluateStaticXPath2('json("file:///path.to.json")("prop1")("prop2")("prop3")(10)').toString);

Although JSONiq's syntax it is kind of ugly, it still supports my old custom object syntax:

Code: [Select]

uses xquery_json;
writeln(TXQueryEngine.evaluateStaticXPath2('json("file:///path.to.json").prop1.prop2.prop3(10)').toString);


[BeniBela]

I added a new backend besides synapse and wininet: The Apache HTTPComponents of Android, called with JNI.

 

Incredible how large this is getting, it already has almost 50 000 lines, and 20 000 test cases. (and 10 000€ lost savings by spending so much time on it)