* * *

Author Topic: Internet Tools, now with more http  (Read 28764 times)

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with arbitrary precision bcd floating point math
« Reply #15 on: November 19, 2013, 12:17:55 am »
Since fpc's float functions are so horrible inaccurate, I added an arbitrary precision bcd float library.

It supports numbers between 10^-19327352814 and 10^19327352814 with 4831838208  decimal digit precision and the usual operators. It also has exact native float to string conversion functions.

edit: should have mentioned that I also rewrote the type system to be able to store a xml schema  (meaning it does not support xml schema yet, but you could add it with only new source without changing anything of the existing source), and added two operators from XPath/XQuery 3.
« Last Edit: November 20, 2013, 06:02:05 pm by BeniBela »

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with anonymous and higher order functions
« Reply #16 on: February 08, 2015, 10:27:19 pm »
I have added some features of XPath/XQuery 3.

Most notable anonymous and higher order functions.

E.g. you can calculate the Fibonacci numbers very slowly:

Code: [Select]
process('', 'let $fibhf := function($fibhf, $i) {'+
            '                if ($i <= 2) then 1 else $fibhf($fibhf, $i - 2) + $fibhf($fibhf, $i - 1)' +
            '              }, ' +
            '    $fib := $fibhf($fibhf, ?)' +
            'return for-each(1 to 10, $fib)') 

Or not so slowly:

Code: [Select]
process('', 'let $fibhf := function($fibhf, $count, $prevprev, $prev) { '+
            '                if ($count eq 0) then ($prevprev)'+
            '                else $fibhf($fibhf, $count - 1, $prev, $prevprev + $prev)'+
            '              }, '+
            '    $fib := $fibhf($fibhf, ?, 0, 1)' +
            'return for-each(1 to 1000, $fib)')
« Last Edit: February 08, 2015, 10:30:15 pm by BeniBela »

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with anonymous and higher order functions
« Reply #17 on: July 05, 2015, 05:59:17 pm »
It does now support all the XQuery 3.0 control expressions:

The W3C keeps making this syntax stranger and stranger. There is the "tumbling window":

Code: [Select]
  process('', 'for tumbling window $window in 0 to 10 start at $j when true() end at $k when $k - $j = 2 return sum($window)')

Which would calculates the sum of three adjacent items in the input sequence: t 0+1+2, 3+4+5, 6+7+8, ...

And there is the "sliding window":

Code: [Select]
  process('', 'for sliding window $window in 0 to 10 start at $j when true() end at $k when $k - $j = 2 return sum($window)')

Which would calculates the sum of three adjacent  items in the input sequence,  but without skipping any: 0+1+2, 1+2+3, 2+3+4, 3+4+5, 5+6+7,...

There is "group by":

Code: [Select]
  process('', 'for $i in 0 to 10 group by $mod := $i mod 2 return sum($i)')

Which would calculate two sums, the sum of all even and the sum of all odd numbers.

And finally there is try/catch, which does exactly the same as try/except in Pascal. Except that it is an expression instead a statement:

Code: [Select]
  process('', 'try { 0 div 0 } catch Q{http://www.w3.org/2005/xqt-errors}FOAR0001  { "invalid" } ')

Which catches the division by zero error and returns the string 'invalid' instead

halim

  • New member
  • *
  • Posts: 11
Re: Internet Tools, now with more XQuery 3.0
« Reply #18 on: July 06, 2015, 12:19:04 am »
i'm new with lazarus..i've try to install internettools but i got error
lazarus.pp(1,1) Fatal: Cannot find internettools used by Lazarus, incompatible ppu=C:\lazarus\components\pl_internettools\lib\i386-win32\internettools.ppu, package internettools
what should i do??

(sorry for my english)

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with more XQuery 3.0
« Reply #19 on: July 07, 2015, 01:04:05 am »
Strange



Did it compile correctly? Open internettools.lpk and just click "compile", not "use" or installation to test that

 


What fpc / Lazarus version do you use?
« Last Edit: July 07, 2015, 01:08:31 am by BeniBela »

halim

  • New member
  • *
  • Posts: 11
Re: Internet Tools, now with more XQuery 3.0
« Reply #20 on: July 07, 2015, 02:05:44 am »
i'm currently using FPC 2.6.4 and Lazarus IDE v.1.40
there's no error while compile all necessary file..but error raised when it finished
yes, i'm only open internettools.lpk

thank you for reply benibela..

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with more XQuery 3.0
« Reply #21 on: July 07, 2015, 07:39:21 pm »
Seems you need to remove ";\home\benito\hg\components\lazarus\dialogs" from the include path options of the package, before installing it...

Although Lazarus is right when it says, there is no need to install it, since there are no visual components.


eny

  • Hero Member
  • *****
  • Posts: 1553
Re: Internet Tools, now with more XQuery 3.0
« Reply #23 on: November 10, 2015, 10:46:23 pm »
All posts based on: Win10 (Win64); Lazarus 1.4.2 'stable' (#49524 win32) unless specified otherwise...

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with more XQuery 3.0
« Reply #24 on: November 10, 2015, 11:33:32 pm »

It seems the  XQuery layer of the Internet Tools works so well at interacting with webpages that the main problem is now not to pass the data to webpages, but to pass it to the XQuery layer.

So I have added a query function that converts its open array parameter to XQuery variables $_1, $_2. Furthermore there are chainable map/filter/query methods with the same behavior, where you can also access an old sequence with $_ or a single value of the sequence with .

For example, if you have an IXQValue containing a number, you can multiply it with 100 and add 17 like this:

Code: Pascal  [Select]
  1. uses xquery;
  2. xqvalue(123)
  3.    .map('. * $_1 + $_2', ['100', '17'] )
  4.    .toInt64
  5.  

and get 12317.


Or, if you want to get a subsequence of all numbers from 10 downto 1 you can do

Code: Pascal  [Select]
  1.   uses xquery;
  2.   query('1 to $_1', ['10'])
  3.     .order('$_ descending')
  4.     .query('subsequence($_, $_1, $_2)', ['3', '5'])
  5.     .toJoinedString()
  6.  

which becomes 8 7 6 5 4

Or, if you have a string containing a list of numbers, want to remove all odd numbers, and get the sum of the smallest three numbers, you can do the following:


Code: Pascal  [Select]
  1.   uses xquery;
  2.   xqvalue('1 43 112 31 14 124 12 10 129')
  3.     .query('tokenize($_, " ")')
  4.     .map('number()')
  5.     .filter('. mod 2 = 0')
  6.     .order()
  7.     .filter('position() <= 3')
  8.     .query('sum($_)')
  9.     .toString()
  10.  

and get 36.

If the value is a reference to a webpage, you can use the retrieve method to download and parse it.
For example, if you want to post in this thread, you can do:

Code: Pascal  [Select]
  1. uses xquery, xquery_json, synapseinternetaccess;
  2. begin
  3.   xqvalue('http://forum.lazarus.freepascal.org/index.php?action=login').retrieve()
  4.     .map('form(id("frmLogin"), {"user": $_1, "passwrd": $_2})', ['YOUR-USERNAME', 'YOUR-PASSWORD']).retrieve();
  5.  
  6.   xqvalue('http://forum.lazarus.freepascal.org/index.php?topic=13786').retrieve()
  7.     .map('//a[contains(@href, "action=post")]').get(1).retrieve()
  8.     .map('form(//form[contains(@action, "post")], {"message": $_1})', ['Test !!!']).retrieve();
  9.    
  10. end;
  11.  

Since this is much more flexible, the process function from all the examples above, should no longer be used. The old

Code: [Select]
process('webpage', 'query')

can be replaced by one of

Code: Pascal  [Select]
  1. query('doc("webpage") query')

Code: Pascal  [Select]
  1. query('doc($_1) query', ['webpage'])

Code: Pascal  [Select]
  1. xqvalue('webpage').retrieve().map('query')


christensen

  • Full Member
  • ***
  • Posts: 124
Re: Internet Tools, now with a functional LINQ-like interface
« Reply #25 on: November 28, 2015, 08:11:02 pm »
Hi,

I'm trying to access a webpage and submit some information but i can't.

The webpage is like:

Code: Pascal  [Select]
  1. <FORM ACTION = "/192.168.0.1/index" METHOD = "post">
  2. <select name="MyPage" size="15">
  3.  
  4.  
  5. </select><br><INPUT TYPE = submit NAME="Submit" VALUE="Stop"><br>
  6.  
  7. <INPUT TYPE = submit NAME="Submit" VALUE="Refresh"><br>
  8.  
  9. File:<INPUT NAME="File" VALUE="" SIZE="50"><br>
  10.  
  11. Params:<INPUT NAME="Param" VALUE="" SIZE="40"><br>
  12.  
  13. <INPUT TYPE = submit NAME="Submit" VALUE="Create"><br>
  14.  
  15. </FORM>
  16.  

I want to submit in two steps, at first 'File' then submit.
the second step: 'File' + 'Param' then submit.

How could i do this? i've been trying a few times but no result.

Lazarus 1.4.4, FPC 2.6.4, Windows 7 64bit, AMD Athlon 7750 black edition, Asus M3N78-CM
Lenovo L540 i7-4702MQ, Windows 10 x64,Lazarus 1.6.2,FPC 3.0

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with a functional LINQ-like interface
« Reply #26 on: November 30, 2015, 07:21:51 pm »
I'm trying to access a webpage and submit some information but i can't.

Why not? Believe in yourself. You can do it  ;D

Code: Pascal  [Select]
  1. <FORM ACTION = "/192.168.0.1/index" METHOD = "post">
  2. <select name="MyPage" size="15">
  3.  
  4.  
  5. </select><br><INPUT TYPE = submit NAME="Submit" VALUE="Stop"><br>
  6.  
  7. <INPUT TYPE = submit NAME="Submit" VALUE="Refresh"><br>
  8.  
  9. File:<INPUT NAME="File" VALUE="" SIZE="50"><br>
  10.  
  11. Params:<INPUT NAME="Param" VALUE="" SIZE="40"><br>
  12.  
  13. <INPUT TYPE = submit NAME="Submit" VALUE="Create"><br>
  14.  
  15. </FORM>
  16.  

That is a strange form. The url looks like an ip, but is an directory; and the file is a text field, not a file?


I want to submit in two steps, at first 'File' then submit.
the second step: 'File' + 'Param' then submit.

Like this:

Code: [Select]
var page: IXQValue;
begin
  page := xqvalue('http://....').retrieve();
  page.map('form(//form, {"File": "...", "Submit": "Create"})').retrieve();
  page.map('form(//form, {"File": "...", "Param": "..", "Submit": "Create"})').retrieve();

or

Code: [Select]
var page: IXQValue;
begin
  page := xqvalue('http://....').retrieve();
  page.map('form(//form, {"File": $_1, "Submit": "Create"})', ['...']).retrieve();
  page.map('form(//form, {"File": $_1, "Param": $_2, "Submit": "Create"})', ['..', '..']).retrieve();

Get the page containing the <form> and call the form() function of it with the relevant values

christensen

  • Full Member
  • ***
  • Posts: 124
Re: Internet Tools, now with a functional LINQ-like interface
« Reply #27 on: December 01, 2015, 10:22:53 pm »
Thanks, now i've solved in this way:

Code: Pascal  [Select]
  1. procedure TfrmMain.btnConnectClick(Sender: TObject);
  2.  var
  3.      URL: string;
  4.          Params: string;
  5.          Response: TMemoryStream;
  6.        
  7. begin
  8.      
  9.    Response := TMemoryStream.Create;
  10.  
  11.   try
  12.     URL := '/192.168.0.1/index';
  13.  
  14.     Params := 'File=' + EncodeURLElement('c:\sharing_enable.bat') + '&Submit=Create';
  15.  
  16.     if HttpPostURL(URL, Params, Response) then
  17.  
  18.       Response.SaveToFile('d:\response.txt');
  19.  
  20.   finally
  21.     Response.Free;
  22.   end;
  23.  
  24.     end;  

Via this FORM i can access a work station and execute processes, or copy files ...
Lazarus 1.4.4, FPC 2.6.4, Windows 7 64bit, AMD Athlon 7750 black edition, Asus M3N78-CM
Lenovo L540 i7-4702MQ, Windows 10 x64,Lazarus 1.6.2,FPC 3.0

christensen

  • Full Member
  • ***
  • Posts: 124
Re: Internet Tools, now with a functional LINQ-like interface
« Reply #28 on: December 11, 2015, 10:30:18 pm »
Hi,

Now i'm stuck with the field value reading.

Code: Pascal  [Select]
  1. <INPUT NAME="first" VALUE="1111">
  2. <INPUT NAME="second" VALUE="2222">
  3. <INPUT NAME="third" VALUE="3333">
  4. <INPUT NAME="fourth" VALUE="4444">
  5.  

for instance i want to read from a webpage the value from each fields.

and the second issue is reading radio buttons, which is active:
Code: Pascal  [Select]
  1. <input type="radio" name="6" value="1" >FirstRB<br>
  2.  
  3. <input type="radio" name="6" value="2" checked="checked">SecondRB<br>
  4.  
  5. <input type="radio" name="6" value="3" >ThirdRB<br>
  6.  

i tryied to sort out but now positive result :(


« Last Edit: December 11, 2015, 11:03:49 pm by christensen »
Lazarus 1.4.4, FPC 2.6.4, Windows 7 64bit, AMD Athlon 7750 black edition, Asus M3N78-CM
Lenovo L540 i7-4702MQ, Windows 10 x64,Lazarus 1.6.2,FPC 3.0

BeniBela

  • Hero Member
  • *****
  • Posts: 555
    • homepage
Re: Internet Tools, now with a functional LINQ-like interface
« Reply #29 on: December 12, 2015, 10:52:39 pm »
If the inputs are in a form with post action, you can use the form  command from above.

Code: [Select]
uses xquery;
xqvalue('http://your-site').retrieve().map('form(//form).post').toString

which would give a string first=1111&second=2222&third=3333&fourth=4444&6=2

Then you have the problem of splitting it up. You could do that like

Code: [Select]
uses xquery, xquery_json;
obj := xqvalue('http://your-site').retrieve().map('{| tokenize(form(//form).post, "&amp;") ! {substring-before(., "="): substring-after(., "=") } |}');
obj.getProperty("first").toString
obj.getProperty("second").toString

Or get a single input directly:

Code: [Select]
uses xquery, xquery_json;
page := xqvalue('http://your-site').retrieve();
page.map('//input[@name="first"]/@value').toString
page.map('//input[@name="second"]/@value').toString

Similar for the radio

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus