* * *

Author Topic: Internet Tools, now with more http  (Read 27132 times)

christensen

  • Full Member
  • ***
  • Posts: 124
Re: Internet Tools, with a functional LINQ-like interface
« Reply #45 on: December 17, 2015, 08:26:11 pm »
I got: 'File not open' and i don't know why??
EDIT:  this made the error: writeln

i've solved with tmemo:

Code: Pascal  [Select]
  1. page := xqvalue('http://192.168.0.1/index').retrieve();
  2. Memo1.Append(page.map('form(//form,{"5":"1","6":"1","7":"2","8":"1","9":"1","button":"set"})').debugAsStringWithTypeAnnotation());
  3. Memo1.Append(page.map('form(//form,{"6":"1","7":"2","8":"1","9":"1","button":"set"})').debugAsStringWithTypeAnnotation());
  4.  

got:

object: {method: string: GET, url: string: http://192.168.0.1/index?5=1&6=1&7=2&8=1&9=1&button=set}
object: {method: string: GET, url: string: http://192.168.0.1/index?5=1&6=1&7=2&8=1&9=1&button=set}

The only difference between form submit and my project is the submit order:

on the form is like:
 http://192.168.0.1/index?&button=set&5=1&6=1&7=2&8=1&9=1

and on my project:
http://192.168.0.1/index?5=1&6=1&7=2&8=1&9=1&button=set


EDIT:

found it:

i've tried to submit in the ie address bar like this:

http://192.168.0.1/index?5=1&6=4&7=4&8=4&9=4&button=set

and it not working, only like this:

http://192.168.0.1/index?button=set&5=1&6=4&7=4&8=4&9=4

but if i modify the order:
Code: Pascal  [Select]
  1. Memo1.Append(page.map('form(//form,{"button":"set","6":"1","7":"2","8":"1","9":"1"})').debugAsStringWithTypeAnnotation());
  2.  
will be the same:

Code: Pascal  [Select]
  1. object: {method: string: GET, url: string: http://192.168.0.1/index?5=1&6=1&7=2&8=1&9=1&button=set}

If i submit the button alone:

Code: Pascal  [Select]
  1. Memo1.Append(page.map('form(//form,{"button":"set})').debugAsStringWithTypeAnnotation());
  2.  

I will get:
Code: Pascal  [Select]
  1. object: {method: string: GET, url: string: http://192.168.0.1/index?5=1&6=1&7=2&8=1&9=1&button=set}

i've tried this way, with INPUT:

Code: Pascal  [Select]
  1. Memo1.Append(page.map('form(//input,{"button":"set","6":"1","7":"2","8":"1","9":"1"})').debugAsStringWithTypeAnnotation());
  2.  

the reply is correct, but is executing 22times :(

Code: Pascal  [Select]
  1. object: {method: string: GET, url: string: http://192.168.0.1/index?button=set&5=1&6=1&7=2&8=1&9=1}

« Last Edit: December 17, 2015, 09:26:14 pm by christensen »
Lazarus 1.4.4, FPC 2.6.4, Windows 7 64bit, AMD Athlon 7750 black edition, Asus M3N78-CM
Lenovo L540 i7-4702MQ, Windows 10 x64,Lazarus 1.6.2,FPC 3.0

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, with a functional LINQ-like interface
« Reply #46 on: December 17, 2015, 11:52:55 pm »
but if i modify the order:
Code: Pascal  [Select]
  1. Memo1.Append(page.map('form(//form,{"button":"set","6":"1","7":"2","8":"1","9":"1"})').debugAsStringWithTypeAnnotation());
  2.  
will be the same:[/code]

Because it reorders the fields to match the inputs on the page

All inputs, except buttons that is

I uploaded a new version that orders buttons, too

Code: Pascal  [Select]
  1. Memo1.Append(page.map('form(//input,{"button":"set","6":"1","7":"2","8":"1","9":"1"})').debugAsStringWithTypeAnnotation());
  2.  

the reply is correct, but is executing 22times :(

 :o

Yes, once for every input

You could use //html once for the webpage.  Although it is not supposed to work with anything, but forms. Nor is the order guaranteed

christensen

  • Full Member
  • ***
  • Posts: 124
Re: Internet Tools, with a functional LINQ-like interface
« Reply #47 on: December 18, 2015, 05:39:02 pm »
it work like a charm! THANKS

about the feedback of executing is not clear to me.

i need to submit four editbox but each indepedently. on the form i have button for each. if i send it at one time all the only first two fields are filed.


i want to submit from a single button but in order, after executin one try the second one, if the second is ok than try the 3rd one and soo on.

something like boolean ....
Lazarus 1.4.4, FPC 2.6.4, Windows 7 64bit, AMD Athlon 7750 black edition, Asus M3N78-CM
Lenovo L540 i7-4702MQ, Windows 10 x64,Lazarus 1.6.2,FPC 3.0

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, with a functional LINQ-like interface
« Reply #48 on: December 18, 2015, 08:20:01 pm »
about the feedback of executing is not clear to me.

Retrieve returns the page you get

Then it depends on what the server has sent as answer


i need to submit four editbox but each indepedently. on the form i have button for each. if i send it at one time all the only first two fields are filed.

Code: [Select]
    page := xqvalue('http://192.168.0.1/index').retrieve();
    answer := page.map('form(//form,{"button":"btn1"})').retrieve();
    if not answer.map(' ... some test ... ').toBooleanEffective then raise exception.create('...');
    answer := page.map('form(//form,{"button":"btn2"})').retrieve();
    if not answer.map(' ... some test ... ').toBooleanEffective then raise exception.create('...');
    ...

christensen

  • Full Member
  • ***
  • Posts: 124
Re: Internet Tools, with a functional LINQ-like interface
« Reply #49 on: December 20, 2015, 10:44:44 am »
Thanks, i solved by converting the server replay to boolean and than compare it....

Very helpful component and you helped me alot.
I've almost finish my project, now i just need to deal with java class.

Thank you!

Lazarus 1.4.4, FPC 2.6.4, Windows 7 64bit, AMD Athlon 7750 black edition, Asus M3N78-CM
Lenovo L540 i7-4702MQ, Windows 10 x64,Lazarus 1.6.2,FPC 3.0

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, now with complete XPath 2, FLRE and math layer
« Reply #50 on: February 07, 2016, 11:53:07 pm »
And it is done!

At least the XPath 2 mode, in which it should now pass all relevant test cases in the XQuery test suite.

It now uses FLRE (from here), because poor little regexpr had no chance at passing those tests.  (although even FLRE combined with my own regular expression pre parser still fails 8 test cases. But I hope Bero will fix that soon).
Anyways, you can still set a define to use regexpr.

And I added a unit xquery_module_math, which implements the math module of XPath/XQuery 3.

If you ever needed to calculate the cosine of a webpage, you can do it now:

Code: [Select]
uses xquery, xquery_module_math;
begin
  registerModuleMath;
  writeln(query('math:cos(avg(string-to-codepoints(doc("http://example.org"))))').toString);
end.

It is 0.0038171526409082005. Seems the webpage has a very small angle. edit: of course not. I am stupid. it is a very big angle. Almost rectangular
« Last Edit: February 08, 2016, 11:56:11 am by BeniBela »

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
You can now convert numbers to strings.

 It follows the W3C rules exactly. I did not want to implement it at first, because they are just too silly, but then I was traveling and did not have anything better to do..

The easy case, converting floating point numbers to strings:

Code: Pascal  [Select]
  1. query('format-number(7.47, "00.0000000000")')    
  2.  

becomes '07.4700000000'

And you can convert integers, too, which is much more complex:

The simple case

Code: Pascal  [Select]
  1. query('format-integer(123, "0")')
  2.  

becomes '123'

Or with separators

Code: Pascal  [Select]
  1. query('format-integer(123, "0-0:0-0:0-0:0-0")')
  2.  

for '0-0:0-0:0-1:2-3'

Or with other digits

Code: Pascal  [Select]
  1. query('format-integer(1234567, "٠")')  // ١٢٣٤٥٦٧    (that's arabic-indic)
  2. query('format-integer(1234567, "𝟘")')  // ۱۲۳۴۵۶۷    (this too)
  3. query('format-integer(1234567, "𝟘")')  // 𝟙𝟚𝟛𝟜𝟝𝟞𝟟
  4. query('format-integer(1234567, "𝟎")')  // 𝟏𝟐𝟑𝟒𝟓𝟔𝟕
  5. query('format-integer(1234567, "߀")')  // ߁߂߃߄߅߆߇    (this is nko)
  6. query('format-integer(1234567, "೦")')  // ೧೨೩೪೫೬೭    (kannada. does not have anything to do with canada)
  7. query('format-integer(1234567, "꧐")')  // ꧑꧒꧓꧔꧕꧖꧗    (japanase)
  8. query('format-integer(1234567, "༠")')  // ༡༢༣༤༥༦༧    (tibetian)
  9.  
(edit: seems the forum converts  the stranger digits to entities:  http://www.fileformat.info/info/unicode/char/1d7d8/mathematical_doublestruck_digit_zero.png )

but it gets better!

Code: Pascal  [Select]
  1. query('format-integer(1234567, "a")') //brfgi      (for a table column)
  2. query('format-integer(1234567, "i")') //mmmmm...mdlxvii      (roman, with 1234 m's)
  3. query('format-integer(1234567, "w")') //one million two hundred thirty-four thousand five hundred sixty-seven
  4. query('format-integer(1234567, "w;o")') //one million two hundred thirty-four thousand five hundred sixty-seventh (note the th)
  5. query('format-integer(1234567, "Ww", "de")') //Eine Million Zweihundertvierunddreißigtausendfünfhundertsiebenundsechzig (it's German)
  6. query('format-integer(1234567, "Ww;o", "de")') //Eine Million Zweihundertvierunddreißigtausendfünfhundertsiebenundsechzigste (German female number)
  7. query('format-integer(1234567, "Ww;o(-er)", "de")') //Eine Million Zweihundertvierunddreißigtausendfünfhundertsiebenundsechzigster (German male number)
  8.  

it can handle large numbers too:

Code: Pascal  [Select]
  1. query('format-integer(9223372036854775807, "w")') //nine quintillion two hundred twenty-three quadrillion three hundred seventy-two trillion thirty-six billion eight hundred fifty-four million seven hundred seventy-five thousand eight hundred seven
  2.  

oh, that is tiny. Perhaps this is big?

Code: Pascal  [Select]
  1. query('format-integer(1E200, "w")') //one hundred quinquasexagintillion
  2.  

No, that is still rather small.

Let's go BIG with a factorial:

Code: Pascal  [Select]
  1. query('format-integer(x:product(1 to 100000), "w")') //two duoquinquagintacentillinonagintacentillion eight hundred twenty-four duoquinquagintacentillinovemoctogintacentillion two hundred twenty-nine duoquinquagintacentillioctooctogintacentillion four hundred seven duoquinquagintacentilliseptemoctogintacentillion nine hundred sixty duoquinquagintacentillisexoctogintacentillion three hundred forty-seven duoquinquagintacentilliquinquaoctogintacentillion eight hundred seventy-four duoquinquagintacentilliquattoroctogintacentillion two hundred ninety-three duoquinquagintacentillitresoctogintacentillion four hundred twenty-one duoquinquagintacentilliduooctogintacentillion five hundred seventy-eight duoquinquagintacentilliunoctogintacentillion twenty-four duoquinquagintacentillioctogintacentillion five hundred thirty-five duoquinquagintacentillinovenseptuagintacentillion five hundred eighteen duoquinquagintacentillioctoseptuagintacentillion four hundred seventy-seven duoquinquagintacentilliseptenseptuagintacentillion four hundred ninety-four duoquinquagintacentilliseseptuagintacentillion nine hundred twenty-six duoquinquagintacentilliquinquaseptuagintacentillion ninety-one duoquinquagintacentilliquattorseptuagintacentillion two hundred twenty-four duoquinquagintacentillitreseptuagintacentillion eight hundred fifty duoquinquagintacentilliduoseptuagintacentillion five hundred seventy-eight duoquinquagintacentilliunseptuagintacentillion nine hundred eighteen duoquinquagintacentilliseptuagintacentillion eighty-six duoquinquagintacentillinovensexagintacentillion five hundred forty-two duoquinquagintacentillioctosexagintacentillion nine hundred seventy-seven duoquinquagintacentilliseptensexagintacentillion nine hundred fifty duoquinquagintacentillisesexagintacentillion nine hundred one duoquinquagintacentilliquinquasexagintacentillion sixty-three duoquinquagintacentilliquattorsexagintacentillion seventeen duoquinquagintacentillitresexagintacentillion eight hundred seventy-two duoquinquagintacentilliduosexagintacentillion five hundred fifty-one duoquinquagintacentilliunsexagintacentillion seven hundred seventy-one duoquinquagintacentillisexagintacentillion four hundred thirteen duoquinquagintacentillinovenquinquagintacentillion eight hundred thirty-one duoquinquagintacentillioctoquinquagintacentillion one hundred sixty-three duoquinquagintacentilliseptenquinquagintacentillion six hundred ten duoquinquagintacentillisesquinquagintacentillion seven hundred thirteen duoquinquagintacentilliquinquaquinquagintacentillion six hundred eleven duoquinquagintacentilliquattorquinquagintacentillion seven hundred thirty-seven duoquinquagintacentillitresquinquagintacentillion three hundred sixty-one duoquinquagintacentilliduoquinquagintacentillion nine hundred sixty-two duoquinquagintacentilliunquinquagintacentillion nine hundred fifty-one duoquinquagintacentilliquinquagintacentillion four hundred seventy-four duoquinquagintacentillinovenquadragintacentillion nine hundred ninety-six duoquinquagintacentillioctoquadragintacentillion one hundred eighty-three duoquinquagintacentilliseptenquadragintacentillion one hundred twenty-three duoquinquagintacentillisesquadragintacentillion nine hundred eighteen duoquinquagintacentilliquinquaquadragintacentillion twenty-two duoquinquagintacentilliquattorquadragintacentillion seven hundred twenty-six duoquinquagintacentillitresquadragintacentillion seventy-three duoquinquagintacentilliduoquadragintacentillion four hundred nine duoquinquagintacentilliunquadragintacentillion ninety-three duoquinquagintacentilliquadragintacentillion eight hundred thirty-two duoquinquagintacentillinoventrigintacentillion four hundred twenty-two duoquinquagintacentillioctotrigintacentillion five duoquinquagintacentilliseptentrigintacentillion five hundred fifty-six duoquinquagintacentillisestrigintacentillion nine hundred sixty-eight duoquinquagintacentilliquinquatrigintacentillion eight hundred sixty-six duoquinquagintacentilliquattortrigintacentillion seven hundred eighty-four duoquinquagintacentillitrestrigintacentillion thirty-eight duoquinquagintacentilliduotrigintacentillion thirty-seven duoquinquagintacentilliuntrigintacentillion seven hundred thirty-seven duoquinquagintacentillitrigintacentillion nine hundred forty-four duoquinquagintacentillinovemviginticentillion four hundred ninety-six duoquinquagintacentillioctoviginticentillion one hundred twenty-six duoquinquagintacentilliseptemviginticentillion eight hundred thirty-eight duoquinquagintacentillisesviginticentillion fourteen duoquinquagintacentilliquinquaviginticentillion seven hundred eighty-seven duoquinquagintacentilliquattorviginticentillion five hundred eleven duoquinquagintacentillitresviginticentillion one hundred ninety-six duoquinquagintacentilliduoviginticentillion six hundred ninety duoquinquagintacentilliunviginticentillion six hundred thirty-eight duoquinquagintacentilliviginticentillion six hundred four duoquinquagintacentillinovendecicentillion four hundred ninety-two duoquinquagintacentillioctodecicentillion six hundred fourteen duoquinquagintacentilliseptendecicentillion four hundred fifty-three duoquinquagintacentillisedecicentillion eight hundred eleven duoquinquagintacentilliquinquadecicentillion one hundred thirty-seven duoquinquagintacentilliquattordecicentillion nine duoquinquagintacentillitredecicentillion sixteen duoquinquagintacentilliduodecicentillion seventy-six duoquinquagintacentilliundecicentillion six hundred eighty-six duoquinquagintacentillidecicentillion six hundred forty duoquinquagintacentillinovencentillion five hundred forty duoquinquagintacentillioctocentillion seven hundred seventeen duoquinquagintacentilliseptencentillion fifty-six duoquinquagintacentillisexcentillion five hundred ninety-five duoquinquagintacentilliquinquacentillion two hundred twenty-six duoquinquagintacentilliquattorcentillion one hundred twenty-nine duoquinquagintacentillitrescentillion eight hundred four duoquinquagintacentilliduocentillion one hundred ninety-five duoquinquagintacentilliuncentillion eight hundred thirty-five duoquinquagintacentillicentillion six hundred seventy-seven duoquinquagintacentillinovenonagintillion eight hundred ninety duoquinquagintacentillioctononagintillion nine hundred four duoquinquagintacentilliseptenonagintillion seven hundred fifty-four duoquinquagintacentillisenonagintillion one hundred fifty-one duoquinquagintacentilliquinquanonagintillion two hundred eighty-seven duoquinquagintacentilliquattornonagintillion one hundred fourteen duoquinquagintacentillitrenonagintillion eighty-three duoquinquagintacentilliduononagintillion six hundred ninety-two duoquinquagintacentilliunnonagintillion four hundred twenty-five duoquinquagintacentillinonagintillion one hundred fifty-three duoquinquagintacentillinovemoctogintillion five hundred twenty-nine duoquinquagintacentillioctooctogintillion three hundred nine duoquinquagintacentilliseptemoctogintillion six hundred twenty-six duoquinquagintacentillisexoctogintillion sixty-seven duoquinquagintacentilliquinquaoctogintillion two hundred twenty-seven duoquinquagintacentilliquattoroctogintillion one hundred three duoquinquagintacentillitresoctogintillion eight hundred seventy-four duoquinquagintacentilliduooctogintillion four hundred twenty-four duoquinquagintacentilliunoctogintillion six hundred eight duoquinquagintacentillioctogintillion eight hundred sixty-three duoquinquagintacentillinovenseptuagintillion five hundred forty-five duoquinquagintacentillioctoseptuagintillion four hundred thirty-six duoquinquagintacentilliseptenseptuagintillion three hundred ninety-eight duoquinquagintacentilliseseptuagintillion two hundred ninety-three duoquinquagintacentilliquinquaseptuagintillion one hundred seventy-four duoquinquagintacentilliquattorseptuagintillion seven hundred seventy-six duoquinquagintacentillitreseptuagintillion one hundred seventy-seven duoquinquagintacentilliduoseptuagintillion ... four hundred ninety-six octilliduotrigintatrecentillion
  2.  

It seems the forum does not suppport large numbers. But I have uploaded the result for you


Also, if anyone remembers, the entire point of this library was to be a wrapper around a standard http library that throws an exception on non 2xx error codes.
But sometimes people want to download the actual error page, so the error code behavior is now customizable
« Last Edit: March 14, 2016, 11:53:18 am by BeniBela »

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, now with more XQuery 3
« Reply #52 on: June 06, 2016, 01:54:23 am »
It now supports all XQuery 3 features even silly ones like the uri-collection function or the "declare context item" statement, which brings the test suite conformance to 99.6%.



And the refactoring for the new strings of fpc 3 has begun: First step is to drop theo's utf8tools, and only copy some functions from it. This saves a few hundred kb in the executable (when using the unicode tables from FLRE), unless you are using them elsewhere, too...
« Last Edit: June 06, 2016, 01:56:15 am by BeniBela »

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, now with _faster_ XQuery 3
« Reply #53 on: November 27, 2016, 07:04:23 pm »
And I have made the query evaluation faster.

Hard to qualify how much faster. It depends a lot on the query. Perhaps 10% to 50% faster, if you have a query that returns a large number of elements, for example //a and there are 10000 links on the page . Mostly from removing the implicit exception blocks. They are lurking everywhere when you are using interfaces. But on specially crafted queries it becomes several orders of magnitudes  faster.

Also there is a new list of available functions inside queries: http://www.benibela.de/documentation/internettools/xpath-functions.html
« Last Edit: December 03, 2016, 02:31:12 am by BeniBela »

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, now with more http
« Reply #54 on: March 26, 2017, 02:39:07 pm »


From all the xquery, I have neglected the basic internet functionality.  Lots of things are becoming outdated. This update updates some of them:

  • Downloaded data is now returned through a callback function, so it can be written directly in the destination buffer without copying, no matter if the destination is a string, stream or file. (e.g. on my computer when the Internet Tools call Synapse to download a 100mb file from localhost into a string, it makes them  30 % faster than Synapse alone)
  • various url/hex encoding functions are merged into one and the default escaping is like in html5.
  • The cookie handling is updated to RFC 6265 and the cookie origin rules are actually enforced, i.e. a cookie set by one server will not be send to another server

CC

  • New member
  • *
  • Posts: 24
Re: Internet Tools, now with more http
« Reply #55 on: April 29, 2017, 08:53:36 pm »
BeniBela,

I am interested in the template  based usage of internettools.  A few examples would be really useful.  Right now I am trying to post a login form ...

 

« Last Edit: April 29, 2017, 09:23:04 pm by CC »

mgear

  • Newbie
  • Posts: 4
Re: Internet Tools, now with more http
« Reply #56 on: May 10, 2017, 01:48:14 am »
*** SOLVED ***, thanks :D

A added the lazutils package requirement and the LazUTF8 unit in uses section. Now the   entities transfrom into chr(160)

The problem was:

====================================

Great library, but I can't understand why IXQValue.toString "eats"   entities? For example, "4 may 16:48" becames "4 may16:48".  I redirected the output to file and saw it in the hex viewer, so it's not the console output issue. Is it wrong because of some options combination or it is the normal behaviour of this library?

My test code is (taken from the examples)
Code: Pascal  [Select]
  1. css := '*** some css3 here ***';
  2. html := '*** some html here ***';
  3. xqe := TXQueryEngine.Create;
  4. tp := TTreeParser.Create;
  5.  
  6. tp.readComments := True;
  7. tp.readProcessingInstructions := True;
  8. tp.parsingModel := pmHTML;
  9. tp.trimText := False;
  10. tp.autoDetectHTMLEncoding := True;
  11. tp.TargetEncoding := CP_UTF8;
  12. tp.parseTree(html, '', 'charset=utf-8');
  13. tp.getLastTree.setEncoding(CP_UTF8, false, true); // I tried the different boolean values here
  14.   for v in xqe.evaluateCSS3(css, tp.getLastTree) do
  15.   begin
  16.     WriteLn(v.toString);
  17.     WriteLn;
  18.     WriteLn(v.toNode.outerHTML);
  19.     WriteLn;
  20.   end;
  21.  

So, .outerHTML works as I expect, i.e. prints these nbsp's , but .toString doesn't
« Last Edit: May 10, 2017, 08:50:24 pm by mgear »

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Internet Tools, now with more http
« Reply #57 on: May 12, 2017, 12:39:16 am »
I am interested in the template  based usage of internettools.  A few examples would be really useful.  Right now I am trying to post a login form ...

The page.map('form(//form example above is a good example for posting. Or the 4th example of reply #24

The main purpose of the templates (I call them patterns now, except for the multipage templates) is to extract multiple, related values from the webpage at once and put each value in a different variable. To post a form you need the value of all the input elements together in a single POST request,  so if you use patterns, you would first split all the values in the different variables and then put the variables back together into one which seems counterproductive.


A added the lazutils package requirement and the LazUTF8 unit in uses section. Now the   entities transfrom into chr(160)


Then it must have written chr(160) as utf-8 in the string without setting the codepage to CP_UTF8.

Guess I need to test every string operation in every function if it still works with fpc 3


 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus