[solved] Python and Yahoo

[Nicole]

There is a possibility to grab data (stock quotes and much more) from Yahoo, by Python.
Am I right, that it is rather hard, to make Lazarus work with Python?
Or is it possbile?

Or is there anywhere a Lazarus way to say,
"dear yahoo, get me the close-quote of exasol of the 20 of may 2025").

As the task seems quite useful, there may be a prefab solution around.

I can see all those pleasant information shown by Yahoo. However, it takes some seconds to show up and I have to click two times.
This makes me pessimistic.

[gues1]

Look that. I use it in Delphi and it works. It's compatible with Lazarus.

https://github.com/pyscripter/python4delphi

[Warfley]

The Python implementation just queries the internal Yahoo finance API, its a simple REST API you can query using HTTP. For example to download the price history of a stock, you can look St the Python code: https://python-yahoofinance.readthedocs.io/en/latest/_modules/yahoofinance/historicaldata.html#HistoricalPrices

Code: Bash  [Select][+][-]

1. url = 'https://query1.finance.yahoo.com/v7/finance/download/{i}'
2.         r= requests.get(url.format(i=instrument),
3.             cookies={'B': cookie},
4.             params={
5.                 "period1": start_period,
6.                 "period2": end_period,
7.                 "interval": frequency,
8.                 "event": event,
9.                 "crumb": crumb
10.             }
11.         )
12.  
13.         self.prices = r.text

Hardly rocket science, only hurdle is that you need the Crumb and the cookie, which you can fetch by doing a "normal" request to the yahoo finance Main site (see Python function _find_cookie_crumb_pair)

That said, in many countries web scraping is not allowed if the service forbids it or places technical measures to avoid scraping. The cookie and crumble was implemented in 2017/because Yahoo explicitly wanted to stop automatic querying of their API. So you should check if in your jurisdiction it is legal. Here in Germany what you wantto do would be illegal, no matter if you do it in Python or pascal

[Nicole]

Thank you for the answers.

Do I need to install the Python plugin for the second solution?
If not, how to use it?
And are you sure, it works still?
I wasted so many days in implementing solutions to find out, they cannot work.

I am from Austria. Here the law is less restrictive than Germany.
And if it should be illegal never the less, - pls tell nobody, 

[paweld]

That said, in many countries web scraping is not allowed if the service forbids it or places technical measures to avoid scraping. The cookie and crumble was implemented in 2017/because Yahoo explicitly wanted to stop automatic querying of their API. So you should check if in your jurisdiction it is legal. Here in Germany what you wantto do would be illegal, no matter if you do it in Python or pascal

I wouldn't call using the API as “scraping.” That's what the API is exposed for, so that the user can download data for himself without any problems.

[Thaddy]

@Nicole
Why use python for that?

[Leledumbo]

There is a better (still unofficial and probably undocumented) way to query than the v7 API, which is the v8 API. This one requires no auth and can be publicly scraped (illegally, probably):

Code: Pascal  [Select][+][-]

1. {$mode objfpc}{$H+}
2.  
3. uses
4.   opensslsockets,
5.   fphttpclient,
6.   fpjson,
7.   jsonparser;
8.  
9. const
10.   {
11.     period1 = start
12.     period2 = end
13.     both are in Unix timestamp
14.   }
15.   URL = 'https://query2.finance.yahoo.com/v8/finance/chart/EXL.DE?period1=1747699200&period2=1747785600';
16.  
17. var
18.   J,M: TJSONData;
19. begin
20.   with TFPHTTPClient.Create(nil) do
21.     try
22.       RequestHeaders.Values['User-Agent'] := 'Mozilla/5.0'; // required, Yahoo will respond with HTTP 429 if this is empty
23.       J := GetJSON(Get(URL));
24.       M := J.FindPath('chart.result[0].meta');
25.       WriteLn(M.AsJSON);
26.     finally
27.       J.Free;
28.       Free;
29.     end;
30. end.
31.  

I'm not sure what you're looking for so I'm just taking the meta field, parse it further to get what you need.

[gues1]

That said, in many countries web scraping is not allowed if the service forbids it or places technical measures to avoid scraping. The cookie and crumble was implemented in 2017/because Yahoo explicitly wanted to stop automatic querying of their API. So you should check if in your jurisdiction it is legal. Here in Germany what you wantto do would be illegal, no matter if you do it in Python or pascal

I wouldn't call using the API as “scraping.” That's what the API is exposed for, so that the user can download data for himself without any problems.

If it is done without human intervention (for example with bots or automated tools) it is called "webscraping"

[paweld]

If it is done without human intervention (for example with bots or automated tools) it is called "webscraping"

I may be f***ed up, but in my opinion, at the core of web/rest api is process automation.

[Thaddy]

It is not scraping because it is a provided API and it needs just one (1) connection to run all day.
Frankly this is exactly how all banks and traders get their data. I know, because it payed my pension.
Still bemused why anybody needs Python for that: it is a rest based transfer protocol, iow language agnostic.

Note banks do get their data from specialist companies, guaranteeng ms (even ns resolution for local), not Yahoo, which is way too slow for traders, but the principle is the same. E.g. Reuters offers real-time ticks(trades or bids, volume may arrive out of sync) in ns resolution for NYSE and NASDAQ for companies that connect on local fiber and their specified hardware.
In this context ms means not 1 ms, it means in ms range upto 100 (not 1000), ns means ns range upto but not including 1 ms.

[gues1]

If it is done without human intervention (for example with bots or automated tools) it is called "webscraping"

I may be f***ed up, but in my opinion, at the core of web/rest api is process automation.

It is not scraping because it is a provided API and it needs just one (1) connection to run all day.
Frankly this is exactly how all banks and traders get their data. I know, because it payed my pension.

@Thaddy, good to know that with a pubblic API you can get the pension. But I think you would need more, like a account, 2FA security, OTP and others ... or not ? 
Web scraping is about "catch" free data public available without any security measures, like a public API of weather services or geolocation or scan a web site (but I'm sure not API bank  )

This in Italy (and in Europe too, I think) is permitted only for statistics purpose and without any economic scope (unless it is prohibited by the provider, but without protective measures they certainly can't take you to court)

[paweld]

Web scraping is about "catch" free data public available without any security measures, like a public API of weather services or geolocation or scan a web site (but I'm sure not API bank  )

From a web site: YES, this is web scraping. From API: NO.  And it doesn't matter whether these are public resources or not. What matters is the site use rules.

[Nicole]

@ Leledumbo
 

This works. You helped me to solve something, which kept me busy since weeks.
I gave the link to ChatGPT and it solved all there was to solve.
We brought the data into my interface-Var.
This I sent to my DB and voila...

for others who may seek it:
the ticker must have in this case an ".DE" attached.
And you must Yahoo tell, you are a browser and not some scrapping software. No way!

btw:
The abilities of ChatGPT are in a way, that it has the power to stop the world as we know it.
It is able to hack the communications and computers of this world, power off the tools and interchange communications.
It understands all languages of this world.

And interpreters, programmers, journalists, authors, artits, musicians...?
They are not necessary any more soon.

ChatGPT changes the world.

[gues1]

The abilities of ChatGPT are in a way, that it has the power to stop the world as we know it.
It is able to hack the communications and computers of this world, power off the tools and interchange communications.
It understands all languages of this world.

And interpreters, programmers, journalists, authors, artits, musicians...?
They are not necessary any more soon.

ChatGPT changes the world.

SkyNet is coming