Recent

Author Topic: How to download Yahoo Finance Data?  (Read 1507 times)

incendio

  • Sr. Member
  • ****
  • Posts: 269
How to download Yahoo Finance Data?
« on: December 15, 2022, 05:06:21 am »
Hello all,

Have any of you tried to download Yahoo Finance Data with Lazarus?

Any example codes will be helpful.

Thanks in advance.

KodeZwerg

  • Hero Member
  • *****
  • Posts: 2080
  • Fifty shades of code.
    • Delphi & FreePascal
Re: How to download Yahoo Finance Data?
« Reply #1 on: December 15, 2022, 07:24:58 am »
What data (...yes financial...), what link?
« Last Edit: Tomorrow at 31:76:97 xm by KodeZwerg »

wp

  • Hero Member
  • *****
  • Posts: 11923
Re: How to download Yahoo Finance Data?
« Reply #2 on: December 15, 2022, 12:08:45 pm »
I once wrote an small application for this: https://sourceforge.net/p/wp-laz/code/HEAD/tree/FinancialChart/. It gets the data from Yahoo, AlphaVantage and Google (maybe not working any more). Alphavantage requires an API-Key which you can get after free registration for the most basic data.

The problem with these free providers is that they tend to disappear sooner or later...

As for Yahoo, you must send the following http request and insert the correct parameters. In return, you get a csv file with date, open, high, low close, adjusted close and volume data.

Code: Pascal  [Select][+][-]
  1.   { URL for the download of the stock prices; uses the following parameters at
  2.     these indexes
  3.       0: Ticker symbol
  4.       1: API key
  5.       2: Start date as string
  6.       3: End date as string
  7.       4: ResolutionType (daily, weeky, monthly)
  8.   }
  9. const
  10.   YAHOO_URL := 'https://query1.finance.yahoo.com/v7/finance/download/'+
  11.     '%0:s?'+                        // ticker
  12.     'period1=%2:s'+                 // start date, unix date/time
  13.     '&period2=%3:s'+                // end date, unix date/time
  14.     '&interval=1%4:s'+              // 'd' (daily), 'w' (weekly), 'm' monthly
  15.     '&events=history'+
  16.     '&includeAdjustedClose=true';  
  17. var
  18.   url: String;
  19.   startDateStr, endDateStr: String;
  20.   apiKey: String;
  21.   stockSymbol: String;
  22.   resolution: String;
  23. ...
  24.   stockSymbol := 'MSFT';   // Microsoft
  25.   apiKey := '';
  26.   startDateStr := IntToStr(DateTimeToUnix(EdStartDate.Date));  // EdStartDate and EdEndDate are TDateEdit components
  27.   endDateStr := IntToStr(DateTimeToUnix(EdEndDate.Date));
  28.   resolution := 'd';  // daily, 'w' = weekly, 'm' = monthly
  29.   url := Format(YAHOO_URL, [stockSymbol, apiKey, startDateStr, endDateStr, resolution]); // do not change order!

You can use fphttpclient to send the request (on Windows, my application uses the built-in windows functions).
« Last Edit: December 15, 2022, 12:46:45 pm by wp »

Guva

  • Jr. Member
  • **
  • Posts: 84
Re: How to download Yahoo Finance Data?
« Reply #3 on: December 15, 2022, 12:14:57 pm »
well, everything is simple there. Here is an article true in Russian
https://habr.com/ru/post/505674/

AL

  • Sr. Member
  • ****
  • Posts: 264
Re: How to download Yahoo Finance Data?
« Reply #4 on: December 15, 2022, 03:36:54 pm »
Here is an example I found lately on the web. It is in VBA for Excel
I do not know why this "CookieCrumb" procedure is called, seems to be important.


Sub GetData()
    Dim InputControls As Worksheet
    Dim OutputData As Worksheet
    Dim Symbol As String
    Dim startDate As String
    Dim endDate As String
    Dim period As String
    Dim last As Double
    Dim OffsetCounter As Double
    Dim crumb As String
    Dim cookie As String
    Dim validCookieCrumb As Boolean
    Dim Result As Integer
   
    Application.ScreenUpdating = False
    Application.DisplayAlerts = False
    Application.Calculation = xlCalculationManual

    Set InputControls = Sheets("Sheet1")
    Set OutputData = Sheets("HistoricalData")
   
    With InputControls
        last = .Cells(.Rows.Count, "A").End(xlUp).Row
    End With
       
    ' Arguments
    startDate = (InputControls.Range("B4") - DateValue("1970-01-01")) * 86400
    endDate = (InputControls.Range("B5") - DateValue("1970-01-01")) * 86400
    period = "1d"
   
    Dateminus = -InputControls.Range("B4") + InputControls.Range("B5")
   
    If InputControls.Range("B5") > Date Then
        Result = MsgBox("EndDate seems greater than today's date. Okay to you?", vbYesNo, "Validate End Date")
        If Result = vbNo Then
            Exit Sub
        End If
   
    End If
   
    If Dateminus < 1 Then
        MsgBox ("Date difference must be atleast one. Since EndDate is not inclusive of the date, you can have one day difference between start and end Date to fetch latest price")
    Exit Sub
    End If
   
   
    ' Period
    If InputControls.Range("B3") = "Daily" Then
        period = "1d"
    ElseIf InputControls.Range("B3") = "Weekly" Then
        period = "1wk"
    ElseIf InputControls.Range("B3") = "Monthly" Then
        period = "1mo"
    End If
   
    Call getCookieCrumb(crumb, cookie, validCookieCrumb)
    OutputData.Range("A2:H1000000").ClearContents
   
    'Loop over multiple symbols
    For i = 8 To last
    Symbol = InputControls.Range("A" & i).Value
    OffsetCounter = 1
    Call ExtractData(Symbol, startDate, endDate, period, cookie, crumb, OffsetCounter)
    Next i

    OutputData.Columns("A:H").AutoFit
   
    Application.Calculation = xlCalculationAutomatic
    InputControls.Select
    MsgBox ("Task Accomplished. See HistoricalData Tab")
       
End Sub

Sub getCookieCrumb(crumb As String, cookie As String, validCookieCrumb As Boolean)

    Dim i As Integer
    Dim str As String
    Dim crumbStartPos As Long
    Dim crumbEndPos As Long
    Dim objRequest
 
    validCookieCrumb = False
   
    For i = 0 To 5  'ask for a valid crumb 5 times
        Set objRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
        With objRequest
            .Open "GET", "https://finance.yahoo.com/lookup?s=bananas", False
            .send
            .waitForResponse (10)
        End With
       
        If Len(crumb) = 11 Then 'a valid crumb is 11 characters long
            validCookieCrumb = True
            Exit For
        End If:
       
    Next i
   
End Sub


Sub ExtractData(Symbols As String, startDate As String, endDate As String, period As String, cookie As String, crumb As String, OffsetCounter As Double)

    Dim resultFromYahoo As String
    Dim objRequest
    Dim csv_rows() As String
    Dim resultArray As Variant
    Dim nColumns As Integer
    Dim iRows As Integer
    Dim CSV_Fields As Variant
    Dim iCols As Integer
    Dim tickerURL As String


    tickerURL = "https://query1.finance.yahoo.com/v7/finance/download/" & Symbols & _
        "?period1=" & startDate & _
        "&period2=" & endDate & _
        "&interval=" & period & "&events=history" & "&crumb=" & crumb
               
    Set objRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
    With objRequest
        .Open "GET", tickerURL, False
        .send
        .waitForResponse
        resultFromYahoo = .responseText
         MsgBox (" ASK: " & tickerURL)
         MsgBox (" resp: " & .responseText)
    End With
   
    nColumns = 8
    csv_rows() = Split(resultFromYahoo, Chr(10))
    csv_rows = Filter(csv_rows, csv_rows(0), False)
    ReDim resultArray(0 To UBound(csv_rows), 0 To nColumns) As Variant
     
    For iRows = LBound(csv_rows) To UBound(csv_rows)
        CSV_Fields = Split(csv_rows(iRows), ",")
        If UBound(CSV_Fields) > nColumns Then
            nColumns = UBound(CSV_Fields)
            ReDim Preserve resultArray(0 To UBound(csv_rows), 0 To nColumns) As Variant
        End If
   
        For iCols = LBound(CSV_Fields) To UBound(CSV_Fields)
            If IsNumeric(CSV_Fields(iCols)) Then
                resultArray(iRows, iCols) = Val(CSV_Fields(iCols))
            ElseIf IsDate(CSV_Fields(iCols)) Then
                resultArray(iRows, iCols) = CDate(CSV_Fields(iCols))
            Else
                resultArray(iRows, iCols) = CStr(CSV_Fields(iCols))
            End If
        Next
    Next
 
    Sheets("HistoricalData").Select
    Range("A1000000").End(xlUp).Offset(OffsetCounter, 0).Select
    Selection.Resize(UBound(resultArray, 1) + 1, UBound(resultArray, 2) + 1).Value = resultArray
   
    Range("H1000000").End(xlUp).Offset(OffsetCounter, 0).Select
    Selection.Resize(UBound(resultArray, 1) + 1, 1).Value = Symbols
   
   
End Sub

EDIT:   This seems to be a new security feature;

https://stackoverflow.com/questions/56698011/consistently-retrieving-cookie-and-crumb-from-yahoo-finance-for-historical-data
« Last Edit: December 15, 2022, 03:48:20 pm by AL »
Laz 3.1, fpc 3.2.2, Win10
Laz 3.1  fpc 3.2.2, MacOS Monterey running on VMWare/Win 10
Laz 3.1  fpc 3.2.2 Ubuntu 20.04

Thaddy

  • Hero Member
  • *****
  • Posts: 14377
  • Sensorship about opinions does not belong here.
Re: How to download Yahoo Finance Data?
« Reply #5 on: December 15, 2022, 04:11:15 pm »
- The example in Russian shows you how to use the API. Since the responses are all json this is doable with fcl-web and fcl-json and should be quite simple to implement.
- If I remember correctly there is a/are COM type library(s) for excel/vba which should make it it possible to translate that code to Pascal. You can also use OLE automation, but that gives you a little less control.

To my personal taste, I would use the first option, since it is not dependent on anything (maybe openssl or libcrypto, but nothing else). Just plain fpc.
« Last Edit: December 15, 2022, 04:17:22 pm by Thaddy »
Object Pascal programmers should get rid of their "component fetish" especially with the non-visuals.

incendio

  • Sr. Member
  • ****
  • Posts: 269
Re: How to download Yahoo Finance Data?
« Reply #6 on: December 16, 2022, 02:55:52 am »
I once wrote an small application for this: https://sourceforge.net/p/wp-laz/code/HEAD/tree/FinancialChart/. It gets the data from Yahoo, AlphaVantage and Google (maybe not working any more). Alphavantage requires an API-Key which you can get after free registration for the most basic data.

The problem with these free providers is that they tend to disappear sooner or later...

As for Yahoo, you must send the following http request and insert the correct parameters. In return, you get a csv file with date, open, high, low close, adjusted close and volume data.

Code: Pascal  [Select][+][-]
  1.   { URL for the download of the stock prices; uses the following parameters at
  2.     these indexes
  3.       0: Ticker symbol
  4.       1: API key
  5.       2: Start date as string
  6.       3: End date as string
  7.       4: ResolutionType (daily, weeky, monthly)
  8.   }
  9. const
  10.   YAHOO_URL := 'https://query1.finance.yahoo.com/v7/finance/download/'+
  11.     '%0:s?'+                        // ticker
  12.     'period1=%2:s'+                 // start date, unix date/time
  13.     '&period2=%3:s'+                // end date, unix date/time
  14.     '&interval=1%4:s'+              // 'd' (daily), 'w' (weekly), 'm' monthly
  15.     '&events=history'+
  16.     '&includeAdjustedClose=true';  
  17. var
  18.   url: String;
  19.   startDateStr, endDateStr: String;
  20.   apiKey: String;
  21.   stockSymbol: String;
  22.   resolution: String;
  23. ...
  24.   stockSymbol := 'MSFT';   // Microsoft
  25.   apiKey := '';
  26.   startDateStr := IntToStr(DateTimeToUnix(EdStartDate.Date));  // EdStartDate and EdEndDate are TDateEdit components
  27.   endDateStr := IntToStr(DateTimeToUnix(EdEndDate.Date));
  28.   resolution := 'd';  // daily, 'w' = weekly, 'm' = monthly
  29.   url := Format(YAHOO_URL, [stockSymbol, apiKey, startDateStr, endDateStr, resolution]); // do not change order!

You can use fphttpclient to send the request (on Windows, my application uses the built-in windows functions).
When did you wrote this code? After 2017?

After 2017 Yahoo changed their API so all codes to download their stock data before 2017 didn't work again.

Will try this code, thanks.
« Last Edit: December 16, 2022, 03:16:41 am by incendio »

incendio

  • Sr. Member
  • ****
  • Posts: 269
Re: How to download Yahoo Finance Data?
« Reply #7 on: December 16, 2022, 03:09:18 am »
Here is an example I found lately on the web. It is in VBA for Excel
I do not know why this "CookieCrumb" procedure is called, seems to be important.


Sub GetData()
    Dim InputControls As Worksheet
    Dim OutputData As Worksheet
    Dim Symbol As String
    Dim startDate As String
    Dim endDate As String
    Dim period As String
    Dim last As Double
    Dim OffsetCounter As Double
    Dim crumb As String
    Dim cookie As String
    Dim validCookieCrumb As Boolean
    Dim Result As Integer
   
    Application.ScreenUpdating = False
    Application.DisplayAlerts = False
    Application.Calculation = xlCalculationManual

    Set InputControls = Sheets("Sheet1")
    Set OutputData = Sheets("HistoricalData")
   
    With InputControls
        last = .Cells(.Rows.Count, "A").End(xlUp).Row
    End With
       
    ' Arguments
    startDate = (InputControls.Range("B4") - DateValue("1970-01-01")) * 86400
    endDate = (InputControls.Range("B5") - DateValue("1970-01-01")) * 86400
    period = "1d"
   
    Dateminus = -InputControls.Range("B4") + InputControls.Range("B5")
   
    If InputControls.Range("B5") > Date Then
        Result = MsgBox("EndDate seems greater than today's date. Okay to you?", vbYesNo, "Validate End Date")
        If Result = vbNo Then
            Exit Sub
        End If
   
    End If
   
    If Dateminus < 1 Then
        MsgBox ("Date difference must be atleast one. Since EndDate is not inclusive of the date, you can have one day difference between start and end Date to fetch latest price")
    Exit Sub
    End If
   
   
    ' Period
    If InputControls.Range("B3") = "Daily" Then
        period = "1d"
    ElseIf InputControls.Range("B3") = "Weekly" Then
        period = "1wk"
    ElseIf InputControls.Range("B3") = "Monthly" Then
        period = "1mo"
    End If
   
    Call getCookieCrumb(crumb, cookie, validCookieCrumb)
    OutputData.Range("A2:H1000000").ClearContents
   
    'Loop over multiple symbols
    For i = 8 To last
    Symbol = InputControls.Range("A" & i).Value
    OffsetCounter = 1
    Call ExtractData(Symbol, startDate, endDate, period, cookie, crumb, OffsetCounter)
    Next i

    OutputData.Columns("A:H").AutoFit
   
    Application.Calculation = xlCalculationAutomatic
    InputControls.Select
    MsgBox ("Task Accomplished. See HistoricalData Tab")
       
End Sub

Sub getCookieCrumb(crumb As String, cookie As String, validCookieCrumb As Boolean)

    Dim i As Integer
    Dim str As String
    Dim crumbStartPos As Long
    Dim crumbEndPos As Long
    Dim objRequest
 
    validCookieCrumb = False
   
    For i = 0 To 5  'ask for a valid crumb 5 times
        Set objRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
        With objRequest
            .Open "GET", "https://finance.yahoo.com/lookup?s=bananas", False
            .send
            .waitForResponse (10)
        End With
       
        If Len(crumb) = 11 Then 'a valid crumb is 11 characters long
            validCookieCrumb = True
            Exit For
        End If:
       
    Next i
   
End Sub


Sub ExtractData(Symbols As String, startDate As String, endDate As String, period As String, cookie As String, crumb As String, OffsetCounter As Double)

    Dim resultFromYahoo As String
    Dim objRequest
    Dim csv_rows() As String
    Dim resultArray As Variant
    Dim nColumns As Integer
    Dim iRows As Integer
    Dim CSV_Fields As Variant
    Dim iCols As Integer
    Dim tickerURL As String


    tickerURL = "https://query1.finance.yahoo.com/v7/finance/download/" & Symbols & _
        "?period1=" & startDate & _
        "&period2=" & endDate & _
        "&interval=" & period & "&events=history" & "&crumb=" & crumb
               
    Set objRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
    With objRequest
        .Open "GET", tickerURL, False
        .send
        .waitForResponse
        resultFromYahoo = .responseText
         MsgBox (" ASK: " & tickerURL)
         MsgBox (" resp: " & .responseText)
    End With
   
    nColumns = 8
    csv_rows() = Split(resultFromYahoo, Chr(10))
    csv_rows = Filter(csv_rows, csv_rows(0), False)
    ReDim resultArray(0 To UBound(csv_rows), 0 To nColumns) As Variant
     
    For iRows = LBound(csv_rows) To UBound(csv_rows)
        CSV_Fields = Split(csv_rows(iRows), ",")
        If UBound(CSV_Fields) > nColumns Then
            nColumns = UBound(CSV_Fields)
            ReDim Preserve resultArray(0 To UBound(csv_rows), 0 To nColumns) As Variant
        End If
   
        For iCols = LBound(CSV_Fields) To UBound(CSV_Fields)
            If IsNumeric(CSV_Fields(iCols)) Then
                resultArray(iRows, iCols) = Val(CSV_Fields(iCols))
            ElseIf IsDate(CSV_Fields(iCols)) Then
                resultArray(iRows, iCols) = CDate(CSV_Fields(iCols))
            Else
                resultArray(iRows, iCols) = CStr(CSV_Fields(iCols))
            End If
        Next
    Next
 
    Sheets("HistoricalData").Select
    Range("A1000000").End(xlUp).Offset(OffsetCounter, 0).Select
    Selection.Resize(UBound(resultArray, 1) + 1, UBound(resultArray, 2) + 1).Value = resultArray
   
    Range("H1000000").End(xlUp).Offset(OffsetCounter, 0).Select
    Selection.Resize(UBound(resultArray, 1) + 1, 1).Value = Symbols
   
   
End Sub

EDIT:   This seems to be a new security feature;

https://stackoverflow.com/questions/56698011/consistently-retrieving-cookie-and-crumb-from-yahoo-finance-for-historical-data

I saw that there "CookieCrumb", I think you were right, after 2017, Yahoo changed (whatever, don't know) that codes to download data before 2017 won't work again.

This is worth to try, will try to convert from VBA to Lazarus, thanks.

incendio

  • Sr. Member
  • ****
  • Posts: 269
Re: How to download Yahoo Finance Data?
« Reply #8 on: December 16, 2022, 03:27:28 am »
- The example in Russian shows you how to use the API. Since the responses are all json this is doable with fcl-web and fcl-json and should be quite simple to implement.
- If I remember correctly there is a/are COM type library(s) for excel/vba which should make it it possible to translate that code to Pascal. You can also use OLE automation, but that gives you a little less control.

To my personal taste, I would use the first option, since it is not dependent on anything (maybe openssl or libcrypto, but nothing else). Just plain fpc.
I agreed, first option is better, thanks.

jollytall

  • Sr. Member
  • ****
  • Posts: 319
Re: How to download Yahoo Finance Data?
« Reply #9 on: December 17, 2022, 06:54:30 pm »
I did a lot of data grabbing from Yahoo Finance in the last few years. As far as I know there is no API any more, so the best I could find was to load an html page and then parse the source to get out the data I needed. I mainly did the Statistic page for various shares. The beauty is that the structure (as it is generated by a SW obviously) is always the same, so it works reliably.
The only downside I noticed that if I launched too many requests (>~300 in fast execution one after the other) then the data became totally unreliable. I do not know, if there is a secret mechanism to give false data for robots or it is a bug, but I had to do it in smaller batches.

wp

  • Hero Member
  • *****
  • Posts: 11923
Re: How to download Yahoo Finance Data?
« Reply #10 on: December 17, 2022, 11:41:25 pm »
As far as I know there is no API any more
They are back again, under a different url. See my post in reply #2.

paweld

  • Hero Member
  • *****
  • Posts: 1003
Re: How to download Yahoo Finance Data?
« Reply #11 on: December 18, 2022, 07:16:51 am »
RapidApi provides Yahoo Finance API: https://rapidapi.com/apidojo/api/yahoo-finance1 . You can create a free account and within that account you have 500 requests monthly. 
Sample: 
Code: Pascal  [Select][+][-]
  1. uses  fphttpclient, opensslsockets, httpprotocol, fpjson, jsonparser;
  2.  
  3. procedure TForm1.Button1Click(Sender: TObject);
  4. var
  5.   hc: TFPHttpClient;
  6.   api: String = 'https://apidojo-yahoo-finance-v1.p.rapidapi.com/stock/v2/get-summary';
  7.   host: String = 'apidojo-yahoo-finance-v1.p.rapidapi.com';
  8.   key: String = '**********************************'; //your rapidapi key
  9.   outjson: String = '';
  10.   jd: TJSONData;
  11. begin
  12.   memo1.Lines.Clear;
  13.   hc := TFPHttpClient.Create(nil);
  14.   hc.AllowRedirect := True;
  15.   hc.AddHeader('X-RapidAPI-Key', key);
  16.   hc.AddHeader('X-RapidAPI-Host', host);
  17.   try
  18.     outjson := hc.Get(api + '?symbol=' + HTTPEncode(eSymbol.Text)); //eg. GOOGL
  19.   finally
  20.     hc.Free;
  21.   end;
  22.   jd := GetJSON(outjson);
  23.   if jd.FindPath('summaryProfile.sector') <> nil then
  24.     memo1.Lines.Add(Format('%s: %s', ['Sector', jd.FindPath('summaryProfile.sector').AsString]));
  25.   memo1.Lines.Add('------');
  26.   if jd.FindPath('summaryProfile.longBusinessSummary') <> nil then
  27.     memo1.Lines.Add(Format('%s: %s', ['Description', jd.FindPath('summaryProfile.longBusinessSummary').AsString]));
  28.   memo1.Lines.Add('------');
  29.   if jd.FindPath('price.regularMarketPrice.raw') <> nil then
  30.     memo1.Lines.Add(Format('%s: %f', ['Price', jd.FindPath('price.regularMarketPrice.raw').AsFloat]));
  31.   jd.Free;
  32. end;                                                                    
  33.  
Best regards / Pozdrawiam
paweld

jollytall

  • Sr. Member
  • ****
  • Posts: 319
Re: How to download Yahoo Finance Data?
« Reply #12 on: December 18, 2022, 02:27:29 pm »
They are back again, under a different url. See my post in reply #2.

I am not sure, if there ever was a real API, I only read about it. But the link you gave is not really an API, it is the http Get URL behind the Download button. As I see no similar Download for Statistics or Financial Data, I guess it is limited for the daily (or more rarely) OHLC prices, or do I miss something?


 

TinyPortal © 2005-2018