How to download a video from an URL like *.m3u8 ?

[Hartmut]

Many videos in the internet have a mp4-Format, but sometimes the URL is like this:

• https://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_20240317_000539_4703373_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csmil/index-f1-v1-a1.m3u8
  
• https://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_20240320_005905_4753932_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csmil/index-f1-v1-a1.m3u8
  
• https://apasfiis.sf.apa.at/ipad/cms-worldwide_episodes/14220193_0005_Q6A.mp4/playlist.m3u8
  

When I download such an URL via 'TFPHTTPClient' or via synapse, then I only get a small textfile with a content like this:

Example1:

Code: Text  [Select][+][-]

1. #EXTM3U
2. #EXT-X-TARGETDURATION:10
3. #EXT-X-ALLOW-CACHE:YES
4. #EXT-X-PLAYLIST-TYPE:VOD
5. #EXT-X-VERSION:3
6. #EXT-X-MEDIA-SEQUENCE:1
7. #EXTINF:4.000,
8. segment-1-f1-v1-a1.ts
9. #EXTINF:6.000,
10. segment-2-f1-v1-a1.ts
11. #EXTINF:10.000,
12. segment-3-f1-v1-a1.ts
13. ...
14. segment-35-f1-v1-a1.ts
15. #EXTINF:10.000,
16. segment-36-f1-v1-a1.ts
17. #EXTINF:2.200,
18. segment-37-f1-v1-a1.ts
19. #EXT-X-ENDLIST

Example2:

Code: Text  [Select][+][-]

1. #EXTM3U
2. #EXT-X-TARGETDURATION:10
3. #EXT-X-ALLOW-CACHE:YES
4. #EXT-X-PLAYLIST-TYPE:VOD
5. #EXT-X-VERSION:3
6. #EXT-X-MEDIA-SEQUENCE:1
7. #EXTINF:4.000,
8. segment-1-f1-v1-a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a-bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subtitles.eai-general.aws.srf.ch
9. #EXTINF:6.000,
10. segment-2-f1-v1-a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a-bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subtitles.eai-general.aws.srf.ch
11. #EXTINF:10.000,
12. ...
13. #EXTINF:10.000,
14. segment-830-f1-v1-a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a-bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subtitles.eai-general.aws.srf.ch
15. #EXTINF:8.960,
16. segment-831-f1-v1-a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a-bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subtitles.eai-general.aws.srf.ch
17. #EXT-X-ENDLIST

Example3:

Code: Text  [Select][+][-]

1. #EXTM3U
2. #EXT-X-VERSION:3
3. #EXT-X-STREAM-INF:BANDWIDTH=2254982,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=960x540
4. chunklist.m3u8

My question is: how can my program download such a video? Must I reinvent the wheel or does FPC have some "library" for that?
Is this somehow possible via 'TFPHTTPClient' or via synapse? Or something other which needs no package installation? Then I would prefer that.

I'm on Linux Ubuntu 22.04 with FPC 3.2.2. Thanks in advance.

[lainz]

You need to parse the text file and append at the URL the .ts file to download it.

https://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_20240317_000539_4703373_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csmil/segment-1-f1-v1-a1.ts

[Hartmut]

Thanks lainz. Yes, but then all possible formats have to be considered and all the single segments have to be concatenated together. I made a quick test and saw, that the sum of all downloaded segment files (1747 MB) is clearly bigger then if I download the same video by ffmpeg or by VLC-player (only 1706 MB). As said, I would reinvent the wheel...

Does FPC not have some unit or "library" for that?
Do 'TFPHTTPClient' or synapse have this feature? Or something other which needs no package installation? Then I would prefer that.

[TRon]

I made a quick test and saw, that the sum of all downloaded segment files (1747 MB) is clearly bigger then if I download the same video by ffmpeg or by VLC-player (only 1706 MB). As said, I would reinvent the wheel...

You might want to consider reading up on MPEG transport.

Does FPC not have some unit or "library" for that?

Sure. It is called ffmpeg 

Kidding aside, no afaik there is not a native implementation for decoding mpeg streams. There are plenty of 3th party libraries that are able to do that for you (such as ffmpeg) and all solutions that are available for FPC/Lazarus uses such a 3th party library one way or another.

[Hartmut]

I made a quick test and saw, that the sum of all downloaded segment files (1747 MB) is clearly bigger then if I download the same video by ffmpeg or by VLC-player (only 1706 MB). As said, I would reinvent the wheel...

You might want to consider reading up on MPEG transport.

Do you mean, that each single downloaded segment file contains several control data from the MPEG transport stream (which must be deleted)? Looks like pure horror...

Does FPC not have some unit or "library" for that?

Sure. It is called ffmpeg 

I used ffmpeg manually in the past to download *.m3u8 videos. But multiple times these videos had gaps and were bumpy/jerky. Sometimes I saw messages like "HTTP error 404 Not Found" or "Failed to open segment of playlist" in the huge output of ffmpeg. It looks like ffmpeg does not retry on problems and simple continues... So I hesitate to trust on ffmpeg.

[TRon]

Do you mean, that each single downloaded segment file contains several control data from the MPEG transport stream (which must be deleted)? Looks like pure horror...

Yes and besides that synchronization frames.

It /is/ horror which is probably why (again afaik) there isn't a native solution. Let others worry about the nitty gritty details and let us (simply) make use of it 

I used ffmpeg manually in the past to download *.m3u8 videos. But multiple times these videos had gaps and were bumpy/jerky. Sometimes I saw messages like "HTTP error 404 Not Found" or "Failed to open segment of playlist" in the huge output of ffmpeg. It looks like ffmpeg does not retry on problems and simple continues... So I hesitate to trust on ffmpeg.

I am not sure how individual solutions solve it but it can be either the commandline or the library (the latter which provides more fine grained control). Note, that I do mention ffpmpeg but there are other libraries out there as well (it is just my limitation of not knowing others from mind).

[Hartmut]

I used ffmpeg manually in the past to download *.m3u8 videos. But multiple times these videos had gaps and were bumpy/jerky. Sometimes I saw messages like "HTTP error 404 Not Found" or "Failed to open segment of playlist" in the huge output of ffmpeg. It looks like ffmpeg does not retry on problems and simple continues... So I hesitate to trust on ffmpeg.

I am not sure how individual solutions solve it but it can be either the commandline or the library (the latter which provides more fine grained control).

Thank you TRon. How can I access ffmpeg through a library? Can you give me some details please? I'm a bloody beginner to that stuff.

[TRon]

Thank you TRon. How can I access ffmpeg through a library? Can you give me some details please? I'm a bloody beginner to that stuff.

First of all there is a wiki page on multimedia/video here.

ffmpeg libraries can be "accessed" by Free Pascal by using/installing the ffmpeg header files, which is mentioned on that wiki-page and can be found here.

Other than that the installation of the ffmpeg libraries is required. How to do that (properly) depends on the platform.

If you are looking for example code then I would have to come up with something (new) because last time I did something like that I was still on windows (Linux now) so that might take quite some time due to time constraints at my side. There are probably plenty of c examples out there (and which usually can be translated into Pascal).

[cdbc]

Hi
Search for user @Metis & ffmpeg here on the forum, he does a lot with ffmpeg
edit: See here:https://forum.lazarus.freepascal.org/index.php/topic,26666.msg464784.html#msg464784 Reply #140
Regards Benny

[Hartmut]

Thank you TRon for those links. I checked them but got stuck at https://github.com/DJMaster/ffmpeg-fpc. Don't know what "headers binding" is (has it to do with C?). There are plenty of source files, I had a look in some of them, but did not see something which helped me.

Thanks cdbc for your post. My search got 26 matches, but I found nothing which helped me to access ffmpeg as a library.

I think ffmpeg as a library is far beyond my horizon. And as I said in reply #4, I hesitate to trust on ffmpeg.

So I'm looking for another solution. There exists a FPC unit for curl called 'libcurl'. Does somebody know, if curl can download a video from an URL like *.m3u8? If the answer is definetively 'yes' then I would dive into it.

[cdbc]

Hi
Vlc is another possibility ...and fpc has got bindings for that and I think a component too...
Regards Benny

[TRon]

I think ffmpeg as a library is far beyond my horizon. And as I said in reply #4, I hesitate to trust on ffmpeg.

Why ? and I really meant why. There is nothing wrong with ffmpeg and is a trustworthy tool (that is when compiling yourself or downloaded from a trustworthy source).

Let's do a rewind first...

When copying the streams with ffmpeg (e.g. no conversion at all)
URL1: 93.953.449 bytes
URL2: 2.214.927.005 bytes
URL3: 7.527.485 bytes

The first 2 URL's do not have any issue copying the stream but the 3th last one does.

Code: [Select]

[https @ 0x7f32a4062d00] HTTP error 403 Forbidden
[hls @ 0x55816ff06200] Failed to open segment 14 of playlist 0
[hls @ 0x55816ff06200] Segment 14 of playlist 0 failed too many times, skipping

If attempting that URL a second time:

Code: [Select]

[https @ 0x563518add340] HTTP error 403 Forbidden
[hls @ 0x563518a57200] parse_playlist error Server returned 403 Forbidden (access denied) [https://vod-ww.mdn.ors.at/cms-worldwide_episodes_nas/_definst_/nas/cms-worldwide_episodes/online/14220193_0005_Q6A.mp4/chunklist.m3u8]
[AVIOContext @ 0x563518adce40] Statistics: 126 bytes read, 0 seeks
https://apasfiis.sf.apa.at/ipad/cms-worldwide_episodes/14220193_0005_Q6A.mp4/playlist.m3u8: Server returned 403 Forbidden (access denied)

That tells me that you are not suppose to download that stream without proper authentication (that is, if you are ever suppose/allowed to download it at all).

There are solutions to that problem but people have been going to jail for that or their projects taken down by DMCA. You do have to understand what you are playing with as there are risks involved. Personally I do not judge one way or another.

So, I am going to mention a tool that can be used (you can invoke it from within Pascal code using TProcess for example) but you would have to decode it with rot47 and is named "JE\5=A" (without the quotes). Keen people already know what tool it is but it is the only tool I know of that is open source and reasonably reliable in downloading such URL's.

[Hartmut]

Hi cdbc, I spent some hours searching for those vlc components and their documentation and investigating them. I found FPC units 'libvlc' and 'vlc' and package 'lazvlc'. The only documentation for them I found was https://www.freepascal.org/~michael/articles/lazvlc/lazvlc.pdf, but this deals only with playing a video by VLC, not with downloading a video into a file. So I see no way/sense to proceed with VLC with a reasonable effort.

Thank you TRon for your detailed post. I will investigate it tomorrow.

What about curl? There exists a FPC unit for curl called 'libcurl'. Does somebody know, if this curl unit can download a video from an URL like *.m3u8?

[TRon]

What about curl? There exists a FPC unit for curl called 'libcurl'. Does somebody know, if this curl unit can download a video from an URL like *.m3u8?

Not with the URL's that you provided, or at least not that I am aware of. More information is required.  You can check that out yourself by using the Curl command-line program as that is just a wrapper around libCurl.

That is why I proposed to use a 3th party tool so that you do not have to worry about nitty gritty details (and there are quite a lot of them associated with proper authentication).

[Hartmut]

The first 2 URL's do not have any issue copying the stream but the 3th last one does.

Code: [Select]

[https @ 0x7f32a4062d00] HTTP error 403 Forbidden
[hls @ 0x55816ff06200] Failed to open segment 14 of playlist 0
[hls @ 0x55816ff06200] Segment 14 of playlist 0 failed too many times, skipping


I tried the 3rd URL with ffmpeg and got also "HTTP error 403 Forbidden". Then I downloaded from the same URL via VLC-Player the whole video without problems... One more reason to avoid ffmpeg. I got the 3 URL's from a public database where you can search, view and download videos from the German public television. So nothing not legal. My intention is not to download illegal videos.

Then I checked the curl command-line program to download those URL's:

Code: Text  [Select][+][-]

1. curl <URL> -o <outfile>

I got always only the segment list, not the video. I found no curl option to change this.

Same results with wget:

Code: Text  [Select][+][-]

1. wget -O <outfile> <URL>

I got always only the segment list, not the video. I found no wget option to change this.

Seams that VLC-Player is the best solution.