[video] Download TS files from video stream

Videos on most sites make use of progressive downloading, which means that the video is downloaded to my computer, and easy to trace. There are lots of extensions out there to do this, and even in the dev-tools this is easily done.

On certain websites videos are streamed. which means that we do no just download 1 file, we download lots of small packages. In the dev-tools these packages can be traced. The website I'm interested in is: http://www.rtlxl.nl/#!/goede-tijden-slechte-tijden-10821/c8e2bff7-5a5c-45cb-be2b-4b3b3e866ffb.

-The packages have a .TS extension.

-Packages can be saved by copying the url of the request

-I can not play these files.

I must have done something wrong, or I'm missing something. I want to know what I am doing wrong. I want to create a chrome extension for personal use which captures the urls of all the packages. when I have all the urls I want to pass them on to a php scripts which downloads them and uses ffmpeg to paste them into a mp4 file.

Please help me get the packages.

This question is related to video stream ffmpeg download package

The answer is


While this shouldn't have ever been asked on SO and got through the vetting processing in the first place, I have no idea... but I'm giving my answer anyway.

After exploring basically all of the options presented here, it turns out the simplest is often the best.

First download ffmpeg from: https://evermeet.cx/ffmpeg/

Next, after you have got your .m3u8 playlist file (most probably from the webpage source or network traffic), run this command:

ffmpeg -i "http://host/folder/file.m3u8" -bsf:a aac_adtstoasc -vcodec copy -c copy -crf 50 file.mp4

I tried running it from a locally saved m4u8 file, and it didn't work, because the ffmpeg download procedure downloads the chunks which are relative to the URL, so make sure you use the website url.


using this post

  • Open Firefox / chrome

  • open page the video

  • Play Video

  • click F12 on keyboard -> network

  • in Filter URLs ts

  • copy link of ts

  • remove index and ts extension from link

    for example:

    http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000_454.ts
    

    will be copied as

     http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000
    

insert in below script under LINK

#!/bin/bash

# insert here urls
LINK=(

'http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000' # replace this with your url 

)

mkdir my-videos
cd mkdir my-videos

CNT=0

for URL in ${LINK[@]}
do
  # create folder for streaming media
  CNT=$((CNT + 1))
  mkdir $CNT
  cd $CNT

  (

   DIR="${URL##*/}"

   # download all videos
   wget $URL'_'{0..1200}.ts

   # link videos
   echo $DIR'_'{0..1200}.ts | tr " " "\n" > tslist
   while read line; do cat $line >> $CNT.mp4; done < tslist

   rm -rf media* tslist
   ) &
   cd ..

done

wait

EDIT

adding script in python - runs on windows and linux

import urllib.request
import os
import shutil

my_lessons = [
   #  http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000_454.ts
    "http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000" # replace this with your url 


]

lesson_dir = "my_vids"
try:
    shutil.rmtree(lesson_dir)
except:
    print "ok"

os.makedirs(lesson_dir)
os.chdir(lesson_dir)

for lesson, dwn_link in enumerate(my_lessons):
    print ("downloading lesson  %d.. " % (lesson), dwn_link)
    file_name = '%04d.mp4' % lesson
    f = open(file_name, 'ab')
    for x in range(0, 1200):
        try:
            rsp = urllib.request.urlopen(dwn_link + "_%04d.ts" % (x) )
        except:
            break
        file_name = '%d.mp4' % lesson
        print "downloading  %d.ts" % (x)
        f.write(rsp.read())
    f.close()



print "done good luck!! ==================  "

if the script fails, or downloads empty file, try removing the try wrap to see what fails


  • Download VLC Player
  • Media
  • Convert/Save
  • Network (Tab)
  • Enter URL of [playlist].m3u8
  • Follow remaining wizard steps to set the stream destination (File)
  • Set appropriate transcoding profile (MP4 at the time of this answer)
  • Watch video

Easy youtube-dl example on macOS (in the command line Terminal; Windows supported too):

# List variants (resolutions/bitrates)
$ youtube-dl -F https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
[generic] f08e80da-bf1d-4e3d-8899-f0f6155f6efa: Requesting header
[generic] f08e80da-bf1d-4e3d-8899-f0f6155f6efa: Downloading m3u8 information
[info] Available formats for f08e80da-bf1d-4e3d-8899-f0f6155f6efa:
format code           extension  resolution note
audio-English_stereo  mp4        audio only [en] 
628                   mp4        320x180     628k , avc1.42c00d, video only
928                   mp4        480x270     928k , avc1.42c00d, video only
1728                  mp4        640x360    1728k , avc1.42c00d, video only
2528                  mp4        960x540    2528k , avc1.42c00d, video only
4928                  mp4        1280x720   4928k , avc1.42c00d, video only
9728                  mp4        1920x1080  9728k , avc1.42c00d, video only (best)

# Choose a variant to download, and use its format code below
$ youtube-dl --format 628 https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
...
frame= 5257 fps=193 q=-1.0 Lsize=    6746kB time=00:03:30.16 bitrate= 263.0kbits/s speed=7.73x    
video:6679kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.998669%
[ffmpeg] Downloaded 6907810 bytes
[download] 100% of 6.59MiB in 00:29

$ open f08e80da-bf1d-4e3d-8899-f0f6155f6efa-f08e80da-bf1d-4e3d-8899-f0f6155f6efa.mp4

Use the browser's Developer Tools > Network to get the m3u8 (HLS manifest) URL when starting a streaming video.


I came up with an efficient parralelized one-line that concatenate a sequence of .ts files into one .mp4 file using GNU Parallel:

 parallel -k curl https://example.com/video/seg-{}-f4-v1-a1.ts ::: {1..279} >> result.mp4

The key is to replace the variant part of your url with {} and to set {a..b} with the lower and upper bound. That's it!


1) Please read instructions by @aalhanane (after "paste URL m3u8" step you have to type name for the file, eg "video" then click on "hand" icon next to "quality" and only after that you should select "one on one" and "download").

2) The stream splits video and audio, so you need to download them separately and then use the same m3u8x to join them https://youtu.be/he-tDNiVl2M (optionally convert to mp4).

3) m3u8x can download video without any issues but in my case it cannot extract audio links. So I simply downloaded the *.m3u8 file and searched for line which contains GROUP-ID="audio-0" and then scroll right and copied the link (!including token!) and paste it straight into "Quality URL" field of m3u8x app. Then "one on one" and download it similar to video stream.

Once I had both video and audio, I joined and success =)

p.s. in case automatic extraction will stop working in the future, you can use the same method to extract video links manually.


You can use Xtreme Download Manager(XDM) software for this. This software can download from any site in this format. Even this software can change the ts file format. You only need to change the format when downloading.

like:https://www.videohelp.com/software/Xtreme-Download-Manager-


Copy and paste one of the .ts video files into a new tab in Chrome. Remove the identifying number of the .ts file (0,1,2,3 etc. or whatever number it is) and change the extension from ".ts" to ".mp4". That should bring up the video file in your browser as usual.


I made some changes to dina's answer to avoid attempting to download/combine 1200 parts if there aren't that many.

I also found it helpful to sort by waterfall in the network tab of chrome. This will sort by the time the files are downloaded, so when you are streaming a video the most recently downloaded parts will be at the top, making it easy to find the .ts links.

#!/bin/bash

# Name of the containing folder
GROUP="My Videos"

# Example link: https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE_0.ts
# Insert below as: https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE

# INSERT LINKS TO VIDEOS HERE
LINK=(
'Title for the video link'
'https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE'
'Title for the next video'
'https://vids.net/EECEADFE/EECEADFE/m3u8/EECEADFE/EECEADFE'
)

# ------------------------------------------------------------------------------
mkdir "$GROUP"
cd "$GROUP"

I=0
while [ $I -lt ${#LINK[@]} ]
do
  # create folder for streaming media
  TITLE=${LINK[$I]}
  mkdir "$TITLE"
  cd "$TITLE"
  mkdir 'parts'
  cd 'parts'

  J=$((I + 1))
  URL=${LINK[$J]}

  I=$((I + 2))

  DIR="${URL##*/}"

  # download all streaming media parts
  VID=-1
  while [ $? -eq 0 ];
  do
    VID=$((VID + 1))
    wget $URL'_'$VID.ts
  done

  # combine parts
  COUNTER=0
  while [  $COUNTER -lt $VID ]; do
    echo $DIR'_'$COUNTER.ts | tr " " "\n" >> tslist
    let COUNTER=COUNTER+1
  done
  while read line; do cat $line >> $TITLE.ts; done < tslist

  rm -rf tslist
  mv "$TITLE.ts" "../$TITLE.ts"

  cd ..
  rm -rf 'parts'
  cd ..

done

---> Open Firefox

---> open page the video

---> Play Video

Click ---> Open menu

Click ---> open web developer tools

Click ---> Developer Toolbar

Click ---> Network

---> Go to Filter URLs ---> Write "M3u8" --> for Find "m3u8"

---> Copy URL ".m3u8"

========================

Now Download software "m3u8x" ----> https://tajaribsoft-en.blogspot.com/2016/06/m3u8x.html#downloadx12

---> open software "m3u8x"

---> paste URL "m3u8"

---> chose option "One...One"

---> Click Download

---> Start Download

========================

image "Open menu" ===>

a busy cat

image "Developer Toolbar" ===>

a busy cat

image "m3u8x" ===>

enter image description here

enter image description here


Addition to @aalhanane and @Micheal Espinola Jr

As m3u8x is only available for windows. Once you have identified the m3u8 url you can also use Jdownloader2 or VLC Media Player to download and concatenate the stream.

Jdownloader2: Just copy the m3u8 url when it the Jdownloader is open. It will recognize the stream in Linkgrabber tab.

VLC 3:

Open Network -> Paste m3u8 url -> Checkmark Streamoutput -> Select Settings. Choose output file, container , video and audio encoding. (e.g output.mp4, container: mpeg4, video: h264, audio: mp4a) Start Stream. It will not play the video, but encode it, showing the encoding progress by moving the video play back progress bar.

WARNING: Previously suggesteed chrome extension Stream Video Downloader contains malware. See reddit post


You would need to download all of the transport stream (.ts) files, and concatenate them into a single mpeg for playback. Transport streams such as this have associated playlist files (.m3u8) that list all of the .ts files that you need to download and concatenate. If available, there may be a secondary .m3u8 playlist that will separately list subtitle steam files (.vtt).


  • Get one Link from Network tab of developer tools
  • Remove index and ts extension from link

With following script you can save movie to Videos folder

Example usage:

download-video.sh https://url.com/video.mp4 video-name

download-video.sh

#!/bin/bash
LINK=$1
NAME=$2

START=0
END=2000

help()
{
    echo "download-video.sh <url> <output-name>"
    echo "<url>: x.mp4 (without .ts)"
    echo "<output-name>: x (without .mp4)"
} 

create_folders()
{
    # create folder for streaming media
    cd ~/Videos
    mkdir download-videos
    cd download-videos
}

print_variables()
{
    echo "Execute Download with following parameters"
    echo "Link $LINK"
    echo "Name $NAME"
}

check_video()
{
    i=$START
    while [[ $i -le $END ]]
    do
        URL=$LINK'-'$i.ts
        STATUS_CODE=$(curl -o /dev/null --silent --head --write-out '%{http_code}\n' $URL)
        if [ "$STATUS_CODE" == "200" ]; then
            break
        fi
        ((i = i + 1))
    done

    if [ "$STATUS_CODE" == "200" ]; then
        START=$i
        echo "START is $START"
    else 
        echo "File not found"
    fi
} 


download_video()
{
    i=$START
    e=$END
    while [[ $i -le $END ]]
    do
        URL=$LINK'-'$i.ts
        STATUS_CODE=$(curl -o /dev/null --silent --head --write-out '%{http_code}\n' $URL)
        if [ "$STATUS_CODE" != "200" ]; then
            break
        fi
        wget $URL
        e=$i
        ((i = i + 1))
    done

    END=$e
}

concat_videos()
{
    DIR="${LINK##*/}"

    i=$START
    echo "i is $i"
    while [[ $i -le $END ]]
    do
        FILE=$DIR'-'$i.ts
        echo $FILE | tr " " "\n" >> tslist
        ((i = i + 1))
    done
    while read line; 
    do 
        echo "gugu"$line
        cat $line >> $NAME.mp4; 
    done < tslist

    rm *.ts tslist
}

if [ "$1" == "" ]; then
    echo "No video url provided"
    help
else
    LINK=$1
    if [ "$2" == "" ]; then
        echo "No video output-name provided"
        help
    else
        NAME=$2
        create_folders
        print_variables
        check_video
        download_video
        concat_videos
    fi
fi

I needed to download HLS video and audio streams from a e-learning portal with session-protected content with application/mp2t MIME content type.

Manually copying all authentication headers into the downloading scripts would be too cumbersome.

But the task got much easier with help of Video DownloadHelper Firefox extension and it's Companion App. It allowed to download both m3u8 playlists with TS chunks lists and actual video and audio streams into mp4 files via a click of button while correctly preserving authentication headers.

The resulting separate video and audio files can be merged with ffmpeg:

ffmpeg -i video.mp4 -i audio.mp4 -acodec copy -vcodec copy video-and-audio.mp4 

or with mp4box:

mp4box -add audio.mp4#audio video.mp4 -out video-and-audio.mp4

Tried Video DownloadHelper Chrome extension too, but it didn't work for me.


Examples related to video

How to handle "Uncaught (in promise) DOMException: play() failed because the user didn't interact with the document first." on Desktop with Chrome 66? How to make a movie out of images in python HTML5 Video autoplay on iPhone How do we download a blob url video Twitter - How to embed native video from someone else's tweet into a New Tweet or a DM How to embed new Youtube's live video permanent URL? How to disable auto-play for local video in iframe Writing an mp4 video using python opencv How to extract 1 screenshot for a video with ffmpeg at a given time? Bootstrap 3 - Responsive mp4-video

Examples related to stream

Why does calling sumr on a stream with 50 tuples not complete How to read/write files in .Net Core? How to get the stream key for twitch.tv Download TS files from video stream How to get streaming url from online streaming radio station Save byte array to file How to get error message when ifstream open fails Download large file in python with requests ASP.Net MVC - Read File from HttpPostedFileBase without save Fastest way to check if a file exist using standard C++/C++11/C?

Examples related to ffmpeg

OCI runtime exec failed: exec failed: (...) executable file not found in $PATH": unknown Using ffmpeg to change framerate FFMPEG mp4 from http live streaming m3u8 file? ffprobe or avprobe not found. Please install one How to extract 1 screenshot for a video with ffmpeg at a given time? What steps are needed to stream RTSP from FFmpeg? How to create a video from images with FFmpeg? Download TS files from video stream Best approach to real time http streaming to HTML5 video client Cutting the videos based on start and end time using ffmpeg

Examples related to download

how to download file in react js How do I download a file with Angular2 or greater Unknown URL content://downloads/my_downloads python save image from url How to download a file using a Java REST service and a data stream How to download file in swift? Where can I download Eclipse Android bundle? How to download image from url android download pdf from url then open it with a pdf reader Flask Download a File

Examples related to package

ModuleNotFoundError: No module named 'sklearn' Python: How to pip install opencv2 with specific version 2.4.9? Relative imports - ModuleNotFoundError: No module named x Is __init__.py not required for packages in Python 3.3+ "pip install unroll": "python setup.py egg_info" failed with error code 1 Unable to Install Any Package in Visual Studio 2015 beyond top level package error in relative import How can I specify the required Node.js version in package.json? "installation of package 'FILE_PATH' had non-zero exit status" in R Error in installation a R package