Videos on most sites make use of progressive downloading, which means that the video is downloaded to my computer, and easy to trace. There are lots of extensions out there to do this, and even in the dev-tools this is easily done.
On certain websites videos are streamed. which means that we do no just download 1 file, we download lots of small packages. In the dev-tools these packages can be traced. The website I'm interested in is: http://www.rtlxl.nl/#!/goede-tijden-slechte-tijden-10821/c8e2bff7-5a5c-45cb-be2b-4b3b3e866ffb.
-The packages have a .TS extension.
-Packages can be saved by copying the url of the request
-I can not play these files.
I must have done something wrong, or I'm missing something. I want to know what I am doing wrong. I want to create a chrome extension for personal use which captures the urls of all the packages. when I have all the urls I want to pass them on to a php scripts which downloads them and uses ffmpeg to paste them into a mp4 file.
Please help me get the packages.
While this shouldn't have ever been asked on SO and got through the vetting processing in the first place, I have no idea... but I'm giving my answer anyway.
After exploring basically all of the options presented here, it turns out the simplest is often the best.
First download ffmpeg from: https://evermeet.cx/ffmpeg/
Next, after you have got your .m3u8 playlist file (most probably from the webpage source or network traffic), run this command:
ffmpeg -i "http://host/folder/file.m3u8" -bsf:a aac_adtstoasc -vcodec copy -c copy -crf 50 file.mp4
I tried running it from a locally saved m4u8 file, and it didn't work, because the ffmpeg download procedure downloads the chunks which are relative to the URL, so make sure you use the website url.
using this post
Open Firefox / chrome
open page the video
Play Video
click F12
on keyboard -> network
in Filter URLs
ts
copy link of ts
remove index and ts extension from link
for example:
http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000_454.ts
will be copied as
http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000
insert in below script under LINK
#!/bin/bash
# insert here urls
LINK=(
'http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000' # replace this with your url
)
mkdir my-videos
cd mkdir my-videos
CNT=0
for URL in ${LINK[@]}
do
# create folder for streaming media
CNT=$((CNT + 1))
mkdir $CNT
cd $CNT
(
DIR="${URL##*/}"
# download all videos
wget $URL'_'{0..1200}.ts
# link videos
echo $DIR'_'{0..1200}.ts | tr " " "\n" > tslist
while read line; do cat $line >> $CNT.mp4; done < tslist
rm -rf media* tslist
) &
cd ..
done
wait
EDIT
adding script in python - runs on windows and linux
import urllib.request
import os
import shutil
my_lessons = [
# http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000_454.ts
"http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000" # replace this with your url
]
lesson_dir = "my_vids"
try:
shutil.rmtree(lesson_dir)
except:
print "ok"
os.makedirs(lesson_dir)
os.chdir(lesson_dir)
for lesson, dwn_link in enumerate(my_lessons):
print ("downloading lesson %d.. " % (lesson), dwn_link)
file_name = '%04d.mp4' % lesson
f = open(file_name, 'ab')
for x in range(0, 1200):
try:
rsp = urllib.request.urlopen(dwn_link + "_%04d.ts" % (x) )
except:
break
file_name = '%d.mp4' % lesson
print "downloading %d.ts" % (x)
f.write(rsp.read())
f.close()
print "done good luck!! ================== "
if the script fails, or downloads empty file, try removing the try wrap to see what fails
Easy youtube-dl example on macOS (in the command line Terminal; Windows supported too):
# List variants (resolutions/bitrates)
$ youtube-dl -F https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
[generic] f08e80da-bf1d-4e3d-8899-f0f6155f6efa: Requesting header
[generic] f08e80da-bf1d-4e3d-8899-f0f6155f6efa: Downloading m3u8 information
[info] Available formats for f08e80da-bf1d-4e3d-8899-f0f6155f6efa:
format code extension resolution note
audio-English_stereo mp4 audio only [en]
628 mp4 320x180 628k , avc1.42c00d, video only
928 mp4 480x270 928k , avc1.42c00d, video only
1728 mp4 640x360 1728k , avc1.42c00d, video only
2528 mp4 960x540 2528k , avc1.42c00d, video only
4928 mp4 1280x720 4928k , avc1.42c00d, video only
9728 mp4 1920x1080 9728k , avc1.42c00d, video only (best)
# Choose a variant to download, and use its format code below
$ youtube-dl --format 628 https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
...
frame= 5257 fps=193 q=-1.0 Lsize= 6746kB time=00:03:30.16 bitrate= 263.0kbits/s speed=7.73x
video:6679kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.998669%
[ffmpeg] Downloaded 6907810 bytes
[download] 100% of 6.59MiB in 00:29
$ open f08e80da-bf1d-4e3d-8899-f0f6155f6efa-f08e80da-bf1d-4e3d-8899-f0f6155f6efa.mp4
Use the browser's Developer Tools > Network to get the m3u8 (HLS manifest) URL when starting a streaming video.
I came up with an efficient parralelized one-line that concatenate a sequence of .ts
files into one .mp4
file using GNU Parallel:
parallel -k curl https://example.com/video/seg-{}-f4-v1-a1.ts ::: {1..279} >> result.mp4
The key is to replace the variant part of your url with {}
and to set {a..b}
with the lower and upper bound. That's it!
1) Please read instructions by @aalhanane (after "paste URL m3u8" step you have to type name for the file, eg "video" then click on "hand" icon next to "quality" and only after that you should select "one on one" and "download").
2) The stream splits video and audio, so you need to download them separately and then use the same m3u8x to join them https://youtu.be/he-tDNiVl2M (optionally convert to mp4).
3) m3u8x can download video without any issues but in my case it cannot extract audio links. So I simply downloaded the *.m3u8 file and searched for line which contains GROUP-ID="audio-0" and then scroll right and copied the link (!including token!) and paste it straight into "Quality URL" field of m3u8x app. Then "one on one" and download it similar to video stream.
Once I had both video and audio, I joined and success =)
p.s. in case automatic extraction will stop working in the future, you can use the same method to extract video links manually.
You can use Xtreme Download Manager(XDM) software for this. This software can download from any site in this format. Even this software can change the ts file format. You only need to change the format when downloading.
like:https://www.videohelp.com/software/Xtreme-Download-Manager-
Copy and paste one of the .ts video files into a new tab in Chrome. Remove the identifying number of the .ts file (0,1,2,3 etc. or whatever number it is) and change the extension from ".ts" to ".mp4". That should bring up the video file in your browser as usual.
I made some changes to dina's answer to avoid attempting to download/combine 1200 parts if there aren't that many.
I also found it helpful to sort by waterfall
in the network tab of chrome. This will sort by the time the files are downloaded, so when you are streaming a video the most recently downloaded parts will be at the top, making it easy to find the .ts
links.
#!/bin/bash
# Name of the containing folder
GROUP="My Videos"
# Example link: https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE_0.ts
# Insert below as: https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE
# INSERT LINKS TO VIDEOS HERE
LINK=(
'Title for the video link'
'https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE'
'Title for the next video'
'https://vids.net/EECEADFE/EECEADFE/m3u8/EECEADFE/EECEADFE'
)
# ------------------------------------------------------------------------------
mkdir "$GROUP"
cd "$GROUP"
I=0
while [ $I -lt ${#LINK[@]} ]
do
# create folder for streaming media
TITLE=${LINK[$I]}
mkdir "$TITLE"
cd "$TITLE"
mkdir 'parts'
cd 'parts'
J=$((I + 1))
URL=${LINK[$J]}
I=$((I + 2))
DIR="${URL##*/}"
# download all streaming media parts
VID=-1
while [ $? -eq 0 ];
do
VID=$((VID + 1))
wget $URL'_'$VID.ts
done
# combine parts
COUNTER=0
while [ $COUNTER -lt $VID ]; do
echo $DIR'_'$COUNTER.ts | tr " " "\n" >> tslist
let COUNTER=COUNTER+1
done
while read line; do cat $line >> $TITLE.ts; done < tslist
rm -rf tslist
mv "$TITLE.ts" "../$TITLE.ts"
cd ..
rm -rf 'parts'
cd ..
done
---> Open Firefox
---> open page the video
---> Play Video
Click ---> Open menu
Click ---> open web developer tools
Click ---> Developer Toolbar
Click ---> Network
---> Go to Filter URLs ---> Write "M3u8" --> for Find "m3u8"
---> Copy URL ".m3u8"
========================
Now Download software "m3u8x" ----> https://tajaribsoft-en.blogspot.com/2016/06/m3u8x.html#downloadx12
---> open software "m3u8x"
---> paste URL "m3u8"
---> chose option "One...One"
---> Click Download
---> Start Download
========================
image "Open menu" ===>
image "Developer Toolbar" ===>
image "m3u8x" ===>
Addition to @aalhanane and @Micheal Espinola Jr
As m3u8x is only available for windows. Once you have identified the m3u8 url you can also use Jdownloader2 or VLC Media Player to download and concatenate the stream.
Jdownloader2: Just copy the m3u8 url when it the Jdownloader is open. It will recognize the stream in Linkgrabber tab.
VLC 3:
Open Network -> Paste m3u8 url -> Checkmark Streamoutput -> Select Settings. Choose output file, container , video and audio encoding. (e.g output.mp4, container: mpeg4, video: h264, audio: mp4a) Start Stream. It will not play the video, but encode it, showing the encoding progress by moving the video play back progress bar.
WARNING: Previously suggesteed chrome extension Stream Video Downloader contains malware. See reddit post
You would need to download all of the transport stream (.ts) files, and concatenate them into a single mpeg for playback. Transport streams such as this have associated playlist files (.m3u8) that list all of the .ts files that you need to download and concatenate. If available, there may be a secondary .m3u8 playlist that will separately list subtitle steam files (.vtt).
With following script you can save movie to Videos folder
Example usage:
download-video.sh https://url.com/video.mp4 video-name
download-video.sh
#!/bin/bash
LINK=$1
NAME=$2
START=0
END=2000
help()
{
echo "download-video.sh <url> <output-name>"
echo "<url>: x.mp4 (without .ts)"
echo "<output-name>: x (without .mp4)"
}
create_folders()
{
# create folder for streaming media
cd ~/Videos
mkdir download-videos
cd download-videos
}
print_variables()
{
echo "Execute Download with following parameters"
echo "Link $LINK"
echo "Name $NAME"
}
check_video()
{
i=$START
while [[ $i -le $END ]]
do
URL=$LINK'-'$i.ts
STATUS_CODE=$(curl -o /dev/null --silent --head --write-out '%{http_code}\n' $URL)
if [ "$STATUS_CODE" == "200" ]; then
break
fi
((i = i + 1))
done
if [ "$STATUS_CODE" == "200" ]; then
START=$i
echo "START is $START"
else
echo "File not found"
fi
}
download_video()
{
i=$START
e=$END
while [[ $i -le $END ]]
do
URL=$LINK'-'$i.ts
STATUS_CODE=$(curl -o /dev/null --silent --head --write-out '%{http_code}\n' $URL)
if [ "$STATUS_CODE" != "200" ]; then
break
fi
wget $URL
e=$i
((i = i + 1))
done
END=$e
}
concat_videos()
{
DIR="${LINK##*/}"
i=$START
echo "i is $i"
while [[ $i -le $END ]]
do
FILE=$DIR'-'$i.ts
echo $FILE | tr " " "\n" >> tslist
((i = i + 1))
done
while read line;
do
echo "gugu"$line
cat $line >> $NAME.mp4;
done < tslist
rm *.ts tslist
}
if [ "$1" == "" ]; then
echo "No video url provided"
help
else
LINK=$1
if [ "$2" == "" ]; then
echo "No video output-name provided"
help
else
NAME=$2
create_folders
print_variables
check_video
download_video
concat_videos
fi
fi
I needed to download HLS video and audio streams from a e-learning portal with session-protected content with application/mp2t
MIME content type.
Manually copying all authentication headers into the downloading scripts would be too cumbersome.
But the task got much easier with help of Video DownloadHelper Firefox extension and it's Companion App. It allowed to download both m3u8 playlists with TS chunks lists and actual video and audio streams into mp4 files via a click of button while correctly preserving authentication headers.
The resulting separate video and audio files can be merged with ffmpeg:
ffmpeg -i video.mp4 -i audio.mp4 -acodec copy -vcodec copy video-and-audio.mp4
or with mp4box:
mp4box -add audio.mp4#audio video.mp4 -out video-and-audio.mp4
Tried Video DownloadHelper Chrome extension too, but it didn't work for me.
Source: Stackoverflow.com