summaryrefslogtreecommitdiff
path: root/src/content/blog/2018/12/21/ytdl-subs.adoc
diff options
context:
space:
mode:
authorEuAndreh <eu@euandre.org>2025-03-30 17:34:46 -0300
committerEuAndreh <eu@euandre.org>2025-03-30 17:34:46 -0300
commit7979d77fa8aca002282ad1e4c03e1244aa5d1dd1 (patch)
treee2514b61f76f81379a91092bd76440c37aded1e6 /src/content/blog/2018/12/21/ytdl-subs.adoc
parentsrc/content/security.txt: Derive it (diff)
downloadeuandre.org-7979d77fa8aca002282ad1e4c03e1244aa5d1dd1.tar.gz
euandre.org-7979d77fa8aca002282ad1e4c03e1244aa5d1dd1.tar.xz
src/content/blog/: Upgrade 4 files to asciidoc
Diffstat (limited to 'src/content/blog/2018/12/21/ytdl-subs.adoc')
-rw-r--r--src/content/blog/2018/12/21/ytdl-subs.adoc387
1 files changed, 196 insertions, 191 deletions
diff --git a/src/content/blog/2018/12/21/ytdl-subs.adoc b/src/content/blog/2018/12/21/ytdl-subs.adoc
index 183c624..b6540b7 100644
--- a/src/content/blog/2018/12/21/ytdl-subs.adoc
+++ b/src/content/blog/2018/12/21/ytdl-subs.adoc
@@ -1,274 +1,279 @@
----
-title: Using "youtube-dl" to manage YouTube subscriptions
-date: 2018-12-21
-layout: post
-lang: en
-ref: using-youtube-dl-to-manage-youtube-subscriptions
----
-I've recently read the
-[announcement](https://www.reddit.com/r/DataHoarder/comments/9sg8q5/i_built_a_selfhosted_youtube_subscription_manager/)
-of a very nice [self-hosted YouTube subscription
-manager](https://github.com/chibicitiberiu/ytsm). I haven't used
-YouTube's built-in subscriptions for a while now, and haven't missed
-it at all. When I saw the announcement, I considered writing about the
-solution I've built on top of [youtube-dl](https://youtube-dl.org/).
-
-## Background: the problem with YouTube
-
-In many ways, I agree with [André Staltz's view on data ownership and
-privacy](https://staltz.com/what-happens-when-you-block-internet-giants.html):
-
-> I started with the basic premise that "I want to be in control of my
-> data". Sometimes that meant choosing when to interact with an internet
-> giant and how much I feel like revealing to them. Most of times it
-> meant not interacting with them at all. I don't want to let them be in
-> full control of how much they can know about me. I don't want to be in
-> autopilot mode. (...) Which leads us to YouTube. While I was able to
-> find alternatives to Gmail (Fastmail), Calendar (Fastmail), Translate
-> (Yandex Translate), *etc.* YouTube remains as the most indispensable
-> Google-owned web service. It is really really hard to avoid consuming
-> YouTube content. It was probably the smartest startup acquisition
-> ever. My privacy-oriented alternative is to watch YouTube videos
-> through Tor, which is technically feasible but not polite to use the
-> Tor bandwidth for these purposes. I'm still scratching my head with
-> this issue.
-
-Even though I don't use most alternative services he mentions, I do
-watch videos from YouTube. But I also feel uncomfortable logging in to
-YouTube with a Google account, watching videos, creating playlists and
-similar things.
-
-Using the mobile app is worse: you can't even block ads in there.
-You're in less control on what you share with YouTube and Google.
-
-## youtube-dl
-
-youtube-dl is a command-line tool for downloading videos, from YouTube
-and [many other sites](https://rg3.github.io/youtube-dl/supportedsites.html):
-
-```shell
+= Using "youtube-dl" to manage YouTube subscriptions
+
+:ytsm-ann: https://old.reddit.com/r/DataHoarder/comments/9sg8q5/i_built_a_selfhosted_youtube_subscription_manager/
+:ytsm-code: https://github.com/chibicitiberiu/ytsm
+:ytdl: https://youtube-dl.org/
+
+I've recently read the {ytsm-ann}[announcement] of a very nice
+{ytsm-code}[self-hosted YouTube subscription manager]. I haven't used YouTube's
+built-in subscriptions for a while now, and haven't missed it at all. When I
+saw the announcement, I considered writing about the solution I've built on top
+of {ytdl}[youtube-dl].
+
+== Background: the problem with YouTube
+
+:net-giants: https://staltz.com/what-happens-when-you-block-internet-giants.html
+
+In many ways, I agree with {net-giants}[André Staltz's view on data ownership
+and privacy]:
+
+____
+I started with the basic premise that "I want to be in control of my data".
+Sometimes that meant choosing when to interact with an internet giant and how
+much I feel like revealing to them. Most of times it meant not interacting with
+them at all. I don't want to let them be in full control of how much they can
+know about me. I don't want to be in autopilot mode. (...) Which leads us to
+YouTube. While I was able to find alternatives to Gmail (Fastmail), Calendar
+(Fastmail), Translate (Yandex Translate), _etc._ YouTube remains as the most
+indispensable Google-owned web service. It is really really hard to avoid
+consuming YouTube content. It was probably the smartest startup acquisition
+ever. My privacy-oriented alternative is to watch YouTube videos through Tor,
+which is technically feasible but not polite to use the Tor bandwidth for these
+purposes. I'm still scratching my head with this issue.
+____
+
+Even though I don't use most alternative services he mentions, I do watch videos
+from YouTube. But I also feel uncomfortable logging in to YouTube with a Google
+account, watching videos, creating playlists and similar things.
+
+Using the mobile app is worse: you can't even block ads in there. You're in
+less control on what you share with YouTube and Google.
+
+== youtube-dl
+
+:other-sites: https://rg3.github.io/youtube-dl/supportedsites.html
+
+youtube-dl is a command-line tool for downloading videos, from YouTube and
+{other-sites}[many other sites]:
+
+[source,shell]
+----
$ youtube-dl https://www.youtube.com/watch?v=rnMYZnY3uLA
[youtube] rnMYZnY3uLA: Downloading webpage
[youtube] rnMYZnY3uLA: Downloading video info webpage
[download] Destination: A Origem da Vida _ Nerdologia-rnMYZnY3uLA.mp4
[download] 100% of 32.11MiB in 00:12
-```
-
-It can be used to download individual videos as showed above, but it
-also has some interesting flags that we can use:
-
-- `--output`: use a custom template to create the name of the
- downloaded file;
-- `--download-archive`: use a text file for recording and remembering
- which videos were already downloaded;
-- `--prefer-free-formats`: prefer free video formats, like `webm`,
- `ogv` and Matroska `mkv`;
-- `--playlist-end`: how many videos to download from a "playlist" (a
- channel, a user or an actual playlist);
-- `--write-description`: write the video description to a
- `.description` file, useful for accessing links and extra content.
+----
+
+It can be used to download individual videos as showed above, but it also has
+some interesting flags that we can use:
+
+* `--output`: use a custom template to create the name of the downloaded file;
+* `--download-archive`: use a text file for recording and remembering which
+ videos were already downloaded;
+* `--prefer-free-formats`: prefer free video formats, like `webm`, `ogv` and
+ Matroska `mkv`;
+* `--playlist-end`: how many videos to download from a "playlist" (a channel, a
+ user or an actual playlist);
+* `--write-description`: write the video description to a `.description` file,
+ useful for accessing links and extra content.
Putting it all together:
-```shell
+[source,shell]
+----
$ youtube-dl "https://www.youtube.com/channel/UClu474HMt895mVxZdlIHXEA" \
- --download-archive ~/Nextcloud/cache/youtube-dl-seen.conf \
- --prefer-free-formats \
- --playlist-end 20 \
- --write-description \
- --output "~/Downloads/yt-dl/%(uploader)s/%(upload_date)s - %(title)s.%(ext)s"
-```
+ --download-archive ~/Nextcloud/cache/youtube-dl-seen.conf \
+ --prefer-free-formats \
+ --playlist-end 20 \
+ --write-description \
+ --output "~/Downloads/yt-dl/%(uploader)s/%(upload_date)s - %(title)s.%(ext)s"
+----
-This will download the latest 20 videos from the selected channel, and
-write down the video IDs in the `youtube-dl-seen.conf` file. Running it
-immediately after one more time won't have any effect.
+This will download the latest 20 videos from the selected channel, and write
+down the video IDs in the `youtube-dl-seen.conf` file. Running it immediately
+after one more time won't have any effect.
If the channel posts one more video, running the same command again will
-download only the last video, since the other 19 were already
-downloaded.
+download only the last video, since the other 19 were already downloaded.
-With this basic setup you have a minimal subscription system at work,
-and you can create some functions to help you manage that:
+With this basic setup you have a minimal subscription system at work, and you
+can create some functions to help you manage that:
-```shell
+[source,shell]
+----
#!/bin/sh
export DEFAULT_PLAYLIST_END=15
download() {
- youtube-dl "$1" \
- --download-archive ~/Nextcloud/cache/youtube-dl-seen.conf \
- --prefer-free-formats \
- --playlist-end $2 \
- --write-description \
- --output "~/Downloads/yt-dl/%(uploader)s/%(upload_date)s - %(title)s.%(ext)s"
+ youtube-dl "$1" \
+ --download-archive ~/Nextcloud/cache/youtube-dl-seen.conf \
+ --prefer-free-formats \
+ --playlist-end "$2" \
+ --write-description \
+ --output "~/Downloads/yt-dl/%(uploader)s/%(upload_date)s - %(title)s.%(ext)s"
}
export -f download
download_user() {
- download "https://www.youtube.com/user/$1" ${2-$DEFAULT_PLAYLIST_END}
+ download "https://www.youtube.com/user/$1" "${2-$DEFAULT_PLAYLIST_END}"
}
export -f download_user
download_channel() {
- download "https://www.youtube.com/channel/$1" ${2-$DEFAULT_PLAYLIST_END}
+ download "https://www.youtube.com/channel/$1" "${2-$DEFAULT_PLAYLIST_END}"
}
export -f download_channel
download_playlist() {
- download "https://www.youtube.com/playlist?list=$1" ${2-$DEFAULT_PLAYLIST_END}
+ download "https://www.youtube.com/playlist?list=$1" "${2-$DEFAULT_PLAYLIST_END}"
}
export -f download_playlist
-```
+----
With these functions, you now can have a subscription fetching script to
download the latest videos from your favorite channels:
-```shell
+[source,shell]
+----
#!/bin/sh
download_user ClojureTV 15
-download_channel "UCmEClzCBDx-vrt0GuSKBd9g" 100
-download_playlist "PLqG7fA3EaMRPzL5jzd83tWcjCUH9ZUsbX" 15
-```
-
-Now, whenever you want to watch the latest videos, just run the above
-script and you'll get all of them in your local machine.
-
-## Tradeoffs
-
-### I've made it for myself, with my use case in mind
+download_channel 'UCmEClzCBDx-vrt0GuSKBd9g' 100
+download_playlist 'PLqG7fA3EaMRPzL5jzd83tWcjCUH9ZUsbX' 15
+----
+
+Now, whenever you want to watch the latest videos, just run the above script
+and you'll get all of them in your local machine.
-1. Offline
+== Tradeoffs
- My internet speed it somewhat reasonable[^internet-speed], but it is really
- unstable. Either at work or at home, it's not uncommon to loose internet
- access for 2 minutes 3~5 times every day, and stay completely offline for a
- couple of hours once every week.
+=== I've made it for myself, with my use case in mind
- Working through the hassle of keeping a playlist on disk has payed
- off many, many times. Sometimes I even not notice when the
- connection drops for some minutes, because I'm watching a video and
- working on some document, all on my local computer.
- There's also no quality adjustment for YouTube's web player, I
- always pick the higher quality and it doesn't change during the
- video. For some types of content, like a podcast with some tiny
- visual resources, this doesn't change much. For other types of
- content, like a keynote presentation with text written on the
- slides, watching on 144p isn't really an option.
+[qanda]
+Offline::
+My internet speed it somewhat
+reasonable{empty}footnote:[
+ Considering how expensive it is and the many ways it could be better, but also
+ how much it has improved over the last years, I say it's reasonable.
+], but it is really unstable. Either at work or at home, it's not uncommon to
+loose internet access for 2 minutes 3~5 times every day, and stay completely
+offline for a couple of hours once every week.
++
+Working through the hassle of keeping a playlist on disk has payed off many,
+many times. Sometimes I even not notice when the connection drops for some
+minutes, because I'm watching a video and working on some document, all on my
+local computer.
++
+There's also no quality adjustment for YouTube's web player, I always pick the
+higher quality and it doesn't change during the video. For some types of
+content, like a podcast with some tiny visual resources, this doesn't change
+much. For other types of content, like a keynote presentation with text written
+on the slides, watching on 144p isn't really an option.
++
+If the internet connection drops during the video download, youtube-dl will
+resume from where it stopped.
++
+This is an offline first benefit that I really like, and works well for me.
- If the internet connection drops during the video download,
- youtube-dl will resume from where it stopped.
- This is an offline first benefit that I really like, and works well
- for me.
+Sync the "seen" file::
+I already have a running instance of Nextcloud, so just dumping the
+`youtube-dl-seen.conf` file inside Nextcloud was a no-brainer.
++
+You could try putting it in a dedicated git repository, and wrap the script with
+an autocommit after every run. If you ever had a merge conflict, you'd simply
+accept all changes and then run the following to tidy up the file:
++
+[source,shell]
+----
+$ uniq youtube-dl-seen.conf > youtube-dl-seen.conf
+----
-2. Sync the "seen" file
- I already have a running instance of Nextcloud, so just dumping the
- `youtube-dl-seen.conf` file inside Nextcloud was a no-brainer.
+Doesn't work on mobile::
+My primary device that I use everyday is my laptop, not my phone. It works well
+for me this way.
++
+Also, it's harder to add ad-blockers to mobile phones, and most mobile software
+still depends on Google's and Apple's blessing.
++
+If you wish, you can sync the videos to the SD card periodically, but that's a
+bit of extra manual work.
- You could try putting it in a dedicated git repository, and wrap the
- script with an autocommit after every run. If you ever had a merge
- conflict, you'd simply accept all changes and then run:
- ```shell
- $ uniq youtube-dl-seen.conf > youtube-dl-seen.conf
- ```
+=== The Good
- to tidy up the file.
-3. Doesn't work on mobile
+[qanda]
+Better privacy::
+We don't even have to configure the ad-blocker to keep ads and trackers away!
++
+YouTube still has your IP address, so using a VPN is always a good idea.
+However, a timing analysis would be able to identify you (considering the
+current implementation).
- My primary device that I use everyday is my laptop, not my phone. It
- works well for me this way.
- Also, it's harder to add ad-blockers to mobile phones, and most
- mobile software still depends on Google's and Apple's blessing.
+No need to self-host::
+There's no host that needs maintenance. Everything runs locally.
++
+As long as you keep youtube-dl itself up to date and sync your "seen" file,
+there's little extra work to do.
- If you wish, you can sync the videos to the SD card periodically,
- but that's a bit of extra manual work.
-### The Good
+Track your subscriptions with git::
+After creating a `subscriptions.sh` executable that downloads all the videos,
+you can add it to git and use it to track metadata about your subscriptions.
-1. Better privacy
- We don't even have to configure the ad-blocker to keep ads and
- trackers away!
+=== The Bad
- YouTube still has your IP address, so using a VPN is always a good
- idea. However, a timing analysis would be able to identify you
- (considering the current implementation).
-2. No need to self-host
+[qanda]
+Maximum playlist size is your disk size::
+This is a good thing for getting a realistic view on your actual "watch later"
+list. However I've run out of disk space many times, and now I need to be more
+aware of how much is left.
- There's no host that needs maintenance. Everything runs locally.
- As long as you keep youtube-dl itself up to date and sync your
- "seen" file, there's little extra work to do.
+=== The Ugly
-3. Track your subscriptions with git
+We can only avoid all the bad parts of YouTube with youtube-dl as long as
+YouTube keeps the videos public and programmatically accessible. If YouTube
+ever blocks that we'd loose the ability to consume content this way, but also
+loose confidence on considering YouTube a healthy repository of videos on the
+internet.
- After creating a `subscriptions.sh` executable that downloads all
- the videos, you can add it to git and use it to track metadata about
- your subscriptions.
-### The Bad
+== Going beyond
-1. Maximum playlist size is your disk size
+Since you're running everything locally, here are some possibilities to be
+explored:
- This is a good thing for getting a realistic view on your actual
- "watch later" list. However I've run out of disk space many
- times, and now I need to be more aware of how much is left.
-### The Ugly
+=== A playlist that is too long for being downloaded all at once
-We can only avoid all the bad parts of YouTube with youtube-dl as long
-as YouTube keeps the videos public and programmatically accessible. If
-YouTube ever blocks that we'd loose the ability to consume content this
-way, but also loose confidence on considering YouTube a healthy
-repository of videos on the internet.
-
-## Going beyond
+You can wrap the `download_playlist` function (let's call the wrapper
+`inc_download`) and instead of passing it a fixed number to the `--playlist-end`
+parameter, you can store the `$n` in a folder (something like
+`$HOME/.yt-db/$PLAYLIST_ID`) and increment it by `$step` every time you run
+`inc_download`.
-Since you're running everything locally, here are some possibilities to
-be explored:
+This way you can incrementally download videos from a huge playlist without
+filling your disk with gigabytes of content all at once.
-### A playlist that is too long for being downloaded all at once
-You can wrap the `download_playlist` function (let's call the wrapper
-`inc_download`) and instead of passing it a fixed number to the
-`--playlist-end` parameter, you can store the `$n` in a folder
-(something like `$HOME/.yt-db/$PLAYLIST_ID`) and increment it by `$step`
-every time you run `inc_download`.
+=== Multiple computer scenario
-This way you can incrementally download videos from a huge playlist
-without filling your disk with gigabytes of content all at once.
+The `download_playlist` function could be aware of the specific machine that it
+is running on and apply specific policies depending on the machine: always
+download everything; only download videos that aren't present anywhere else;
+_etc._
-### Multiple computer scenario
-The `download_playlist` function could be aware of the specific machine
-that it is running on and apply specific policies depending on the
-machine: always download everything; only download videos that aren't
-present anywhere else; *etc.*
+== Conclusion
-## Conclusion
+youtube-dl is a great tool to keep at hand. It covers a really large range of
+video websites and works robustly.
-youtube-dl is a great tool to keep at hand. It covers a really large
-range of video websites and works robustly.
+Feel free to copy and modify this code, and send me suggestions of improvements
+or related content.
-Feel free to copy and modify this code, and
-[send me](mailto:{{ site.author.email }}) suggestions of improvements or related
-content.
-
-## *Edit*
+== _Edit_
2019-05-22: Fix spelling.
-
-[^internet-speed]: Considering how expensive it is and the many ways it could be
- better, but also how much it has improved over the last years, I say it's
- reasonable.