[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / r / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip / qa] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
Settings Mobile Home
/mlp/ - Pony

[Advertise on 4chan]


Thread archived.
You cannot reply anymore.


[Advertise on 4chan]


File: 398131.png (104 KB, 689x547)
104 KB
104 KB PNG
Alright guys it's getting down to the wire here. Here's a quick list of things that need assistance right now:

>submitting videos
>sorting videos
>scraping for links

>submitting videos
A site has been created to easily submit pony links to. It'll accept either individual videos (up to 1000 at a time) or playlists (up to 5 at a time, playlists can add up to longer than 1000) and will automatically filter on unlisted and pre-2017 to get exactly what needs to be saved. Make sure to filter on
Submission link: https://derpy.me/4RApE

Part of what I've been doing is going to youtube, searching "mlp music" or "pony ytpmv", filtering on playlists, and hammering in playlists, especially longer ones.
Also, if you filter on "before:YYYY-MM-DD" and "after:YYYY-MM-DD", you can find older playlists that way. Neat!
Some ideas for search terms are pony SFM, mlp YTPMV, pony/TF2, PMVs, pony tumblr/comic dubs, etc. but searching specific musicians or creators might yield some fruitful results too. Sometimes adding a number like "pony music 7" will get you some person who just saves a shitton of playlists of the stuff, and those are great, we love those people.


>sorting videos
All links submitted in the above fashion, and some help is needed to determine what stuff is "pony" or not (utilizing the "Pony?" column). It's pretty easy to do, just a "Y" if something is pony and an "N" if it isn't. Anything blank in the "Pony?" column hasn't been determined yet.
Err on the side of "Y" for pony, but do consider putting "K" for "Kinda pony" for stuff like "Discord remix for 15 minutes!!!! I luv this song xD" or that weird creepy child clickbait shit like https://www.youtube.com/watch?v=zK9f0t8DaRA
Spreadsheet: https://derpy.me/FkhNl

Also, utilizing https://github.com/Pyxia-Code/unlisted-pony-tools/tree/master/video_browser (from https://github.com/Pyxia-Code/unlisted-pony-tools) and https://github.com/itallreturnstonothing/panicpony/blob/main/make_metadata_json.py (from https://github.com/itallreturnstonothing/panicpony) can help speed up sorting SIGNIFICANTLY.

>scraping for links
It's getting a little late for this, but if someone can think of some pony website, forum, subreddit, server, etc. to scrape too, by all means, be our guest. Open to any and all suggestions of places to scrape too.

Some ideas are pony threads on non-pony forums since they're often covered in videos rather than actual discussion (can start here https://u.smutty.horse/mbtjrjzafwv.txt), places like https://knowyourmeme.com/memes/subcultures/my-little-pony-friendship-is-magic#videos or the Rainbow Dash Network (.warc of a scrape of RBN is available), or going back into the Wayback Machine and searching captures of old pony people's profiles for links.

Thank you to anyone and everyone that can assist. Even one person can help tremendously.

Thread theme: https://www.youtube.com/watch?v=X_DVS_303kQ
>>
>>37277397
*Make sure to filter on "100% pony" or not, this will aid in saving links faster.
>>
the end is neigh
>>
We also need help to actually scrape the sites for links. Please do help if you know how to do this.
>>
I wish I had had the foresight to save and catalogue even MLP-related thing when I was in high school.....
>>
>>37277397
Found this 2 links on my saved list.
https://mega.nz/folder/oMkXgCgB#_LuOZCX5iT0vP9NZkcV4BA
https://mega.nz/folder/Ho9xFaiT#ZeDRgXGhAfPKKZuyScefrA
Its supposed to be a collection of pmvs but the first one is empty, hope it helps.
>>
Does anyone has or knows how to quickly make an AI to detect if an image is pony related or not? That could speed things up and make it possible to get pony videos from unlistedvideos.com which don't have any pony related tags but have ponies in the thumbnail. It wouldn't have to be perfect, just good enough to narrow down the list by removing most non pony videos.
>>
Somebody on the knowyourmeme thread pointed out there's a lot of pony content from 2010 and upwards there. I think that's another good place to look for unlisted videos.
>>
>>37277518
In 4 days? Not feasible.
>>
>new CAPTCHA eats yet another post

>>37277518
I did this a while back:
https://colab.research.google.com/drive/1mPzBzPCM4ULLixlmyQVBiAORVjJDg5Gm?usp=sharing

It is a notebook to train and run a model for generating tags on an image. Might work here. I don't have a prepped model at the moment, but everything should be set up in the notebook to gather and basically do everything for you.
>>
>>37277565
Thanks a lot! I don't have much experience with AI but I'll try to train it and make it work with youtube thumbnails.
>>
>>37277565
Speaking of AI, I think what would be great in the future is some language processing on titles and descriptions. It might pick up on certain phrases common to pony works.
>>
>>37277518
>Does anyone has or knows how to quickly make an AI to detect if an image is pony related or not?

The irony is we all know someone who has that but he fucking hates this franchise.
>>
>>37277703
lel
>>
>>37277518
fuck off glowmod I WILL post ponies on other boards and take my 3-day after a janny manually deletes it for free
>>
>>37277397
Im late to the party but how do i help?
Gather links from unlistedvideos.com and add them to the spreadsheet?
>>
File: 1625547265186.png (105 KB, 506x355)
105 KB
105 KB PNG
you can also use youtube-dl to archive youtube channels, here's the command so you don't have to fiddle with it.

youtube-dl -f bestvideo+bestaudio/best --download-archive archive.txt --merge-output-format mkv -i --all-subs --embed-subs --add-metadata --write-annotations --write-info-json --write-thumbnail --write-description -o "%(uploader)s/%(title)s %(id)s.%(ext)s" --yes-playlist ChannelUrl

also I've been saving pony related youtube stuff for a bit now, but I've been uploading it all to archive.org. Dunno if there is much point, linking it here, but there they are, if you care:

https://archive.org/details/xXHomerSimpsonXx
https://archive.org/details/wooden-toaster
https://archive.org/details/brony-dance-party
https://archive.org/details/lundi-brony
https://archive.org/details/acousti-mando-brony
>>
>>37278034
also, forgot to add that these channels exist, they may help you out finding old content inb4 but I already know about them, baka
https://www.youtube.com/c/DeletedPonySongs/videos
https://www.youtube.com/channel/UCQo0QzNHcl3q3Qt2i4p-oEg
>>
>>37278034
Aaaand last one for tonight, if you can get ffmpeg, do it, it'll auto embed the video subtitles for you.
>>
>>37278019
There was a list of 600k videos extracted from unlistedvideos.com: https://u.smutty.horse/mcedbeccohs.txt I believe it's being processed right now. I think unlistedvideos.com should be fine for now.

Speaking of which, I have another list but it's 100k links: https://u.smutty.horse/mceghpjpquu.tsv I'm planning to add anything with pony words in the title first, but highly doubt that I can go through the rest.

So you could help mark out the ones that are currently in the spreadsheet: https://derpy.me/FkhNl

Or suggest other sites that you know have a lot of old pony youtube videos.

Or find more channels and playlists and submit those to the submission form.
>>
Has anyone contacted Makenshi (the EqD/HMH pony music guy, https://twitter.com/Makenshi179) about this (I don't have Twitter)? With how autistic he is about pony music I think he might have a few playlists with obscure music and PMVs.
>>
>>37277518
You could try using CLIP to perform zero-shot image classification. Wouldn't need any training, so it could be ready right now (though it couldn't hurt to train a linear layer on top). You just feed it a thumbnail and a prompt like "A picture of a pony" and it will tell you if the image is likely to be pony-related or not.
I could try this out. Or anyone else could, if they're interested. It would be helpful to have a collection of pony video thumbnails and titles for finetuning--guess those could be scraped from TPA.
>>
>>37278067
Oh, does that mean that an AI thing is no longer required? Or maybe it could be useful for processing other big lists
>>37278174
He posted his archive of PMVs a while ago, but I assume that was all saved and processed back then. I'd guess his channel has also been scraped, though maybe he has secret playlists or something
>>
>>37277518
Unlistedvideos does not have thumbnails, YouTube does. You could instead pull the data for all the IDs with the YouTube API and filter by keywords in title/channel/description/tags.
>>
>>37278067
Found a few videos on pony.tube.
Also checked some other places but /pone/ is dead, there are no ponies on /ñ/, the last site is down and dont bother with bitchute is full of eqg and reaction videos.
>>
>>37278232
>Unlistedvideos does not have thumbnails, YouTube does.
I know, I'll download the thumbnails using yt-dlp.
>You could instead pull the data for all the IDs with the YouTube API and filter by keywords in title/channel/description/tags.
That's what I'm currently doing, but there are some videos that don't contain anything pony related in the title/description/tags.
>>
>>37278174
Speaking of Makenshi, https://horsemusicherald.com/ would be a great place to scrape for music.
>>
>>37278240
>the last site is down
https://endchan.net/pone/ ?
There's also ponychan if that somehow hasn't been scraped yet.
>>
>>37278067
Can we break those up so we don't duplicate work?
There's no way one person can look through 700K links in time.
>>
>>37278326
No but also went to check the endchan and didnt found anything, the place is slow and nobody seems to be talking about youtube happening.
They barely look alive.
>>
im fucking late but is there someway to see what videos are unlisted on a playlist
>>
>>37278503
not conveniently on the youtube site.
>>
>>37278503
This here https://github.com/itallreturnstonothing/panicpony has "find_unlisted_videos_in_playlist.py" which should help. But if the whole playlist is pony, you can throw it into the submission link and it'll do most of the work for you.
>>
As people are filtering IDs, it would be handy to collect public videos that haven't been archived yet for archival later (assuming this isn't done already). It might also be handy to collect non-pony IDs for automatic filtering. If some non-pony videos show up over and over, such a list would save the effort of removing them every time. Of course, the non-pony videos might always be unique, and it's probably too late to start doing this anyway.
>>
What is happening and do sites like fimfiction know. And can we tell them
>>
>>37279160
If we get sites fimfiction to work with us we can cover a lot more ground
>>
>>37277397
has anyone been putting in the japanese moon runes for unlisted?
>>
>>37279243
Good point, have people looked at other languages at all?
>>
>>37277397
not over till it's over
>>
>>37278465
Lurk a little bit more.

It doesn't cost you much energy to figure out where they discuss this topic...
>>
File: 1067308.png (37 KB, 250x250)
37 KB
37 KB PNG
I'm not sure who or why this video was put in, as its not pony at all, but it was exactly what I needed right now. so thank you
https://www.youtube.com/watch?v=0TmnnD5B7Nk

also, be sure to save this absolute classic gem of a youtube vid before it's gone: https://www.youtube.com/watch?v=gDsj5UZ_1bA
may be good to download comments on this one too because.. oh its just so good
>>
File: snicker.gif (1.02 MB, 425x405)
1.02 MB
1.02 MB GIF
https://www.youtube.com/watch?v=-0UORghn1ug
>>
I'm just waiting until a complete downloadable archive of these videos that have been saved exists after it happens.
>>
File: Rainbow Dash wallpaper 3.jpg (1.24 MB, 1920x1080)
1.24 MB
1.24 MB JPG
>>37277397
Pony isn't going to get lost on my watch. Today, tomorrow and the day after tomorrow I will go chronologically through as many EQD articles as possible, noting down every video and channel that are unarchived—especially the unlisted ones—so they can be downloaded. But it is too much to do it all on my own: I need people to download the videos, search channels with unlisted videos if they have any more, and also take further timeslots. I think I can get everything up to 2013 done but that means I need people to cover the other four years. Who's with me?

January 2011:
mlpringlet: https://www.youtube.com/watch?v=ZHcsnPO_AYA
MrBBShockWave: https://www.youtube.com/watch?v=g0aVCuFHt88
UNLISTED ekkosangen: https://www.youtube.com/watch?v=Pi6Xxw-hFOs https://www.youtube.com/watch?v=mFCyQQXFicc https://www.youtube.com/watch?v=pM0q0UQSvdI (more unlisted?)

February 2011:
Yari the Impaler: https://www.youtube.com/channel/UCQsylKUld2FjPBDgQgmvTvw
weabro: https://www.youtube.com/channel/UCQsylKUld2FjPBDgQgmvTvw
UNLISTED Leo Toss: https://www.youtube.com/watch?v=cJhWrqI6Qvk https://www.youtube.com/watch?v=irQ_d78feMA (more unlisted?)
Sprocket man: https://www.youtube.com/watch?v=iVP95IGgyMI
Mortal Anonymous: https://www.youtube.com/watch?v=IiC16-ctND4

March 2011:
RealJollyRoger: https://www.youtube.com/user/RealJollyRoger/videos
drunkill: https://www.youtube.com/watch?v=OZWMtHMRsZk https://www.youtube.com/watch?v=dRn_WtjEmjw https://www.youtube.com/watch?v=LqpyAlvyP3Y
CrazyJezus: https://www.youtube.com/watch?v=j8djg8H0yzM
Alex / GiggersBack: https://www.youtube.com/watch?v=VTlGYS6rAfA https://www.youtube.com/watch?v=_rESjMjmBZE
AQuickAlias: https://www.youtube.com/user/AQuickAlias/videos
DrTYHMG: https://www.youtube.com/user/DrTYHMG/videos

April 2011 (some above are from april too):
TheSaltyJustice: https://www.youtube.com/user/TheSaltyjustice/videos
ddude56: https://www.youtube.com/watch?v=ZF441bbf9ro
JazyKuun: https://www.youtube.com/user/JazyKuun/videos
UNLISTED Thebrotherblume: https://www.youtube.com/watch?v=XK0NAG-k4nQ https://www.youtube.com/watch?v=svtIAOuZK_U https://www.youtube.com/watch?v=svtIAOuZK_U
Sam Hutchinson: https://www.youtube.com/watch?v=zFeeJ7Vx3HM
UNLISTED: TheHolyRayne: https://www.youtube.com/watch?v=WRsVGQv_13k
Spikero2: https://www.youtube.com/user/Spikero2/videos
PirateKing2000: https://www.youtube.com/user/PirateKing2000/videos
l1li!L / Modberp: https://www.youtube.com/user/ModBerp/videos
UNLISTED wrensux: https://www.youtube.com/watch?v=BE5NBwlY_zY https://www.youtube.com/watch?v=kjWRjHXaGRw + other videos
Spotdart3: https://www.youtube.com/watch?v=2qS5jmm2HPU
brian577: https://www.youtube.com/user/brian577/videos
>>
>>37279796
May 2011:
BowenKainZ / Spiritual Beast Tamer, Bowind: https://www.youtube.com/user/BowenKainZ/videos
Gofast Carly: https://www.youtube.com/watch?v=l7OzBBN7LtA
Urimas Ebonheart: https://www.youtube.com/user/UrimasEbonheart/videos
LunaSymphony: https://www.youtube.com/watch?v=Y5nQnEzlXRw
CJTheMusicDude: https://www.youtube.com/watch?v=_9g2sabUQYE
RDreamseer / The Brothers Newton: https://www.youtube.com/user/RDreamseer/videos
Kil6969: https://www.youtube.com/c/kil6969/videos
4TensCakes: https://www.youtube.com/user/4TensCakes/videos
Zeldendorf: https://www.youtube.com/user/Zeldendorf/videos
odbo: https://www.youtube.com/channel/UCEK086Jy0JW-ZzXnFLUxv5g
demonman905: https://www.youtube.com/user/demonman905/videos
Flashcode: https://www.youtube.com/watch?v=gxr4tpuXSYw
zonic14game/zonic19game: https://www.youtube.com/user/zonic14game/videos
Myles O'Neill / mylesandzorg: https://www.youtube.com/watch?v=zO6tc_YKjXU
Killer7Samurai: https://www.youtube.com/user/Killer7Samurai/videos
elmerOzo: https://www.youtube.com/watch?v=wW67HE7ML8k
Outroller / TriforceTrio: https://www.youtube.com/user/TriforceTrio/videos
GrapePie / toplak666: https://www.youtube.com/c/toplak666/videos
UNLISTED AliosPony: https://www.youtube.com/watch?v=IAsOw3K9qcA
UNLISTED Sirachi: https://www.youtube.com/watch?v=Pem4lXG4gzw
respectcopyrightmyass: https://www.youtube.com/user/rspectcopyrightmyass/videos
dietsodadude: https://www.youtube.com/user/dietsodadude/videos
UNLISTED Keenarnor / The Space Daddys: https://www.youtube.com/watch?v=yS9Yrc8iBME
outofthechannel: https://www.youtube.com/user/outofthechannel/videos
MasterXYMER64: https://www.youtube.com/user/masterxymer/videos
beatledude64: https://www.youtube.com/user/beatledude64/videos
DJThunderCZ3 / Roiman Cihlar: https://www.youtube.com/user/DJThunderCZ3/videos
Krass McWriter: https://www.youtube.com/watch?v=aIV4Xc2wxG0
RGamer2009: https://www.youtube.com/watch?v=a2ME-oHoKoY https://www.youtube.com/watch?v=FaQX1Xvf0QI
On vimeo for some reason https://www.equestriadaily.com/2011/05/pmv-shes-cute.html
Lexy: https://www.youtube.com/watch?v=oIonHMxL01g
UNLISTED stargater 94 / Stahry CL: https://www.youtube.com/watch?v=UvK1wBXQPAw
UNLISTED TheSoi: https://www.youtube.com/watch?v=e1Lg4tAzEr0 https://www.youtube.com/watch?v=HglXpenyK_k
UNLISTED Fengor: https://www.youtube.com/watch?v=v_FTFh1ajMM
UNLISTED akfg4life: https://www.youtube.com/watch?v=tJGx8M-9SWE https://www.youtube.com/watch?v=cVlCK-avwVE https://www.youtube.com/watch?v=HhdQuK4aSTM + public videos
91.8 The Fan: https://www.youtube.com/watch?v=9rYsyXNY1oM
thattykins - "That" Technique: https://www.youtube.com/user/thattykins/videos
DJOMiY: https://www.youtube.com/watch?v=kfkFsujMmPs
MysteryLarry / Austin: https://www.youtube.com/user/MysteryLarry/videos
>>
>>37279798
July 2011:
I38VWI: https://www.youtube.com/c/I38VWI/videos
UNLISTED beatledude64: https://www.youtube.com/watch?v=0m6v1tTiuCM
stryke1981: https://www.youtube.com/user/stryke1981/videos
MrPhoxSillanpaa: https://www.youtube.com/user/MrPhoxSillanpaa/videos
UNLISTED PokeDanny10: https://www.youtube.com/watch?v=s1msHo2nZZ0
Interrobang Pie: https://www.youtube.com/watch?v=VVyzDVwezf8
Annomaniac: https://www.youtube.com/user/Annomaniac/videos
TheTotalOverflow: https://www.youtube.com/user/TheTotalOverflow/videos
EG Vids: https://www.youtube.com/c/EGVids/videos
cshep99: https://www.youtube.com/watch?v=iOC4EtApdmk
TailsCoinProductions: https://www.youtube.com/user/TailsCoinProductions/videos
20percentcoolerJ: https://www.youtube.com/user/20percentcoolerJ/videos
bobdash101 / Pizza Princess: https://www.youtube.com/watch?v=3vDiTb1nCHQ
MrYaridovich / Yari The Impaler: https://www.youtube.com/c/MrYaridovich/videos
UNLISTED TheLastGherkin: https://www.youtube.com/watch?v=mlqaNZqD_Zg https://www.youtube.com/watch?v=mlqaNZqD_Zg + public videos
EmperorRadiationEmpire: https://www.youtube.com/c/EmperorRadiationEmpire/videos
starrose20: https://www.youtube.com/user/starrose20/videos
Tsuken: https://www.youtube.com/watch?v=-hojHpRDUTQ
Draqua: https://www.youtube.com/user/Draqua/videos
enquaynay: https://www.youtube.com/user/enquaynay/videos
FrozenVegie: https://www.youtube.com/user/FrozenVegie/videos
TickleMeFrosty / FrostheartTheSiamese: https://www.youtube.com/user/FrostheartTheSiamese/videos
Zeldendorf: https://www.youtube.com/user/Zeldendorf/videos
featherchanlah: https://www.youtube.com/user/featherchanlah/videos
executioneer: https://www.youtube.com/user/executioneer/videos
LanClanKilla: https://www.youtube.com/user/LanCIanKilla/videos
JustUploadingThis: https://www.youtube.com/user/JustUploadingThis/videos
ValenHeartX: https://www.youtube.com/user/ValenheartX/videos
sundayrage: https://www.youtube.com/user/sundayrage/videos
Ryoumouinverse: https://www.youtube.com/user/Ryoumouinverse/videos
TheStarryDawn: https://www.youtube.com/watch?v=1wBvN95MW7Y
JohnnyDeppsGirl4lif3 / Saphira Wolf: https://www.youtube.com/user/JohnnyDeppsGirl4lif3/videos
ancara2: https://www.youtube.com/watch?v=MQYbPEbo8BI
AQuickAlias: https://www.youtube.com/user/AQuickAlias/videos
John Laughlin on Vimeo: https://vimeo.com/26731705
DieWernerBrueder: https://www.youtube.com/watch?v=9yW-Hx78N9k
Exscaliber70: https://www.youtube.com/user/Exscaliber70/videos
APonyFarce: https://www.youtube.com/user/APonyFarce/videos
shadowtime23 / Janni: https://www.youtube.com/user/shadowtime23/videos
JakyaStudios: https://www.youtube.com/user/JakyaStudios/videos
>>
>>37279803
HaroldKrell: https://www.youtube.com/watch?v=Gr19PfemRDE
MrPink091231: https://www.youtube.com/watch?v=1LYH4W-WJmE
rulsis: https://www.youtube.com/user/rulsis/videos
Techercizer: https://www.youtube.com/watch?v=aqqVgmbclB0
UNLISTED Kybdi: https://www.youtube.com/watch?v=xc1ZXU0zFKY&list=PLrSxBs7b_fxr3hMWINChiEQZ_J8aaXz1B&index=3 (playlist)
UNLISTED Albert Phan: https://www.youtube.com/watch?v=6j5NW5xlfWA
UNLISTED Michael White / Ikero1990: https://www.youtube.com/watch?v=s-DbT9x5_O8
TenchiFreak5: https://www.youtube.com/user/TenchiFreak5/videos
LvLUpMusic: https://www.youtube.com/watch?v=3y_hWdhCFJ4
Plastiware: https://www.youtube.com/watch?v=_ehnmF27H3A
venjredd: https://www.youtube.com/user/venjredd/videos
Annomaniac: https://www.youtube.com/user/Annomaniac/videos
UNLISTED icefreaki: https://www.youtube.com/watch?v=9b2YzZO5Yhk
nissanbrasil: https://www.youtube.com/watch?v=9b2YzZO5Yhk
WhiteHawkePAUNCH: https://www.youtube.com/user/WhitehawkePAUNCH/search?query=pony
Solomos598: https://www.youtube.com/watch?v=AvtYOLe6Gh8
theperfectone121212 / gemmyrulez : https://www.youtube.com/c/gemmyrulez/videos

August 2011:
UNLISTED DerpyMailNirvash: https://www.youtube.com/watch?v=Lvkorjb4E7U
UNLISTED iRose: https://www.youtube.com/watch?v=BIIiHJMsJ9w
spiritpup: https://www.youtube.com/watch?v=U3qS0odekuU
InsaneChibiisInsane: https://www.youtube.com/user/InsaneChibiisInsane/videos
BaldDumboRat: https://www.youtube.com/user/BaldDumboRat/videos
BronyLettuceTomato: https://www.youtube.com/watch?v=e44BGkQ8eD8
SeanNH94: https://www.youtube.com/user/SeanNH94/videos
FurryFoxfire: https://www.youtube.com/user/FurryFoxfire/videos
KingoftheCanter: https://www.youtube.com/user/KingoftheCanter/videos
UNLISTED dvdp21 / Wolf Nanaki: https://www.youtube.com/watch?v=zdAGVp7K-H4
sparklesandswirls / Sparklepeep Serafina Swirlington: https://www.youtube.com/user/sparklesandswirls/videos
UNLISTED Thund3rhead: https://www.youtube.com/watch?v=BoFoivTmGEI + public videos
MidnightMidna: https://www.youtube.com/user/MidnightMidna/videos
Menelven: https://www.youtube.com/user/Menelven/videos
UNLISTED kind sounds / derpidety: https://www.youtube.com/watch?v=PJVP7s0cT_I https://www.youtube.com/watch?v=zmGlaD5kxQk
firespikez / Toaster RepairUnit: https://www.youtube.com/user/firespikez/videos
RinAldrin: https://www.youtube.com/c/RinAldrin/videos
TomoAnimators / TomoNews Funnies: https://www.youtube.com/watch?v=Gg-oxLOZBhk
penguindf12 / Stephen Dirth: https://www.youtube.com/user/penguindf12/videos
Coltsuke / PonE - Sharp: https://www.youtube.com/user/Coltsuke/videos
>>
>>37279804
Moontheblueneko: https://www.youtube.com/watch?v=cUR38187b_U
Coy / derpy101: https://www.youtube.com/c/Derpy101/videos
KnowYourMemeStudio: https://www.youtube.com/watch?v=olLDrvc1qt4
UNLISTED sgtwhp / Sabrepulse: https://www.youtube.com/watch?v=Fpx7QEdqmDk
ooklah: https://www.youtube.com/watch?v=4UO1Q5EiYzk
RadiumPony: https://www.youtube.com/user/RadiumPony/videos
UNLISTED PinkieCakeMusic: https://www.youtube.com/watch?v=1EF4Ta8JXa4
1001HELL / the ride never ends: https://www.youtube.com/user/1001HELL/videos
4thSymmetriad: https://www.youtube.com/user/4thSymmetriad/videos
TheVelourFog85: https://www.youtube.com/user/TheVelourFog85/videos
ForgeTechOfficial: https://www.youtube.com/watch?v=kwSrOt_xhOM
SethGline: https://www.youtube.com/user/SethGline/videos
UNLISTED powar1 / Powar Productions: https://www.youtube.com/watch?v=tu4ZPv-0ZfM
Dubs Rewatcher / Linkspartner: https://www.youtube.com/watch?v=WhKQU37oGTc
SurvivorMurk: https://www.youtube.com/user/SurvivorMurk/videos (cut out the furry stuff though)
UNLISTED ColaEuphoria: https://www.youtube.com/watch?v=zin5kyEEj8w (Cola is archived but this video is not, if I'm not mistaken)
Ronthecyborg: https://www.youtube.com/watch?v=gbWprXqZ298
Paper Coelacanth / PaperCoelacanthMusic: https://www.youtube.com/c/PaperCoelacanthMusic/videos
patrickhwillems: https://www.youtube.com/watch?v=CTy5HJnHeFs
Batty Gloom: https://www.youtube.com/watch?v=9xHPLNxHuCo
Octopimp: https://www.youtube.com/watch?v=8ONW7LXaPns (more?)
UNLISTED TheWetOnion: https://www.youtube.com/watch?v=JqQJPo0Jcas
DarkDriumPony: https://www.youtube.com/user/DarkdriumPony/videos
Mathematical: https://www.youtube.com/watch?v=fbLMh8qCFI0
SupaSpeedStrut: https://www.youtube.com/user/SupaSpeedStrut/videos
dragoartofficial: https://www.youtube.com/c/dragoartofficial/videos
KurtissThePonY: https://www.youtube.com/watch?v=kQlWJulpIS4
unintelligentgenius / James Hopkins: https://www.youtube.com/user/unintelligentgenius/videos
Bigserann: https://www.youtube.com/user/Bigserann/videos
Quinn Stephens: https://www.youtube.com/watch?v=tReXtRiW7dI
UNLISTED TarbyRocks / Tarby's Basement: https://www.youtube.com/watch?v=dhfe6uoy6NA + https://www.youtube.com/watch?v=R_3RSgyVX_U&list=PLE4X5VdHVBU_F7_8lath-LGV597_O8wwR&index=1
TheLegionDuelists: https://www.youtube.com/watch?v=usD3bBxeCGo
orangecowdotorg: https://www.youtube.com/watch?v=9a2Syj5gyC0
>>
>>37279796
I think EQG was already scraped, right?
>>
>>37279892
If it is they aren't added to the archive yet. Someone please confirm this.
>>
I am currently scraping all links in https://u.smutty.horse/mbtjrjzafwv.txt
Will take a while because there are 22k forum pages and I'm using a generous delay between requests. Tomorrow I will grep for links and post them. Then I'll move on to other sites that haven't been done yet (would be nice if there was a spreadsheet to track this). I'll also try the AI thing for pre-filtering of the 100k list if I have time
>>
>>37279796
Hey friend, EQD was scraped back in 2019 and everything that was embedded and still alive was saved. Got it confirmed back on the 2nd day this was happening.

However, if you want to go through DerpyNews (was a rival to EQD, especially back in the day) and check for links there, that would be fantastic. I don't think anybody's looked through there yet.
>>
>>37278221
I don't think any AI related stuff can be whipped up quickly within the next 3 days, but any extra help is always welcome. I just think that there are other things that have higher priority right now.

>>37278345
I haven't started on them yet and I'm very much open to anyone helping me out with both of those! Could you quickly filter for unlisted videos uploaded before 2017 and send them back here? After that I'll filter again for pony words in the titles and then we can split up the work from there.
>>
>>37279953
>Could you quickly filter for unlisted videos uploaded before 2017 and send them back here?
I can do that easily. Might take a while to process that many ids though.
>>
>>37279946
Ah, thanks, I didn't know and just wanted to chime in. Doesn't matter to me though, it's kind of fun going through old videos. But if they were scraped all this time, why aren't they added to TPA? And with what level of precision was this done? Some channels have 1 unlisted video linked but then they have a whole playlist of unlisted videos still listed. Are those included?

>However, if you want to go through DerpyNews (was a rival to EQD, especially back in the day) and check for links there, that would be fantastic. I don't think anybody's looked through there yet.
Sure, let me find where to start and I'll be on it.
>>
>>37279953
Just to be clear: I'm starting on all ids in both lists. You said the 600k one is being processed. Should I skip that one and just do the 100k one?
>>
>>37278207
If you need a bunch of pony related videos for thumbnails, you can use the top ten pony videos as a small set: https://docs.google.com/spreadsheets/d/1rEofPkliKppvttd8pEX8H6DtSljlfmQLdFR-SlyyX7E

>>37278269
I think everything on Horse Music Herald has been featured on EqD. Makenshi does EqD's music posts and the latest music almost always goes up there first and then later on HMH.

>>37279962
Found out that apparently the 600k list wasn't processed either, so you can do that one too.
>>
>>37279960
They were added in TPA, the videos are just in a couple different folders and it's hard to get all the IDs on the whole place. How are you checking to be sure they're in TPA?

>with what level of precision
I'm not sure, but from what I gathered, it sounded like everything that was embedded was saved, but the channels themselves weren't gone to to have everything saved from there.

>Sure, let me find where to start and I'll be on it.
https://mlp.derpynews.com/
However, even though you're on the "pony" part of the site here, it doesn't filter the tags correctly anymore. They started reaching out to other shows at a certain point, but I believe most of the good stuff in there would be pony.

>>37279971
>I think everything on Horse Music Herald has been featured on EqD.
Oh really? Huh okay, that would make things easy, but I'll go confirm regardless.
>>
>>37279975
>How are you checking to be sure they're in TPA?
I always check the current channel name and the one they registered with (which you can see by going to the "videos" page). There were probably a couple of oversights, but all the names I posted above could be found in the archive.

>However, even though you're on the "pony" part of the site here, it doesn't filter the tags correctly anymore
I'll start with the pony tag.
>>
>>37279988
could NOT be found
>>
>>37279953
If this AI works properly, it should require no training. (A bit of training could improve things, but like you said, there's not enough time.) I won't spend too long on it--just a few hours to see if it can help prioritize the 700k list.
>>37279971
Thanks, this spreadsheet will be useful
>>
>>37279988
/youtube/ isn't the only directory that has youtube videos. There's /youtube/ and then there's also /quickchive/, which is the default download place for single videos and playlists (which was how EQD was done), and /eqd_yt/, which was initally going to be used for all the EQD stuff but the downloading was then moved to /quickchive/ since there were going to be a ton of duplicates if they did it the original way.
So while usually you'd only need to look in /youtube/ or /quickchive/, you also potentially have /eqd_yt/ for this particular case. And also, videos downloaded in /quickchive/ that are from channels that happen to have their own /youtube/ folder aren't automatically moved over.
>>
>>37280055
Well that was stupid of me. It would be great if all archived channel names could be found in a single simple text file, because checking 2 or 3 pages for the same website is cumbersome.
>>
should I submit /mlp/ 4chan cup videos?
found https://www.youtube.com/watch?v=dB0zXeUJX0E in a folder
>>
>>37280070
you have no idea

>>37280074
absolutely
>>
>>37280074
isn't there an official channel with mlp games?
>>
Here's a forum I found on Derpy News: https://www.everypony.com/
>>
>>37232486
>https://u.smutty.horse/mbznynkcqlr.txt
Question, in the list of videos not in TPA have they already been downloaded by anons or is that still in the prep phase?

>>37279988
>>37279991
>>37279796
https://www.equestriadaily.com/2011/12/music-mr-discords-song-winter-start-up.html
https://www.youtube.com/watch?v=APJEw944XyM
https://www.youtube.com/watch?v=3kCTl2vhBuQ
I have been finding a lot of unisted stuff in early EQD myself.


Hope it is archived. If not, check through music posts and the daily/nightly roundups, especially. I'll try to skim through some myself.

>>37279946
>DerpyNews
Also might want to check out
http://eqmegathreads.blogspot.com/
Had a large community and lots of posts for awhile.
Secondarily:
http://www.equestriagaming.net/
Not sure how many but it certainly wouldn't surpirse me to find a unlisted video or two there.
>>
>>37280115
>Question, in the list of videos not in TPA have they already been downloaded by anons or is that still in the prep phase?
I've downloaded them myself but I don't know if they've been downloaded by TPA or not.
>>
>>37280115
>>37280126
Check against this: https://theponyarchive.com/archive/quickchive/archive.txt
Any unlisted videos that have been added recently should end up there.
>>
>>37280130
Roger.

>>37280093
This forum goes back to 2011, definitely should at least be skimmed through.
>>
>>37278326
I still highly recommend ponychan. There's a lot of ponychan threads on wayback. The video urls are sometimes in the old format, so make sure to look out for those
e.g.
http://www.youtube.com/v/CSlnZxvi37s http://web.archive.org/web/20110622181048/http://www.ponychan.net/chan/collab/res/5925.html#8219
>>
File: smol bab.png (1.44 MB, 4000x3915)
1.44 MB
1.44 MB PNG
Reminder that if anyone feels like sorting in the spreadsheet to be liberal with marking things with "Y". Things regarding the fandom are okay, joke videos are good, and quality is not a prerequisite. Seeing a pony word or PMV or something in the title should be good enough, but be sure to be thorough if you have to open a vid, sometimes the pony is the punchline.
>>
>>37280154
TIL videos could even BE in this format.
>>
>>37280166
Yeah I was pretty surprised too. I don't know if any of the other sites needs to be checked again because of this, but if it helps, I've only ever seen this format in abundance on these archived ponychan threads and barely on any other old sites.
>>
>>37280171
Goes without saying that this should be checked from now on of course. I could just be biased since I've been through the archived ponychan threads a bit more than other pony forums.
>>
>>37279953
https://u.smutty.horse/mcenehdccrl.txt
Here's a csv in id,title format.
This is made from both those files filtered for unlisted videos uploaded before 2017. I also removed videos that are already in the spreadsheet as of a few days ago, videos from this list >>37246524 and videos I've personally processed.
I have more metadata for each video that I can add if you want more than just the title.
>>
>>37279953
https://u.smutty.horse/mcenikpfoac.txt
Here's what I filtered out, but only the unlisted, pre-2017 videos. This should be fine to ignore but I'm sharing it for completeness.
>>
Don't know if anyone did this already, but there's more here: https://www.youtube.com/c/TopBronyMusic/playlists
>>
File: 1514737150051.png (31 KB, 551x537)
31 KB
31 KB PNG
>>37277397
Wait, what's happening in 4 days?
>>
>>37280383
Hiroshimoot decides to shut down /mlp/
>>
Going through ancient articles I found this:
https://www.equestriadaily.com/2012/10/brand-licensing-in-europes-twilight.html
First time Twilicorn was seen (although there was a plushie with Twilight wings a month prior that didn't grab the fandom's attention).
>WAT. Is this accurately predicting and spoiling a future episode plot? We all know what happened with Princess Cadence and Shining Armor first appearing in a cardboard standup.
>The day that happens is the day there is a shitstorm that makes Derpygate look like a happy-go-lucky tea party.
Boy were they right.
>>
File: IMG_1583.jpg (131 KB, 825x960)
131 KB
131 KB JPG
>>37280496
so it wasnt the first time then
because I vividly remember the twilicorn plusie in the funrise office
just because people didn't believe it at first doesnt mean it doesnt count
there was no reason for people to believe it because, twilicorn had no foresight or thought behind it
it was just stapled onto the show at the cost of everything else and amounted to nothing even after 6 additionally full length seasons, they didnt even attempt to do much setup for it in the half of season 3 that wasn't allocated to equestria girls
the difference between derpygate and twiilicorn is that derpygate was sjws attacking the show and twilicorn was hasbro themselves attacking the show without anyone asking them to, so yeah it is a bit worse
>>
>>37280630
First time the news focused attention on the matter then, which was really my point with the quote.
>>
>>37280630
Also Derpygate can also be considered Hasbro attacking the show, as the pressured WeLoveFine to take down Derpy T-shirts, edited her episode and censored her name. The real difference is that Twilicorn fundamentally changed the narrative of the show whereas Derpy is just a sore, mostly fandom-sided incidnet.
>>
>>37280655
not wrong, but derpygate was people asking hasbro to attack the show, and twilicorn was doing it themselves without anyone asking
>>
>>37277364
>>37276901
>>37276842
>>37276704
I finished scraping mlpfanart wiki for unlisted videos and also scanned all of the found playlist links and got 176 unlisted videos:
https://u.smutty.horse/mceoeogtisb.txt

>>37280166
>>37280171
recursive_file_scanner.py should now support these links. I'll later rescan the already scanned websites with the newer version.

I'm also scanning every playlist posted here >>37258536 (not only the ones containing pony terms in titles) for unlisted videos and as of now I've found 3794 new unlisted pony videos.
Here's my current list (15520 videos):
https://u.smutty.horse/mceoqsmgftr.txt
>>
>>37280383
Any video that's uploaded before 2017 and currently unlisted will be automatically set to private.
>>
>>37276550
Forgot to post this. These are all the unlisted videos before 2017 from the liked playlists I scraped back in 2019: https://u.smutty.horse/mceoxbkzctk.txt
There's 17k ids in there.
>>
>>37277397
>37 of my vids on the list
LET THEM DIE YOU FUCKING AUTISTS
>>
>>37280739
When you post something on the internet it's there forever Anon
>>
>>37280729
I should add that these weren't filtered against TPA or any other list yet.
>>
>>37280201
>https://u.smutty.horse/mcenehdccrl.txt
Thanks heaps! Yes I would also prefer having view count and channelId

>>37280729
This is also me if you've got time to add the extra columns for these too.
>>
>>37280739
No.
>>
Is it within the realm of possibility that we can create an add-on that can supply the right video when confronted with a "privatized" message? And if so, would it be wise to also note the URL of each video, along with other data?
>>
>>37281145
So there's a userscript that can point to the archived video on Hobune right now >>37271707
But yeah such a userscript would be very helpful for the archive to be much more accessible.
>>
>>37281171
I'm asking because depending on how much time we can afford, we could make a dash and save as much metadata as possible. As far as I know (I haven't been too involved) we only save title and artist name.
>>
Is the userscript for marking videos in the overview mentioned in >>37255234 available?
>>
You people are making history with what you're doing.
>>
I spend last night creating a simple database from the files of >>37278067
The format is title, id, I ignored all other data as to simplify the process of weeding out non-pony content

Procedure:
Cleaning 100K
VIM Regex used
:%s/^IUC[a-zA-Z0-9-_]\+^I[0-9]\+^I[0-9]\+^I[0-9]\+^I[0-9- :]\+^I[0-9]\+^I[0-9]\+^I[0-9]^I/,,/
Where ^I == Tab on keyboard, actually press tab, inserting the caret-I sequence by hand doesn't work

Cleaning 600K
sed -i 's@https://unlistedvideos.com/v/youtube-@@' 600K.txt
sed -i 's@.html$@@' 600K.txt
For some weird reason, some lines started with a quote-comma sequence
sed -i 's@"","",@@' 600K.txt
VIM Regex used
:%s/^ "//
:%s/^\s//
:%s/","/,,/
:%s/"$//

Then I used this disposable python script I wrote, here is where the database was created:
files.catbox.moe/mdpuhv.py

The resulting database has 703,976 rows. It is also configured to never allow duplicates (or so I hope there weren't two identical ids one of which had quotes (600K file) To avoid duplicates the database has UNIQUE(id) ON CONFLICT IGNORE.
From here you should be able to lookup pony videos. Such sqlite3 query would look like:
sqlite3 > SELECT id, title FROM videos WHERE title GLOB '*MLP*' OR title GLOB '*mlp*'
For more info on GLOB and other conditions I recommend you to check out:
tutorialspoint.com/sqlite/sqlite_glob_clause.htm

Database (58.4 MB):
files.catbox.moe/ja5hp6.db
>>
>>37281208
I have someone who has saved a lot of metadata, and anyone can too since fetching it with youtube api is very fast. I'm sure it's been done already, but I will save metadata for everything on the spreadsheet tomorrow.

I can dump the scripts I've been writing for getting all kinds of metadata, but it's pretty similar to whatever's in the githubs posted here I think.
>>
>>37281396
Nice, this could also be merged with
>>37280201, which filtered the 700k IDs to just unlisted pre-2017 videos not processed yet
>>
>>37280383
Twilight's expecting to give birth to our beautiful foal that day, Im so excited!
>>
>>37281316
thanks friend, we're doing our best
>>
>>37280154
>http://www.youtube.com/v/CSlnZxvi37s
It's privated, did it just get privated?!

>>37280201
Based, is this list all videos you currently downloaded?

>>37280496
>>37280630
I remember... I remember it all...

>>37280739
NEVER!
>>
>>37281301
I have a modified version of this, but it doesn't work 100% right now (probably because I barely know how to javascript): https://u.smutty.horse/mceqqtlzath.txt

It's got a hardcoded smutty.horse link in there which is all the videoIds in TPA as of a week or two ago: https://u.smutty.horse/mcbozvnsiic.txt Feel free to replace this with whatever updated list you're using, but it's got to be in the same format. It'll also check against TPA's quickchive (this will include any unlisted videos saved recently) and hobune.stream/tpa-h.

I think the main problem is that youtube doesn't seem to refresh the whole document when clicking through links, so after a while some of the stray tags from previous pages gets carried over and messes up the other new tags. Basically refreshing the page every few clicks should fix it.

Also it gets slow if you load too many videos on the same page (e.g. long playlist). If you do have to check a long playlist, I suggest extracting the videoIds using a script and compare against the smutty.horse text file I hardcoded in there.

If you know how to solve that issue (or if whatever I modified really sucks), please fix and send it back.
>>
>>37281495
it was privated when I looked last night too
>>
File: Capture.png (90 KB, 1053x842)
90 KB
90 KB PNG
>>37281495
>>37281522
Yeah sorry I was using an example that was private for a long time. If anyone was curious, it was the first video demonstration of Fighting is Magic.

>>37281516
smutty.horse seems to have butchered the unicode symbols or whatever that mess is, but they should be the symbols as seen in attached image. Just replace with those characters
>>
>>37281495
I don't want this to spin into an off-topic discussion, but it's bonkers that the premiere of Twilicorn, the confirmation of EQG, and the C&D of Fighting Is Magic all happened within a span of weeks, as well as trouble surrounding the brony documentaryand YouTube takedowns in general. Reading it back was a strange experience.
>>
If there are any videos of it, dont forget plushiegate. Also archive the recordings of concerts from places like ponyville live and everfree network.
>>
Alright, I went through Derpy News. I went from page 730 (the oldest page public, although it seems the website seems quite older than that) to page 230, after which point the news hasn't been any more than con announcements, episode reveals and basically nothing fandom-sided for the last hundred pages. Here's what I found. The ones with "UNLISTED" have the highest priority, and it should still be checked if any more hidden stuff can be unearthed:
RocketJump: https://www.youtube.com/watch?v=lxHhBx1FtLQ
xCriticalStrikex: https://www.youtube.com/c/xCriticalStrikex/videos
Zoey / FUFriendsUnited: https://www.youtube.com/watch?v=vMlHvc0Gyjo
UNLISTED NightSkye: https://www.youtube.com/watch?v=DCQrqexr0BI
UNLISTED sedthh: https://www.youtube.com/watch?v=AeZz23WJJIk
UNLISTED zedmastermind1: https://www.youtube.com/watch?v=2lP9nDz1IVo
blooblahbleeblehk: https://www.youtube.com/watch?v=oOIKX497jIg
kaxisomo1: https://www.youtube.com/user/kaxisomo1/videos
Spazkidin3D: https://www.youtube.com/watch?v=qw3DnhD8Gls
CliffM85: https://www.youtube.com/user/CliffM85/videos
Swifty Magnets / 227Benjy: https://www.youtube.com/user/227Benjy/videos
NMATV: https://www.youtube.com/watch?v=hcjhG4PQpaE
rosetta mcbride: https://www.youtube.com/watch?v=1Me84Wno0uc
OsakaKunio / Miawooo: https://www.youtube.com/user/OsakaKunio/videos
YBBB / YazzoB: https://www.youtube.com/user/YazzoB/videos
SpazzTH: https://www.youtube.com/user/SpazzTH/videos
qmxonline: https://www.youtube.com/watch?v=rIckeYVuMC0 (more?)
wrightri: https://www.youtube.com/watch?v=ndA69KiffJE
John Doe / sentenal01: https://www.youtube.com/watch?v=y-AeVthdw2w
sonikkuganbatte: https://www.youtube.com/user/sonikkuganbatte/videos
Stephen Granade / LiveGranades: https://www.youtube.com/watch?v=z1dDxoJdQbw
LyraHooves / Lyra Hooves: https://www.youtube.com/user/LyraHooves/videos
AustrianBrony: https://www.youtube.com/user/AustrianBrony/videos
UNLISTED Shirotech: https://www.youtube.com/watch?v=3kzlDSjcxZE
NaoTheFox: https://www.youtube.com/watch?v=P_LiXCBd9-k
The Manjaro / killmanjaro: https://www.youtube.com/user/killmajaro1/videos (is archived but not all videos)
NorthCalBronies: https://www.youtube.com/user/NorthCalBronies/videos (archived, but I think two videos are missing)
UNLISTED honda civic: https://www.youtube.com/watch?v=0YlKp6GpPqs
UNLISTED genericname1234: https://www.youtube.com/watch?v=Y0BVVjQm2x0
XaguarMusic: https://www.youtube.com/user/XaguarMusic/videos
UNLISTED psynwav / Psycosis: https://www.youtube.com/watch?v=nJkBes9cuuI&list=PLVj9-MmaAxRtpGZf7vMIcX75lEBvw2p1k https://www.youtube.com/watch?v=DaZQLJJx9vU
timowi / bufbaf: https://www.youtube.com/c/bufbaf/videos
UNLISTED Uncouth: https://www.youtube.com/watch?v=H_w5ebSG9xM (channel has been archived but this unlisted video has not)
>>
>>37281857
TWITCH: https://www.youtube.com/c/AVINOID/videos (not all videos are archived yet)
Chicken Nuggets / Devorer17: https://www.youtube.com/watch?v=-qGH8l7fvg4
TheITInFit: https://www.youtube.com/user/TheITinFIT/videos
ComicsOnline: https://www.youtube.com/watch?v=8Y2hQTgZWk0
Non-YouTube: https://www.blogtalkradio.com/alltasteexplosion/2013/02/21/maryke-hendrikse-interview--dennis-daniel-show (+ other interviews)
BroniesForChaos: https://www.youtube.com/user/BroniesForChaos/videos
Lux umbra / danceluxubmra: https://www.youtube.com/user/danceluxumbra/videos
Interview on Derpy News (03–26–2013): https://derpynews.com/wp-content/uploads/2013/03/Interview-with-IDW-My-Little-Pony-Comic-Team.mp3
http://mlp.derpynews.com/mad-mlp-sketch-4-dash-bernstein/#more-16676
The MBS Show Podcast: https://www.youtube.com/c/TheMBSShow/videos

Also this list contains a couple of non-YouTube videos, but I thought it prudent to take those in too.
>>
>>37281857
The site is indeed older. I think it may be possible to access older posts using wayback. The site used to be http://www.derpyhoovesnews.com/
>>
File: 1622581465625.png (47 KB, 1419x825)
47 KB
47 KB PNG
>>37281557
and unicorn happened shortly after
truly apocalyptic times
>>
>>37277547
Yes someone please do knowyourmeme. It's guaranteed to be all pony-related videos in that video section: https://knowyourmeme.com/memes/subcultures/my-little-pony-friendship-is-magic/videos
There's currently 3,948 videos there. This should be doable.
>>
>>37281909
unicorn*
>>
>>37281909
>>37281917
*unicon
sorry I'm having a seizure
>>
>>37280754
>Yes I would also prefer having view count and channelId
Working on it. Expect it in the next 90 minutes.

>>37281495
>is this list all videos you currently downloaded?
NO
It's yet-to-be-processed videos. Most of them are non-pony.
>>
>>37281892
>The site used to be http://www.derpyhoovesnews.com/
Using derpyhooves.com I can go all the way back to February 19th 2011, but on that version all embedded media are almost completely noninteractive, but maybe someone else can take a second look at it. It's pretty important stuff after all.
>>
If there's time: http://bronies.memebase.com/

A lot of vintage pony memes had the MyLittleBrony.com watermark, and that's where the site is now. It's mostly images but there should be some old videos in there too.

They're also turning ten years old today.
>>
Has everything in https://ponepaste.org/archive/rewatch-backup/ been processed? Here are the missing ones from season 6 https://u.smutty.horse/mcagwmmecin.zip
>>
>>37281927
Sorry, I meat THIS list:
>>37280674
>Here's my current list (15520 videos)

>>37281853
>plutshiegate
Yeah, a lot of stuff relating to old drama and chaos may get destroyed in this couldn't it?

>>37281954
I'll skim through it and see if I can find anything.

>>37282007
A remnant of My Little Brony is still around?!

>>37281857
> Psycosis: https://www.youtube.com/watch?v=nJkBes9cuuI&list=PLVj9-MmaAxRtpGZf7vMIcX75lEBvw2p1k
This is a clever mass up.
>>
>>37281495
>is this list all videos you currently downloaded?
>>37282138
>Sorry, I meat THIS list:
>>>37280674(You)
>>Here's my current list (15520 videos)
I'm still downloading them. As of now I've downloaded 9270 videos from that list (videos, metadata, thumbnails and comments).
>>
>>37281910
I can't get past page 99 and videos further back aren't loading.
https://knowyourmeme.com/memes/subcultures/my-little-pony-friendship-is-magic/videos/page/99

Hitting sort by oldest just gives me a sever error.
https://knowyourmeme.com/memes/my-little-pony-friendship-is-magic/videos/sort/oldest
>>
>>37282138
Not a remnant, that's the actual site! The domain was always set to redirect to the brony section on Memebase but it seems that it wasn't renewed after a while. A lot of the old ratings and comments are gone now, but otherwise all the posts are still there.

And some history for today: https://web.archive.org/web/20110722165725/http://memebase.com/2011/07/20/memes-introducing-my-little-brony/
>>
>>37282138
>This is a clever mass up.
You mean the fact that the video is SU? There's pony in that playlist, and I just wanted to dump the whole thing instead of picking the important ones out. You can disregard SU and the like of course.
>>
>>37282252
I think it was just doing that for a little while. I also got server error trying to access the main mlp page several minutes ago but now it's fine.

I sorted by oldest and it looks like page 99 is correct, which means the oldest video there should be this: https://knowyourmeme.com/videos/2360-my-little-pony-friendship-is-magic
>>
>>37282138
psycosis did a lot of mashups back then
You could probably still find some of the unlisted pony mashups via tumblr: https://psycosis91.tumblr.com/page/21
>>
>>37282292
I'll grab what I can find.
>>
>>37282321
Also that page 21 was just some arbitrary page I was on, so there's some more before that too.
>>
>>37280754
here's a csv with id,channelId,views,title
https://u.smutty.horse/mcesbtsdhnz.txt
some videos didn't return a viewCount and in that case the views column is NULL
7 ids have disappeared:
>-1mbylZj8VY,Bring your music whenever you go, wherever you go
>2us4pbzLIrI,Resolución Sorteo Tratamiento Queratina
>I4QGRlTb-bQ,*****CLOSED***** 5,000 subscriber giveaway! | JJGeneral1 *****CLOSED*****
>RlZhHtbonRQ,Supply Chain Management: JDA Planning On Demand
>WQABCU2bN0w,World of Final Fantasy Walkthough (Part 12)
>XUnRu5JUby0,Betrayed - The Producers
>_VZ8OXQ-wiU,Lapo pls
>>
>>37282282
No, the Rarrity Rickroll.

>>37282292
I may take a look too.

>>37282216
We are really cutting it close. Aren't we?

>>37282263
>Not a remnant, that's the actual site!
Of all the places to be still kicking... Never was a fav place but even as someone who dislikes the cringe of "brony" from that period I still can't help but have nostalgia for it. It still was a part of the ride.
>>
>>37282440
>No, the Rarrity Rickroll.
I noticed it just now, that's a good mashup!
>>
Psycosis is a hell of a ride. He's got a ton of fun non-pony mashups, but as far as we are concerned, these are the YouTube links, all unlisted:
https://www.youtube.com/watch?v=HS3hvIgTepQ
https://www.youtube.com/watch?v=VQFZJrET3j0
https://www.youtube.com/watch?v=tJwX8G14aQs
https://www.youtube.com/watch?v=ATI-C2KEnV0
https://www.youtube.com/watch?v=OQgKnWha-WY
https://www.youtube.com/watch?v=oCEscoggvsk
https://www.youtube.com/watch?v=21KDu6NrOBg
https://www.youtube.com/watch?v=DaZQLJJx9vU
https://www.youtube.com/watch?v=NDfz9gnR5Ao
https://www.youtube.com/watch?v=FRvxFlRc-XU
https://www.youtube.com/watch?v=NveCOa7fETo
https://www.youtube.com/watch?v=QjpB10PP6Jw
https://www.youtube.com/watch?v=2QDDWavFSEg
https://www.youtube.com/watch?v=VUbTu7IKXpU
https://www.youtube.com/watch?v=EuBbNi1Zbgk (based on Saberspark's top 25 of 2012)

There are also a lot of pony songs that can only be downloaded from Mediafire, or, for lack of a download link, must be ripped from the page itself somehow, but they would be worth it. One thing I cannot find is his Remixes Are Magic 2, seeing that he talks about Remixes Are Magic 3 on page 25. I think his page is supposed to go even further than page 26.

I found these unarchived channels:
https://www.youtube.com/user/crash842/videos
https://www.youtube.com/c/mindlessgonzo/playlists (this one is massive)

Can anyone confirm the stuff in >>37281857 and >>37281863 is processed?
>>
>>37282864
Alright, I will get this for sure.
>>
While searching through some old playlists of mine I stumbled across somewhat of a rarity (hue). The entire Apple Family Reunion episode still somehow up on YouTube from the days when that’s how everyone watched the new episodes. It even has comments intact, might be worth archiving them.
https://youtu.be/NpUDbfRrLAs [Open]
This channel also has spike at your service still up.
Reposting this because I just found another channel hidden away in the same playlist which has a few episodes from S1-3 still up. Sadly COPPA has taken a lot of them but some still have their comments intact.
https://youtube.com/user/ThatOneBronycast
>>
>>37282864
This is a time capsule. Good find! I am downloading comments now.
>>
>>37282931
y u lie?!
>>
>>37277518
The idea with an AI seems to somewhat work. I tried the deepdanbooru one from here >>37277565 but got errors about not enough memory, but the CLIP AI >>37278207 seems to somewhat work. I gave it 3 captions:
-A picture of a pony
-A picture of a cartoon
-A photo
and when searching only by videos where the AI thinks the thumbnail is a pony I get around 8 pony videos per 100 and when searching by videos the AI classifies as a photo I get 0 pony videos per 100. I didn't do any additional training and only used the pre-trained model.
>>
>>37282025
Yes, I just finished rechecking that. I'm waiting on Poneb.in to come back up to check a few pastes that SmileIsPasting did, but otherwise, everything that was unlisted is downloaded.

>>37282292
>>37282440
>>37282804
>Psychosis
I'd looked at all the pony stuff he did on his tumblr and should have it all downloaded but I'll re-check. And I have all the other pony stuff he did that was uploaded to tumblr as .mp3's still in my tabs in Firefox.

>>37282885
Been coming across quite a few like this in the spreadsheet, episodes that are still miraculously up, although the vast majority of them are ded. There's a few that are literally stream recordings or were uploaded they day the episode premiered.
>>
File: 131811.gif (204 KB, 400x280)
204 KB
204 KB GIF
I'm not sure why I didn't think of this before, but if someone could go through the entries on for each of the events and scrape the youtube links, we might be able to find some still up and around:
https://mylittleremix.com/submission/list.php

Also, I contacted the guy who runs https://finnrepo.a2hosted.com/YTPMV_Database/ , gave him a shitton of pony regex, and downloaded any at-risk videos that resulted.
Here's the full list he gave me: https://u.smutty.horse/mcetozchmwy.txt
>>
>>37283114
>stream recordings
I’ve always wanted to find more of those
>>
>>37283145
Most were recorded streams of just the episode, but actually, I just found that there was even more than that:
https://www.youtube.com/watch?v=mHiRC_sNX4w
https://www.youtube.com/watch?v=_akRvLay3Us
https://www.youtube.com/watch?v=hdQwjWgYP8U
https://www.youtube.com/watch?v=8MgIFi04jag
>>
>>37283168
by "more than that", I meant there was more than just the one I thought I found
>>
File: problem.png (91 KB, 1105x666)
91 KB
91 KB PNG
>>37283142
>https://mylittleremix.com/submission/list.php
Can't view profiles without being logged in, BUT, you can see the entries and download some
>>
>>37283357
Yeah the point is to scrape the YouTube links on the "Entries" pages of each event.
>>
here's an exmaple of an episode still up
https://www.youtube.com/watch?v=KS9yECq3Hmk
>>
>>37283371
and another
https://www.youtube.com/watch?v=rhFYkUWHFpg
>>
File: instr.png (746 KB, 842x2209)
746 KB
746 KB PNG
>one.json
https://u.smutty.horse/mcetssopycu.json
>two.json
https://u.smutty.horse/mcetssomkkh.json
>three.json
https://u.smutty.horse/mcetssorxdx.json
>four.json
https://u.smutty.horse/mcetssooeig.json


Hey you. Yeah you. Are you lurking? Want to do something to help?

Go here https://github.com/Pyxia-Code/unlisted-pony-tools click the Code button and then Download ZIP.
Extract that zip somewhere.
Download one of the json files above.
Go where you extracted the zip and open the video_browser folder.
Delete the file called metadata.json.
Copy the file you downloaded from smutty into the video_browser folder and rename it metadata.json.
Open index.html with your web browser and start sorting videos.
When you're done copy the pony ids with the Copy to clipboard button, paste them in a text file, and upload that to smutty.
Post your smutty link in this thread and also make sure to say which json file you used.

And with that you will have helped significantly.


If you don't see any videos after opening index.html you might have this condition:
https://support.mozilla.org/en-US/questions/1264280
On firefox you need to open about:config and change that setting. Change it back after you're finished if you're concerned about that vulnerability, but IMO it's kind of paranoid.
I don't know if Chrome has this restriction.
>>
>>37283569
actually it looks like someone has already done a ton of those, so disregard those json files.
>>
>>37283610
The filtered 700k list is taken care of then?
>>
>>37283799
I don't know. Those json files were made by pulling the google docs sheet and filtering for blanks in the pony column. I'm not sure what the status of that list of 700k links is.
>>
>>37283610
What I did right there was take all the links that don't weren't sorted and used a bunch of regex on their titles in another sheet to determine if they were pony or only possibly. Then pasted all the Y's back in.

There's still about 2k in the sheet that need being sorted through though. And there's more I can add, and probably should.
>>
>>37283856
add them
>>
>>37283874
I'm working on it.
>>
>>37283874
>>37283921
Alright, I added a list of ~6000 IDs from a scrape of multiple pony forums, subreddits, websites, etc. over a list of many years. It used to be a list of 12k, but they've already had some regex performed on them and this is the rest.

Don't forget to possibly identify pony musicians/horsefamous people or try and search who someone is if it's a song or something.
>>
>>37277397
Guys I'm super scared right now.
>>
Not requesting anything, just wanted to post this so is archived somewhere, I'm closing its tab, a playlist that had a few pony vids but got murked
youtube.com/playlist?list=PL31136704D38B4AEA
>>
>>37283569
Here's 1170 videos in a metadata.json if someone has a spare hour or two:
https://u.smutty.horse/mcevvdtxpbx.json

>>37283990
Did those IDs go into the spreadsheet? I feel like 1170 unsorted rows is too few.
>>
>>37283103
>but got errors about not enough memory
Errors or warnings? There's a fair chance you'll see warnings about the GPU memory and that it could be faster if it had more vram, but it shouldn't affect training.
>>
>>37284180
I put them in there, but I guess it just takes time for shit to process, which is annoying. I mean like come on, it's been over an hour at this point.
>>
File: file.png (3 KB, 300x33)
3 KB
3 KB PNG
holy fuck where ARE these things
>>
>>37283821
Hey sorry I didn't update. I crashed hard yesterday and feeling not so good. Feel free to do some regex magic on it because I'm not sure if I'll make it back here in time. Sorry for the trouble earlier.
>>
File: 31116.png (297 KB, 669x350)
297 KB
297 KB PNG
Some updates I guess:

Annotations for all the videos in TPA have already been gone through twice-over and saved.

>https://shr.anjanms.com/c9cf9.txt
has been saved.

>https://u.smutty.horse/mbznynkcqlr.txt
has been fully processed.

Everything at risk from the Gdocs submission we originally started with should be downloaded. There are other channels and videos in there not currently at risk and they'll be saved once Friday hits.

mlpfanlabor links have been saved. Gyropedia has been scraped and saved. If the links to the MLP Fandom wiki were submitted to the hobune sheet, then they're probably either being sorted or saved right now.

>https://mylittleremix.com/submission/list.php
was scraped and everything found that was stil alive was saved.

>https://knowyourmeme.com/memes/subcultures/my-little-pony-friendship-is-magic/videos
has been scraped and is currently awaiting being uploaded to the spreadsheet.

Ponychan threads on Wayback are currently being scraped. Might be a good idea to do the same thing for FiMchan, Lunachan, and MLPchan too.

More comments on unlisted videos are being downloaded right now.

The hobune site should be processing IDs much faster now. Those 6k IDs are now reading and waiting to be sorted.

>To do?
I have some personal to-do's I know I want to do, but is there anything that people think is feasible before it all ends? If push comes to shove, IA is doing a thing where we can stack some videos on them that we don't have time to sort through, and they'll be downloaded at like 360p to at least have it saved. I might do that for the fucking long streams that are sitting on K right now.

>>37284147
Multiple people had scraped for all the videos in that grouping of playlists before it died, so it's all good.
>>
>>37284595
Ponychan links are also now submitted to the spreadsheet, and those other chans should be scraped tomorrow.
>>
>>37283168
Wow, these are gold. Any chance you could drop the other ones you found? I know they’ll be going in the archive anyway but I’d like to save local versions of these
>>
>>37284595
>https://u.smutty.horse/mbznynkcqlr.txt
Good work anon.

Secure old drama/shanagins related vids. For example, Dark Skyes, cringe and spaghetti, stuff that maybe important but obscure, like True Capitalist Radio
https://desuarchive.org/mlp/thread/28493307/#q28521373
Stuff can be found in the depths of desu...
>>
>>37284184
I got a python exception and the code stopped running. Here's the log:
https://u.smutty.horse/mcexxuhwyjk.txt
There's not enough time to debug this so I'll stick with CLIP for now but in the future I might try using AI to find videos with ponies in the thumbnail for the pony video list website.
>>
>>37284595
>https://u.smutty.horse/mbznynkcqlr.txt
mfw I still find stuff not in this list.
https://www.youtube.com/watch?v=DUnthV8SuuA
https://www.youtube.com/watch?v=bi8ndl8QNXQ&t=0s
Now, it maybe in TFA, haven't looked there, too tired.

I recomend anons start going though lost and legacy content threads that were in the past:
https://desuarchive.org/mlp/thread/36811780/
https://desuarchive.org/mlp/thread/28493307
https://desuarchive.org/mlp/thread/31352138/
https://desuarchive.org/mlp/thread/36709461/
>>
It seems like my EQD crawl wasn't fruitless after all. There's a considerable number of videos that I cannot find in any folder on the archive. I'll post them when I'm done.
>>
If anyone has a bunch of thumbnails already downloaded, upload them (Drive is best) and I'll run my AI tools over them. They aren't perfect, but they might help with prioritization.
If not, I can download thumbnails myself, but it will be slow. Which IDs would be best to focus on? They should be videos that can't easily be filtered by channel, title, or tags.

>>37283103
>>37284812
For CLIP, I used ViT-B/16 and experimented with a bunch of different prompts. Here are the top 10 in descending order: (periods are included)
>An online still frame of a pony video.
>An online still frame of a MLP video.
>A screenshot of a MLP video.
>An online still frame of a My Little Pony video.
>An online still frame of a My Little Pony Generation 4 video.
>An online screenshot of a MLP video.
>An online still frame of a MLP: FIM video.
>An online still frame of a My Little Pony Generation 4 fan video.
>An online still frame of a My Little Pony: Friendship is Magic video.
>An online still frame of a My Little Pony: FIM video.
They gave the highest logits for the 1200 "Top 10 pony video" thumbnails, at least.

I also finetuned a YOLOv4-tiny to detect pony faces. It detects 500 out of the 1200 "Top 10" thumbnails. Training was fast (30 min), but downloading training images from Derpi/Poner was slow, so I only used 1k. Accuracy was good, though (mAP@0.5 = 90.24%), so even if it misses a lot of pony faces, hopefully the faces it does detect are correct.
I can throw up a notebook, if you want. Though I might not have time until about 17 hours from now.

>>37279938
I have finished with all forums in this list: https://u.smutty.horse/mbtjrjzafwv.txt
Here are the scraped links: https://u.smutty.horse/mcexzhzigcn.txt
>>
>>37284863
Here are the thumbnails I downloaded:
https://drive.google.com/file/d/1nxknrNyrrHHjHyzrNJFxIU9-3Y-zVhg5/view - these are from a list I've already checked for pony videos but there might be some videos I've missed
https://drive.google.com/file/d/1xtpTYI2_u63vLwgEPruyAtuavIong396/view - these are 10k thumbnails from the unlistedvideos website
>>
>>37282885
These are probably the most painful to lose
https://www.youtube.com/watch?v=-ULOzaDoxZs
https://www.youtube.com/watch?v=Y9za5O0j818
https://www.youtube.com/watch?v=OB03ufqDQsE
https://www.youtube.com/watch?v=wAltQq8b1bA

Part 3 is blocked worldwide. Shame or else it would be complete
>>
>>37284858
Celestia bless you
>>
Remember to go through any old pony threads on forums. e.g. https://www.neogaf.com/threads/my-little-pony-friendship-is-magic.422909/
I'm aware that this and a couple of others are done already, but if you find anymore please share those. Some of them have a lot of videos shared in the early days.
>>
>>37285318
Sad to see these ones go, as long as they are backed up they can still be experienced though
>>
Here's the stuff I found. Fortunately the unlisted videos are less than a handful of singular videos, plus two unlisted playlists. For the record, this is up to October 18th 2011, so one more lead is to work further up from there.
https://u.smutty.horse/mcezwliylot.txt

>>37285318
Though it's probably not helpful in this case, I've had some success with contacting channel owners and kindly requesting them republicize old pony videos (or at least save them from privatization. If people can throw me channels that seem active enough (e.g. maintains an active twitter or has uploaded in the last 6 months), I will gladly shoot them a message.
>>
>>37285356
Well that's a lot of channels right off the bat. However it can assumed that anyone that uploads on a weekly basis should've seen the notice by now, so you'd prioritise active channels that haven't uploaded in at least a month.

(Specifically for the videos I linked, MLPRistvak could only be contacted via the old mlptp forum, but there hasn't been any activity from that account on there either, and I heard account approval there is strict.)
>>
One more thing to look out for. I noticed that the sheet has videos with titles like "part 1" or "1/3" in them but the rest of the set isn't there. I've gone through a couple of these channels and have found the rest in playlists. That could be something to do too since some of these probably won't be watchable without the other parts.
>>
>>37285384
The positive side of approach is that I can even do it after the privatization has occurred, so there isn't extreme hurry. Let's just compile a list and I'll go down it.
>>
Does anyone know how to make youtube-dl print absolutely everything to a file, including errors?
I'm trying to specifically select privated/age-gated videos from a big list of URLs.
I tried piping the output to a file by doing command > file.txt, but that only puts the normal output in the file while the errors still show in the console.
>>
I don't know if any of you guys have seen this yet, but someone on /t/ consolidated all of the pre-2017 videos ripped from unlistedvideos.com into a single text file: >>>/t/1049445
It's about 450,000 videos.
There's pony among them, but I've got no idea how you'd go about finding all of it scattered throughout a list of that size.
>>
>>37284747
Nice
Don't forget the ponychan threads that are still up on the site and not just wayback.
>>
>>37285417
If you're on a gnu/linux system use tee
youtube-dl bla bla bla | tee myfile.txt
For more info on tee
man tee
On your terminal
>>
>>37285451
Okay downloaded it and shared it on smutty.horse
https://u.smutty.horse/mcfaigzqbst.txt
titles are already there so those can be regexed on
>>
>>37285510
I'm actually using exactly that, via Windows Powershell, and it still results in the behavior I described except I can see everything in the console, not just the errors.
Still not getting the errors to appear in the text file.
>>
Is there an indication of which videos on the spreadsheet have already been backed up?
>>
>>37285517
Save some time by deleting the ids you already have

cp 450K.txt bak_450K.txt
while read -r line; do
[4]sed -i "/$line/d" 450K.txt # Use double quotes!
done < some_file_with_already_processed_ids.txt
>>
>>37285521
Merge stdout and stderr
youtube-dl bla bla bla 2>&1 | tee file.txt
>>
>>37284761
Desu has also already been scraped and gone through.

>>37285505
That's a good point, I'll at least see if we can do /arch/ or something.
>>
I barely know anything about how Tabun is organised, but there's probably some blogs on there with youtube videos too.
Some examples to get started:
https://tabun.everypony.ru/blog/music/
https://tabun.everypony.ru/blog/tabun_animators/
https://tabun.everypony.ru/blog/pmv/
https://tabun.everypony.ru/blog/dubbing/
https://tabun.everypony.ru/blog/mlpaudiobooks/
Almost all videos there appear to be pony related. Note that not all youtube links are embedded videos viewable from the pages so individual posts have to be scraped.

Also old looking forum section for videos/music: https://forum.everypony.ru/viewforum.php?f=15
And then the main site itself: https://everypony.ru/
>>
>>37285356
is this the stuff you found on eqd that wasn't saved?
>>
>>37285609
this would be good to go through because at least with these, it's not like a bunch of stupid speedpaints and crap
>>
>>37285660
Yes. Or at least, the stuff I found by going through EQD. It could be that I stumbled upon these videos by digging a little deeper. In any case it seems someone has added them to the spreadsheet already. Do we have a handy tasklist to see which places still need to be scraped, or other stuff people can do?
>>
>>37285666
Yeah I think at this point with under two days left, it's almost impossible to process all the lists. I'd suggest prioritising any lists that are almost certainly guaranteed to be pony, so anything from pony sections like those tabun links. I don't think we'd be doing anymore stuff from unlistedvideos.com after doing a regex for mlp keywords.
>>
>>37284595
Update (not same person)
Downloaded video metadata and comments via youtube API for all pony videos in that hobune spreadsheet up to several hours ago. It's 1.42GB and can share if anyone wants them.
>>
>>37285677
Alright thanks for the clarification. And there really isn't. The Tabun stuff here >>37285609 would probably be good to scrape through, although probably skip the audiobooks. Those are hourssss long and the music, animations, and PMVs are more important anyway. Also the alive ponychan threads >>37285505 , especially on /arch/.

Honestly at this point, we just need to fuckin go ham on sorting shit and keep an eye out especially for music and animations in the sheet. Possibly add MLP playlists you can get online and shove them in too.
>>
>>37285712
I've downloaded audiobooks as videos before and they're usually small and sometimes faster than regular length videos because the file is just a still image with audio. Of course, there's no quick way to differentiate those from other videos with fancier visuals.

tl;dr Yeah skip audiobooks for now
>>
I don't know about other browsers, but reminder that Chrome stores all user bookmarks in plaintext JSON format in a file that can be found in your AppData\Local\Google\Chrome\User Data\Default folder.
You can use this regex
^(?!.*"url": "https?:\/\/www.youtube.com\/.*").*$
with Notepad++'s find & replace feature to clear out all lines that don't contain Youtube links (plus some simpler regex to then trim off the "url": part and the quotation marks,) then use the Remove Empty Lines feature to get a plain URL list of every Youtube bookmark you have.

If you have your pony bookmarks in a specific folder, even better, because the bookmarks file is sequential and labeled, so you can just cut that chunk out separate from all the others before stripping all the information off.
>>
>>37277397
>https://yt-dlp.readthedocs.io/en/latest/README.html
>255kbps audio is extracted from youtube music if premium cookies are given
Just stumbled upon this on the manual page. What's the usual audio quality on videos? Is it worth effort to redownload audio only with this option?
>>
Has this been checked yet?
https://smashboards.com/threads/my-little-pony-general.299338/
>>
>>37285913
I think it only works with music uploaded youtube music, so it would only work for those uploads that have "Provided to YouTube by..." in the description:
https://www.youtube.com/watch?v=_rkmWfh8cUY
https://www.youtube.com/watch?v=WllkHNupwVw
https://www.youtube.com/watch?v=mRLYAgtbQ20
and not a lot of pony music is uploaded like this. I think only bigger music labels can upload videos like this.
>>
Excuse me if I'm being completely retarded and posting things people knew all along, but there's another pack of pony music: https://thatmusicbrony.tumblr.com/post/17257689010/best-of-pony-music-v2-5
>>
>>37285946
Do you mean that only videos with such note would have high audio quality? I've checked a few random videos from my bookmarks and it seems that music.youtube just mirroring all normal videos. So you totally can download all normal YT videos through music.youtube.
>>
Alright, I have everything set to download that's marked as Y in the sheet. However, there's a lot to go through still, like I just found an Egophiliac animatic that might have been lost.
>>
>>37285982
Trying to play normal videos in youtube music just gives me a song isn't available error, so I doubt that would work. I don't have youtube premium so I can't check if yt-dlp would give better quality.
>>
>>37285982
>>37285913
Reading around it looks like the average YouTube video audio is compressed to 126kbps AAC but YouTube Music uploads have the possibility of being put to 256kps AAC. More info here: https://support.google.com/youtubemusic/answer/9076559
It couldn't hurt to try and download them with higher quality audio I guess, but only if they're from marked "music" channels as anon says >>37285946.

Given that these videos on YouTube music channels aren't unlisted anyway it seems like the focus should be on just getting .mp4s for the original horse music videos that will be lost in a few days
>>
>>37283142
>>37283368
Here you go
Disposable script used:
https://u.smutty.horse/mcfbricyrtz.p%E1%BB%B3
By mistake it ends on .pỳ, change it to .py

Resulting json file:
https://u.smutty.horse/mcfbrvrbjra.json

Grep'd youtube urls:
grep -P "(?<=youtube.com|youtu.be).+" mylittleremix_contests.json | sed -e 's@^ "youtube": "@@' -e 's@",@@' > mylittleremix_youtubeids.txt
VIM
:g/^[[:blank:]]\+/d
Deduplicate
uniq mylittleremix_youtubeids.txt > uniq_mylittleremix_youtubeids.txt

983 lines in total
Thread carefully, it contains channel users, youtube redirectors and youtu.be links
https://u.smutty.horse/mcfbsbwqlrn.txt
>>
>>37286021
Yeah. Sorry for misleading. Turns out I've checked PMVs and music videos without label sign. Videos with mostly just visuals are not mirrored to music.youtube.

>>37286056
Thanks for the info! Though not sure how to distinguish if video can be downloaded from music site. Probably some flag in API response? Label sign seems not exclusive attribute.
>>
File: 1623642517760.png (112 KB, 369x401)
112 KB
112 KB PNG
You anons are the best.
>>
Time is running out,folks. Just one final effort!

Keep going!

>>37286493
This
>>
back to page 1
>>
>>37287037
>>
Have the /mlp/ Rewatch Stream playlist pastebins already been fed into https://hobune.stream/submit/ ?
I put all my bookmarks from those through it already and 100% of the videos were already on the spreadsheet, but I don't know whether that's an indication that someone else already put all the Rewatch pastebins through it, or if it's just a coincidence.
>>
>>37287417
I went ahead and just put them in myself, looks like everything at least up to S5 rewatch/January 2021 is already on the spreadsheet. Unfortunately since everyone moved to Ponepaste and Ponepaste is now down, I don't know how to get the pastes for all the rest.
Here are the lists of all the videos in a single document if anyone wants:
With titles: https://u.smutty.horse/mcffggjsznp.txt
Just URLs: https://u.smutty.horse/mcffggjqudt.txt
>>
I've only ever used youtube-dl to download plain audio before, so I've never had to deal with different qualities/resolutions.
Does youtube-dl automatically select the best available quality? Or is there a specific command/launch option that I could use to ensure I'm downloading the best quality?
What options are you guys using for your downloads?
>>
>>37287794
>Does youtube-dl automatically select the best available quality?
Yes
github.com/ytdl-org/youtube-dl/blob/master/README.md#format-selection
>Since the end of April 2015 and version 2015.04.26, youtube-dl uses -f bestvideo+bestaudio/best as the default format selection
>>
What videos are unlisted anyway
>>
File: jj21sx.jpg (12 KB, 480x360)
12 KB
12 KB JPG
>>37286131
Thanks so much for this! For any anons currently going through these links, I've done up to Rainbow and Rooted I (all videos and audios downloaded if they exist, and all links still leading to visible videos fed into hobune.stream).

Should hopefully be able to finish the rest of them tomorrow after some sleep

Good work everybody, final stretch now!
>>
File: 1565001639924.png (1.34 MB, 1981x1387)
1.34 MB
1.34 MB PNG
How are we supposed to know, via the spreadsheet, which videos have been saved already?
Or do we not have to worry about the spreadsheet? Is that stuff being backed up to the pony archive automatically?
>>
File: 905195.png (2.24 MB, 2966x1866)
2.24 MB
2.24 MB PNG
Alright there's a list of 4k links that still need to go through on the sheet, and while a lot of it is non-pony, there's still stuff in there such as taped con panels (there was a 4chan babscon panel in there), ancient pony music remixes, Minty Root rigging videos, and more fun stuff. If three or four people worked on it, we could get that all finished and saved in time (which we assume is midnight in some timezone tomorrow, dunno if it'll be UTC or PST).

Is there anything that's still feasible that hasn't been put into the submit site or gone over yet?

If you want to do any yourself that you're particularly interested in, this is my process:
>Get your list of links and turn them into IDs ( use (?<=v\=)[-a-z0-9_]{11} in sublime text)
>Open https://u.smutty.horse/mcffrlhlnth.html
>Take this set of IDs https://u.smutty.horse/mcffrfiumku.txt and set them in the left box. That's the most recent update for TPA.
>Put your IDs in the right one. What's put in the bottom is everything that isn't on TPA.
>Take the filtered IDs from the bottom and save it in a .txt file.
>Download https://github.com/itallreturnstonothing/panicpony/ and use the get_unlisted_videos.py script to find only the unlisted stuff that's in danger.
>Throw them into the /submit/ area.

I'll still be throwing in playlist of sorted stuff to save, but yeah, this is about it people.

I myself am going to finish out working on finding Everfree Network podcasts (I think we've gotten most of them, but I want to be sure), recheck a few things for the rewatch stream stuff, recheck the last two threads to make sure I didn't miss anything there or in the Gdocs submissions, and then just shove pony playlists I can find of music and PMVs and other things into /submit/.
I'm also considering taking those longass fanfiction readings and just throwing them into a playlist, sending it, and what's saved gets saved.

>>37284595
Here's another update on today:

>Tabun's PMV, dubbing, animator, and music sections >>37285609
Scraped and saved.

>Wayback for FiMchan, Lunachan, and MLPchan
>/arch/ for Ponychan and site captures of old Ponychan and MLPchan
Scraped and saved.

>KYM MLP videos >>37281910
Scraped and saved.

>the DerpyNews stuff and what he got up for EQD >>37281857 >>37285356
Unlisted stuff is saved, I'll have to go back and look at that again someday to save stragglers on EQD.

>>37286131
I believe the unlisted ones here have been saved, but it's always important to have others double check >>37287885

>>37287417
>>37287750
Yes, they've been checked over and saved. I'm gonna do a double check real quick though, there's some pastes I didn't have before.

>>37287936
If the ID is green, it's been saved or is currently being downloaded. I've also saved some board-centric or particularly meme-y videos too.
>>
Are we scraping the information about the videos as well like view count, description, comment count, etc? I know youtube-dl has options for putting that stuff in the output.
>>
File: 1535562199342.jpg (20 KB, 400x400)
20 KB
20 KB JPG
>go through all of my playlists
>find far too many unlisted videos
>get slightly depressed realizing just how many pony videos and channels were unlisted and difficult to find/access (gravey for one)
>submit what I have
>only like two were actually added
Goddamn you based anons, you got practically everything I could find.
>>
>>37277397
>List
>Sort
>Scrap
>Post!
https://www.youtube.com/watch?v=Pq61n9T1TjY
>>
Any final stuff that needs to be scraped/filtered? The AI stuff didn't work out. I know a bunch of sites were suggested, but I'm not sure what hasn't been done yet and should be prioritized.
In the meantime I'll work on the spreadsheet.

>>37284863
It didn't work at all. The pony face detector identified some pony faces, but it also identified a lot of regular faces. CLIP returned a bunch non-pony videos. The prompt was probably overfitted on the Top 10 set.
In short, anon was right that there wasn't enough time to properly setup an AI. I suppose it was worth a shot, though.

>>37284914
Thanks for providing these thumbnails, even if the AI didn't work out.
>>
>>37288148
https://www.youtube.com/watch?v=ZJo7GoQdC9A
>>
>>37288478
could try
https://geek.cheezburger.com/bronies/page/1840 (this is just the last page, it's the original MyLittleBrony site)
http://www.equestriandreamers.com/forum/
https://eqmegathreads.blogspot.com/
http://www.equestriagaming.net/

>>37288481
>its unlisted too
how fitting
>>
>>37288489
>>37288478
oh and if you do, make sure to only get the pony-centric portion of the forum there, just wanna pump things in at this point if we can
>>
Are comments on videos also being saved?
>>
>>37288489
>http://www.equestriagaming.net/
I'll see if there's anything unarchived here.
>>
>>37287956
In the event that we don’t sort everything on the spreadsheet before the deadline, I would say just download everything that hasn’t been sorted and go through it after the fact
>>
>>37288489
Okay, I'm scraping all pages of cheezburger, the video tag on eqmegathreads, and mylittlegamedev.com (linked on equestriagaming.net)
>>
>>37288086
Video info for anything marked as pony in the spreadsheet is being downloaded via youtube api.
>>
>>37288545
Here are the results from that scrape: https://u.smutty.horse/mcfiivxbbta.txt
Good luck to everyone as we enter the last day. Or maybe second-to-last, depending on when the exact cutoff is.
>>
>>37288617
In previous experiences, it tends to be around 9am PST (youtube headquarters).
>>
dawn of the final day
>>
>>37288648
Do we have a precise deadline, or is it just going to happen /somewhere/ on the 23rd?
>>
>>.37288648
This is the /end/...
>>
>>37288665
probably UTC or something
>>
>>37287956
>I myself am going to finish out working on finding Everfree Network podcasts (I
https://www.youtube.com/watch?v=NMcsg_zYqvQ&list=PLP9KbUU6B2IhO1vBooEq_P6hbNhXl7hy4&index=2
Ya got this one?

>>37288491
Yes, from my understanding, TPA, the anon who is posting the giant lists, and I, are saving them.

>>37288642
9 am today or tomorrow?
>>
>>37288677
>Yes, from my understanding, TPA, the anon who is posting the giant lists, and I, are saving them.
Sorry, meant for:
>>37288500
>>
>>37288677
as in 9am pst 23rd july
but don't actually rely on that
it's just something I've noticed
>>
>>37288668
came here to post this.
>>
>>37288677
Yup, sure do.
>>
>>37280739
No, go fuck yourself
Pony is to be preserved
>>
>>37280739
Petition DENIED!

Get out of here.
>>
>>37280739
Kill yourself, traitor. They'll stay preserved even after you die.
>>
>>37288489
>>37288533
Equestria Gaming has been scraped. Enjoy your abandoned pre-pre-alpha game footage. For unlisted videos I found some EquestrianDreamers videos and one from ChurchCrusade, but my biggest catch is 6 hidden Hikouma videos when I went on a tangent through some playlists and I found out he had links shared on his Facebook.
https://u.smutty.horse/mcfjvkyufyy.txt
>>
>>37288938
Apparently only 1 of those Hikouma videos hadn't been added yet. No worries, that's still content preserved. And for EquestrianDreamers I accidentally had "contains some pony" selected when of course it should be 100% pony, I hope that gets fixed.
>>
noticed there's a few little gaps in the hobune spreadsheet with videos not marked/downloaded. Would going through and marking those as pony/downloading them myself be helpful?
Or are they being skipped for a reason?
>>
>>37288969
I have been going through some of the gaps and have found pony videos in there, so I think it's worth doing. I assume one Anon just went over and added Y to any that had obvious pony titles
>>
File: YayThisIsFuun.jpg (78 KB, 772x465)
78 KB
78 KB JPG
>>37277397
>https://derpy.me/4RApE
>>
>>37277397
were unlisted YTPs saved? I remember Danfango unlisted a bunch of shit he didn’t outright delete
>>
>>37288994
We've got Danfango covered
>>
Does anyone have the TheDashDub / Chris Heron playlist? I wanna listen to it before it's gone.
>>
File: 2651204.png (212 KB, 1280x720)
212 KB
212 KB PNG
Hey, I keep looking through the last two threads, but I can't seem to find a video that someone wanted to archive. It's unlisted (of course) and appears to be an animation with a couple horses, one of them looked blue I think.
If anyone knows what I'm talking about, please reply to me with it to ensure it gets saved. It looked real cool.

>>37288617
>>37288938
done and being downloaded

>>37284863
I actually was able to do this too, thought it was way out of the league with how many videos it was, but it actually wasn't too bad.
>>
>>37289026
oh and pic related is me passing tf out right now
>>
>>37280674
Videos can also be in this form: http://www.youtube.com/embed/{videoId}
>>
>>37289015
https://www.youtube.com/playlist?list=PLVLR1RcoLXdheTJY6IQ7jXHDU4vDLXvjl
>>
>>37289031
I don't know if he's even active anywhere online, but if you want to try contacting him about the videos:
https://chrisheronmusic.wordpress.com/contact/
dashdub888<at>gmail<dot>com
>>
>>37289029
It should now also support these links. I'll later rescan and submit links from the websites I've scraped.
>>
>>37289033
Godspeed Anon, I'll do that. Again if people know any more active artists with doomed videos please share them here.
>>
post videos you're going to be sad to see gone
https://www.youtube.com/watch?v=y7tsI8U8QHw
>>
>>37289031
There's some trash at the end of playlist. Wasn't there his own official playlist? I remember him posting it somewhere in comments or description but can't find now.
There's a video in which he announced he'll be moving on https://www.youtube.com/watch?v=Gyr4gh1Tbbs
>>
>>37289047
Interesting. Could've sworn it wasn't always like that. I'm afraid you'd have to go through all the descriptions because I can't remember him posting such a playlist.
>>
>>37289047
>>37289057
Welp, nope. Nevermind. I was mistaken. I found his comment and he had linked this exact playlist. You can find his comment here: https://www.youtube.com/watch?v=RYfobILpGfw
>>
Another set of ids: https://u.smutty.horse/mcfkxiaelnz.txt
Apparently scraped from a collection of pony-related sites.
>>
What's the plan if we aren't able to sort all the videos on the spreadsheet by the end of the day?
>>
>>37289216
lie down
try not to cry
cry a lot
>>
>>37289216
Just start download everything unlisted among unsorted. It's better to sort later than not have anything at all
>>
so where is all this being stored...? who's paying for the storage?
>>
>>37289216
Sorting is just checking if videos contain pony or not and marking it on the spreadsheet?
>>
>>37289277
yeah
>>
File: AllRiiight.jpg (238 KB, 1527x937)
238 KB
238 KB JPG
>>37277397
>>
File: TheseNeed2BSaved.jpg (140 KB, 1146x715)
140 KB
140 KB JPG
>>37277397
:P
>>
File: EveryLastOne.jpg (79 KB, 957x543)
79 KB
79 KB JPG
>>37277397
>>
Found this while sorting
https://www.youtube.com/watch?v=606O6RmdzVc
>Treasuring the times, memories of friends and loved ones by keeping them close to your heart will always be with you, no matter how far apart they seem to be.
>~That is the true magic of love and friendship~
That's a Y.
>>
>>37289434
This one must be saved
>>
File: unlisted screencap.png (2.36 MB, 1812x948)
2.36 MB
2.36 MB PNG
https://www.youtube.com/playlist?list=PLFmK89wpHaNV4VZbE4TMHtyM5gBvJf7u9
Found some Starcraft 2 group named Team Liquid livestreamed a pony marathon back in 2012, I was excited to see 112 unlisted videos that hadn't already been added to the spreadsheet until I realized that most of them were just part of the streams t-shirt giveaways.
Well they also got a personal message from Ashleigh Ball, that's pretty neat.
https://www.youtube.com/watch?v=kv6CCs4_lts
>>
>>37289235
Yeah, to be safe I'm downloading literally everything I mark with a Y, better safe than sorry

We can just check 'em against TPA later
>>
File: 1520246.png (1.44 MB, 5000x9347)
1.44 MB
1.44 MB PNG
I was never more than a casual lurker here, and it's been years since; what the fuck is going on, is this a pony videoacaust?
>>
>>37289615
Pretty much
>>
>>37289615
Everything unlisted prior to 2017 is about to get killed. Pony most affected.
>>
mylittleremix is now fully saved (the links that are still live, anyway)
>>
https://www.youtube.com/watch?v=aNsgmcqZZi8
Cursed or blessed?
>>
>>37289611
Please priotizie downloading what's unsorted and/or with a K first, I'm not touching the unsorted as of right now and the K is really long stuff that is just.. really long.

>>37289836
blessed as fuck

>>37289827
yay
>>
>>37289836
unfathomably based!
>>
>>37289844
K also includes stuff that's already saved elsewhere, are reuploads that are already downloaded, or the original is still up and again probably save already. I put some comments on some of them.
>>
>>37289844
This. We should absolutely save everything unsorted if we run out of time. Better to have to sort though it all afterwards than lose it forever
>>
>>37280739
you'll be fine anon, my lasting legacy will be a lazy 2010-era /v/ meme involving pink dildos and a mac that somehow made its way to bitwit's meme review literally more than 10 years later
>>
>>37289836
Proigral.
>>
Hey, we could get all the stuff that's with a K into https://ajay.app/at-youtube-submitter/ . They're downloading everything that's put in there at 360p, but for a lot of these super long livestreams and fanfiction readings, you don't need anything more than that.

If someone wants to start getting a list together, we can submit that.
>>
>>37289896
Maybe it would be a good idea to sort the spreadsheet by length and just paste the longest ones in there?
>>
>>37289907
The spreadsheet's broken a bit so it wont work.

Here, I'll just get a list of stuff for now. Please don't sort anything.
>>
>>37289907
https://u.smutty.horse/mcfmsdvedpu.txt
Here's the list of K's currently not saved in TPA. Someone else please submit it to there for now.
>>
https://www.youtube.com/watch?v=3FE_z9ZEQLo
https://www.youtube.com/watch?v=84AeLxhFKEk
https://www.youtube.com/watch?v=Ac7DNRR03aE
https://www.youtube.com/watch?v=CiTDzzVlt0A
https://www.youtube.com/watch?v=IhlYpExPL7w
for people who are sorting, stuff like the above gets a Y
>>
>>37289968
https://www.youtube.com/watch?v=KPD8zqnU0RE
this ESPECIALLY gets a Y lol
>>
Cute Christmas time OCs
https://www.youtube.com/watch?v=bgQ1gMFWKc0

>>37289968
Even as little as a cutie mark in a test card? Fair enough, my bad.
>>
>>37289990
it's cool, and yeah a short video like that isn't gonna take up a whole lot of space at all. Wanna be pretty liberal with the Ys.
>>
https://www.youtube.com/watch?v=bKvNF0sXCqA
This might be WIP of a pony song, I'll put it out here for you to decide.
>>
>>37290012
https://www.youtube.com/watch?v=BYLFeVo5B8w
idem dito
>>
>>37290012
>>37290064
>BlackIceMusic
yes, very much assume so
>>
File: 104487transparent.png (197 KB, 500x600)
197 KB
197 KB PNG
Alright, just about everything in the sheet that has a Y should be being downloaded right now or, if it was super long (think like half an hour or more), was submitted to the Internet Archive's project to be downloaded at 360p.

Almost there people.
>>
>>37290293
make sure that the stuff that hasn't yet been given a Y or N gets downloaded if we can't mark it all in time
>>
File: 418645.png (118 KB, 647x425)
118 KB
118 KB PNG
>someone dumps another 825
dude..

>>37290298
yeah, think I'm gonna do that now
>>
Maybe this was already checked, or maybe it's too late, but I just remembered that there is a list of basically all pony subreddits: https://www.reddit.com/r/GreatPonySubDirectory/wiki/index
>>
>>37290379
That big chunk in the middle where it goes from - to 0 to 9 and then from A to Z is part of a huge scrape of 200+ subreddits, so I would hope/think anything from those subreddits should be in there. And what's in there is left over from performing regex on the titles and tags of the videos already, so hopefully most things have already been caught and downloaded.
>>
Is this pony?
https://www.youtube.com/watch?v=fGkxkqvXgnw
>>
File: 376658.gif (27 KB, 116x160)
27 KB
27 KB GIF
I have a request to find a video if someone is up for to the task.

https://www.youtube.com/watch?v=TNIrMVHZhVU
This is a series called "Evolution of a Scene" which details how a scene moves from storyboard phase to flash to animation to clean up. There were 6 of them, but they were deleted way back in season 1.
#1, 2, 4, and 5 were the only ones publicly found until now when someone linked #6 in the spreadsheet. If #3 can be found on this person's channel somehow, we'd be golden. This is like, one of those holy grail things for me, so if we can find it, I'll be over the moon I'll make sure to give a wave to Luna on the way over.

I put a comment on this one too, and thank you to anyone willing to lend a hoof.
>>
>>37290541
I believe he's done pony stuff before, so yeah throw it in. Obscure artists are always nice to come across. It's such a small amount of space it'll take up anyway.
>>
What do we mark down for barbie/EQG videos?
>>
>>37290628
Y. It blows, but really there's not really all that many anyway it seems.
>>
For the scrapers, here's another forum that may have some videos:
https://brony.team/forums/index.php
>>
>>37290641
I don't think I can write a scraper for this in time
So, I'll leave one I found in there
https://www.youtube.com/watch?v=PiBohDujLPs
>>
>>37279432
>>
Alright, caught up again.
>>
Shiiit how long do we still have?
>>
>>37290921
Between 6 to 15 hours, depending on when they pull the plug.
>>
File: 1611584573699.jpg (37 KB, 640x475)
37 KB
37 KB JPG
>>37290921
5 hours
>>
>>37290937
>>37290938
Fug
>>
>>37290641
>>37290681
Forget about that, but do mind, this is the best I can do in less than an hour
Disposable script:
(Warning, it scrapes only the The Lounge section!)
https://u.smutty.horse/mcforqsndgc.py
Cleanup (VIM Regex):
:%s/^\[.\+e\]//
:%s/\[.\+\]//
73 videos, contains non-pony
https://u.smutty.horse/mcforxbpnqd.bin
Change .bin to .txt on your fm
It has that extension because saved it with none
Good luck on the final stretch!
>>
>>37291080
looked through it all and we good to go

this was one of the videos in there, what kinda weirdass shit these people talking about?
https://www.youtube.com/watch?v=NX_vcjZmQ9w
>>
>>37291128
It's from Adventure Time.
>>
https://www.youtube.com/watch?v=o5gw0qUPxgU
>>
How many videos have yet to be sorted?
>>
File: GetThatStuffOffaThere.jpg (18 KB, 480x360)
18 KB
18 KB JPG
>>37280739
>>37280747
>>
File: MOOOOOOOOOOON.png (138 KB, 943x206)
138 KB
138 KB PNG
>>
>>37291350
I think this is manageable. There are a couple of pockets of no more than 120 videos, and a large one between lines 33168 and 34763. I need to catch some sleep now, hope to help a bit more in the morning.

https://www.youtube.com/watch?v=Etyefd24yds
>>
>>37291532
>In the morning
Can we be sure the videos will still be there then?
>>
FEW HOURS REMAIN
>>
>>37291537
don't count on it
I think it'll kick in once morning reaches
>>
https://www.youtube.com/watch?v=SQAka_Oe6R4

Last chance to watch this masterpiece from 2006
>>
>>37291545
In that case, I would suggest someone try to download as many unsorted videos as possible. They can be deleted again once sorted, but it would suck to lose some hidden gems when we’ve gone through this much effort
>>
So is the plan to reupload the videos to ponytube or host them on the ponyarchive? I wish I could have helped more.
>>
>>37291564
I'm massdownloading the bottom 500(?) or so videos on the spreadsheet which hopefully will save a good few decent videos
>>
>>37291583
For anyone else planning to save anything else down the bottom, I'd suggest sorting by number of views and saving in that order. Idea is that more people may tend to remember the higher viewed videos as opposed to some random video watched by only 10 people.
>>
>>37290555
sorry anon, the uploader responded on your comment saying episode #3 in particular is lost. That's a real shame.

Do you have the other 5 downloaded? I'd love to see them all together if you have them collected
>>
Hello all, I want to search through my youtube favorites playlist to find unlisted pre-2017 videos.

Which script in OP do i use?
>>
>>37291799
find_unlisted_videos_in_playlist.py from here: https://github.com/itallreturnstonothing/panicpony
>>
File: CnTYPUB.png (38 KB, 918x290)
38 KB
38 KB PNG
>>37291608
Fuck! Well that sucks, as in the other EoaS video they had, which was also in the sheet, they commented with this image that shows them all. Granted it was 9 years ago, but they were there. Drat. Oh well.

You can find the other four on YouTube by searching" "evolution of a scene" mlp", and now we have this one saved too, so that's good. I don't know why they say it's lost though, are the other ones just not uploaded to

>>37291799
Look in panicpony and its the find_unlisted_videos_in_playlist.py . A little late now but might as well.
>>
>>37291812
>>37291813
So, I don't want to out myself as a complete retard but how do i use this code.
>>
>>37291812
>>37291813
NTA, but what's the etiquette on sharing Hobune with other people/threads?
It's very convenient, and it does have a "contains no pony" option, but I realize that it would also be cluttering the single spreadsheet that we have with more non-pony stuff. Or is that not a big deal as long as it's clearly marked?
>>
>>37291821
If you have a link to that playlist I can quickly scan it for you.
>>
>>37291824
I think that hobune is only for pony videos. For other videos there's this git repo:
https://github.com/ArchiveTeam/youtube-items/
and this site:
https://ajay.app/at-youtube-submitter/
which will get archived by archive team to the internet archive (in 360p).
>>
>>37291827
I appreciate the offer but it's a private playlist and undoubtedly has some personal stuff in it. And, i'm not trying to search for pony material.

So I download python 3, download whatever this 'requests library' is, and then i can paste the code in or something and use it?

t. computer brainlet
>>
File: 682.jpg (131 KB, 1000x1000)
131 KB
131 KB JPG
Look, things are getting down to the wire.
If you happen across a particularly good video(s), please reply here and I'll draw up a playlist/feed it to the bot to download. It feels like there are some people who are intentionally putting all this weird, low-quality speedpaints, foreign language stuff, and non-pony stuff without marking it as so, so try to extract the gems please.

We can at least say though that we downloaded and checked most of the most prominent sites that we could think of and that are still alive (or even not), so I think we've done our homework.

>>37291821
You need to download python among other things. Best to just submit it to the sheet.

>>37291824
I mean, we've been here for over a month now, so you can if you want, but it'd probably be best to just reply here to this post here because something isn't automatically saved just because it gets submitted. I actually thought there was some download behind the scenes just in case until a half a week ago as a failsafe, but that was my error. I've been saving them manually this whole time, up to like playlist 64 or something and they all have over 200/250+ videos lol

>>37291843
Yeah this >>37291834 might be the best option for you. We don't really have time to walk you through it at the eleventh hour.
>>
>>37291834
>I think that hobune is only for pony videos
If that were the case though, why would it have a "does not contain pony" option for the submission box? That would only be used if people were submitting links/playlists that they knew for a fact were not pony, so why have it exist?
>>
>>37291843
You need to download and install python3 and the requests library and then paste the links to playlists.txt. Then you need to run the command line and go to the directory with find_unlisted_videos_in_playlist.py and run python3 -i playlists.txt -o output.txt. It should then save unlisted videos to output.txt.
>>
>>37291850
I'm guessing it's also useful for future use when filtering out nonpony links.
>>
>>37291849
>something isn't automatically saved just because it gets submitted
I know that, I just meant as an easy "shove in dozens of videos to find the one or two that are actually unlisted+pre-2017 so you know which ones to manually back up" tool.
>>
>>37291850
The guy who set up the spreadsheet also archives stuff that isn't just pony, so I believe that's why. However, it's tailored to help us filter for pony stuff.
>>
>>37291855
how do i add requests library?
>>
>>37291868
You need to run that command in cmd not in python.
>>
>>37291843
>private playlist
That script can't look at private playlists.
>>
>>37291868
okay you don't want to actually run the python.exe application (the one that's currently giving you ">>>" in the screencap)
As >>37291873 said, you have to open cmd (ie. command prompt) and run the command in there
>>
>>37291876
Also yeah I just realised this. However I think there's a way to get youtube-dl to output a list of videos from a playlist, and youtube-dl also allows for account credentials to be added. That could be a way for this private playlist.
>>
>>37291886
Type it without the dollar sign
>>
File: halp.jpg (7 KB, 485x63)
7 KB
7 KB JPG
>>37291873
>>37291877
im failing

>>37291876
>>37291887
can i just unprivate the playlist, run the scan, and private it again?
>>
File: clyp.it-5wdhyg4b.jpg (126 KB, 1280x1020)
126 KB
126 KB JPG
>/mlp/ - Pony
>coordinated effort from the start
>made multiple public tools to automatically scan URLs/playlists and grab videos
>even made a website to facilitate it
>built a spreadsheet of 42k known videos
>backed up over half of them

>/g/ - Technology
>barely talked about it despite the fact that hundreds of thousands of non-pony videos are also being lost
>made nothing
>did nothing
>>
>>37291892
yeah you can do that
>>
>>37291887
if you do that you need to use find_unlisted_videos.py and feed it a list of video ids.
>>37291892
>can i just unprivate the playlist, run the scan, and private it again?
that will work too.
also >>37291891
>>
File: pythonas.jpg (9 KB, 957x52)
9 KB
9 KB JPG
>>37291891
>>37291896
>Type it without the dollar sign
>>
>>37291901
run "cd PYTHON_FOLDER" where PYTHON_FOLDER is the folder where you have python
>>
>>37291901
>pythonpath issues
dawg just go through it manually
>>
>>37291908
the playlist that is
>>
>>37291908
>>37291912
What if it has like 500 videos in it?
>>
>>37291908
>>37291904
I think i'm gonna give up. i'm worried im shitting up your thread with all these posts.

>>37291908
>>37291914
>dawg just go through it manually
the playlist contains nearly 5000 videos.
>>
>>37291918
https://www.geeksforgeeks.org/how-to-install-requests-in-python-for-windows-linux-mac/
https://www.youtube.com/watch?v=HdJywzSCGbc
>>
>>37291918
>i'm worried im shitting up your thread with all these posts
It's fine, it's not like there's much other discussion going on right now.
Besides, this could end up helping someone else with the same problem.
>>
why isn't submitting videos working?
>>
>>37291926
>>37291923
just to confirm, i need Python 3.0 and not any newer versions ie Python 3.9 or whatever? because i installed 3.0 if that matters.
>>
>>>/g/82654991
elem mayo
>>
>>37291901
You can do as >>37291904 suggests. Basically cmd currently cannot find python unless it's in the very same folder as python.

However if you want to able to run python anywhere, you'd want to add python to your system's environmental variables. To do that, type "edit the system environmental variables" into that windows start toolbar and select it. Then click Environmental Variables. Select the Path field and then click the Edit button. There should be a list of paths. Click New and paste the path where your python installation is located. Click Ok and from there you can run python commands without having to always be in that folder.

>>37291932
I think it should work on 3.0, but I think everyone these days is on at least 3.6.
>>
>>37291932
The newest version should work. I don't know if it will work with python 3.0.
>>
>>37291928
somebody might have put in a ton and it's taking forever to show up
if you have some that you know you want saved, please post reply with them because right now I'm just picking and choosing from the sheet, it's about all I can do at this point
>>
>>37291849
If anyone wants to find more pony videos here's 2 lists that might still contain pony videos:
https://warm-cat-99.loca.lt/
I already searched the 1st one for pony videos but it might still contain some pony videos. Anyone can easily browse these, just go to that address click continue, select 1st or 2nd list and click on videos to mark them as pony related or not. It will only be saved locally so you need to copy that list to for example hobune and submit it. I've already searched the first 272 pages of the 1st list sorted by date and also by searching for pony keywords.
>>
>>37291928
>>37291957
I'm looking at the logs and it seems up-to-date
>>
File: success.jpg (65 KB, 968x372)
65 KB
65 KB JPG
>>37291939
>>37291940
>>37291923
>>37291926
I did it! I think...

What do I do now?
>>
>>37291966
>>37291966
you have your playlist link right? and it's public? run this in cmd. make sure you're in the folder where python.exe is:
echo "YOUR_PLAYLIST_ID" | python find_unlisted_videos_in_playlist.py -i -
>>
>>37291974
>python find_unlisted_videos_in_playlist.py
do i need to download this script first and put it somewhere?
>>
There are still some unsorted videos on the 33168
>>
>>37291977
oh yea. you do need to do that. either put it in the same folder as your python exe or specify the whole path with something like \users\<you>\Downloads\find_unlisted_videos_in_playlist.py
>>
>>37291849
https://www.youtube.com/watch?v=8jrUNXw5O0w
>>
>>37291986
omnomnom tasty bait
>>
File: common.jpg (23 KB, 1122x89)
23 KB
23 KB JPG
>>37291985
I think I did it right. I just opened the "raw" text for the scrip and right click+saved into the python folder.

Stuck here now, any ideas?
>>
>>37291894
Mama Faust didn't raise no slackers
>>
>>37291994
well you posted the playlist id so I did it for you.
https://u.smutty.horse/mcfqvckdotr.txt
>>
>>37292011
whoops. oh well. i guess it worked out in the end.

thanks anon.
>>
Remember this sad truth, all the porn shall remain
>>
>>37291994
You'd also need to download common.py.
>>
>>37291894
/g/ doesn't do anything now. they're rather soulless.
>>
File: 1590609275201.gif (147 KB, 500x281)
147 KB
147 KB GIF
>>37292016
happy archiving
>>
File: 20210722_202425.jpg (21 KB, 260x197)
21 KB
21 KB JPG
Panic panic panic

>PANIC

PANIC

>PANIC

PANIC

>>>>PANIC<<<<
>>
File: nice.png (29 KB, 533x789)
29 KB
29 KB PNG
>>37277397
Am I doing something wrong here?
>The moment you hit Send it switches selection to videos from playlist, every time
>>
File: 1615484917423.png (76 KB, 350x264)
76 KB
76 KB PNG
>>37277412
>>
>>37292067
It may be identifying it through a regex as playlist
Remove from &list= to the end and try again.
>>
>>37292067
You didn't pray to Celestia first.
>>
>>37292067
you have to submit it from the playlist URL itself, not the video one
>>
anyone here good with youtube-dl?

I'm trying to set a folder for everything to download into, using "-o the/folder/i/want/it/to/go/into" but all it does it makes the video save with the filename of the last folder (in this example, it would name the video "into.mkv" and place it into the "go" folder.
>>
>>37292080
Looks like you are missing a slash on the filepath.
>>
For how much longer can anyone save anything? Less than 30 minutes?
>>
>>37292080
-o is a switch that interprets your string
You need to supply the correct string

Incorrect: Leaving only the path to the output
-o "the/folder/i/want/it/to/go/into"
Correct, writing the path then the string that will format the filename
-o "the/folder/i/want/it/to/go/into/%(id)s-%(title)s.%(ext)s"
https://github.com/ytdl-org/youtube-dl/blob/master/README.md#output-template
Here you can find all strings ytdl lets you put into the filename.
>>
>>37292106
Ah, that's it. thanks bro
>>
https://www.youtube.com/watch?v=0r8l28AM81M
Thank you, who ever saved this one
>>
>>37292067
That's because you're submitting a video link
>>
File: 1004478.gif (138 KB, 216x188)
138 KB
138 KB GIF
>>37292059
>>
File: anon mlp i'm scared.png (255 KB, 1000x948)
255 KB
255 KB PNG
>>37292149
>>
>>37292072
>>37292074
>>37292077
>>37292135
cheers, got it
>>
One minute until removal. DAMNIT!!!!
>>
File: 87852.png (714 KB, 3564x6000)
714 KB
714 KB PNG
>>37292132
HORSE
you're welcome
>>
>>37292183
Still have 12 hours to go, assuming that 9am PST is the actual time
>>
>>37292183
>>37292204
at the very earliest it would be midnight pacific time right, since thats where youtube is headquartered?
>>
>>37292226
Yes, it will be July 23 for Youtube HQ in less than 3 hours
>>
I searched through the sheet for keywords:
>animation
>animatic
>panel
>live
>concert
>bronycon
>eqla
>everfree northwest (and combinations thereof)
>trotcon
>babscon
>ponycon (should catch a few)
>larson
>keating
>faust

Can anyone come up with some other ones that might be important?

Actually came up with this one for Faust where she talks a little bit about her would be plans for The Galaxy Girls: https://www.youtube.com/watch?v=RrFelmdlVBw
>>
>>37292269
just searched for
>ashleigh
>libman
>tabitha
>tara (with a space before/after)
>>
>>37292269
>>37292281
This is a good idea to do now to quickly pick out videos.
>>
I'm randomly going through the spreadsheet and saving everything one by one that catches my eye. Seeing the pure devastation that is about to happen hurts my soul.
>>
https://www.youtube.com/watch?v=0TSqLHEFRqk
>>
>>37292397
Where is the spread sheet
>>
>>37277397
What the fuck is happening?
>>
>>37292481
See >>37277397
https://derpy.me/FkhNl

>>37292492
Death.
>>
WHERE IS FLUTTERSHY BUTT CRUSH
>>
File: 1556389768072.png (334 KB, 1600x1747)
334 KB
334 KB PNG
>>37292492
>he doesn't know
>>
File: why.png (21 KB, 500x500)
21 KB
21 KB PNG
>>37292059
>>
I don't mean to be rude but why on earth the indicator for "saved" is the id cell colored?
Can the anon saving them give me a list of those ids? I'm looking at how to select all colored cells but seems like that's a no-go
ask.libreoffice.org/en/question/46352/how-to-filter-rows-using-row-color-in-libre-calc/
Extracting the table doesn't help, contents.xml only has the text but no formatting options and just crashed the browser, as vim only shows two lines and all cells are shown concatenated
>>
>>37292545
You need a list of all IDs that were saved right? Not the other way around?
>>
for some reason command F does not work on google docs how would i go about checking if the video is already posted
>>
>>37292555
The ones saved (background color green) yes
>>
>>37292559
Ctrl+F works here, but you can try Edit>Search and Replace and only search
>>
File: 69469p.jpg (191 KB, 1200x900)
191 KB
191 KB JPG
>>37292564
here you are: https://u.smutty.horse/mcfrzhmytzt.txt
Note that it's not entirely accurate, but should mostly be. Some of the longer ones might not be saved, but they should at least be in the Internet Archive's project in 360p. And at the end there, I wasn't highlighting green for every one being saved.

>23440 IDs
holy fucking wew and there's some other stuff I saved on top of that. it might even out to around 23k vids saved
>>
>>37292572
>And at the end there, I wasn't highlighting green for every one being saved.
That will do still. Thanks.
>>
>>37291849
Here's a few I kinda liked
https://www.youtube.com/watch?v=xUrgsdpsnNQ
https://www.youtube.com/watch?v=Ys2y94W38pk
https://www.youtube.com/watch?v=CUwUeOwbkBo
https://www.youtube.com/watch?v=5IbF5i3qxzA
https://www.youtube.com/watch?v=Oc0c41XjIh8
https://www.youtube.com/watch?v=hGbo6IABp94
https://www.youtube.com/watch?v=E_YYSxpURvA
https://www.youtube.com/watch?v=Oc0c41XjIh8
https://www.youtube.com/watch?v=A4WXhjeEdzY
>>
>>37291849
This might have a few alright ones, not entirely sure what's in already and what isn't
https://www.youtube.com/playlist?list=PLC630PLg6wVIvHW2aZJoZJLCMcn5Hd50r
>>
>>37292583
>>37292597
added
>>
>>37277397
>>37277408
You need an anchor post.

For people with a separate YouTube account for pony videos, you can dump your "liked" videos with this:
- Open your Liked Playlist in Youtube.
- Hold down Page Down until the full list loads. This will only load 200 videos at a time, but it will eventually load the full list into the page.
- Open up the Developer's Console. In Chrome, you can do this with Ctrl+Shift+J. In Firefox, this is Ctrl+Shift+K.
- Copy/paste in the script from this page: https://pastebin.com/eYQ3aYGd and run that.
- If you have too many links, there should be a "Copy" button that shows up on the bottom. Otherwise, it will write out all the video links.
- You can do the same thing with your Watch History. Go to https://myactivity.google.com/activitycontrols/youtube and do the same thing. The same script will work on this page.
>>
gentlecolts, femanons
we have done more here, than what little can be Humanly done
we have scoured the internet, and told all who would listen
our numbers lessen by the day
and the promised lifetime of pony content shrinks in the cover of night
it is the way things are to go, and it is the way which we uniquely reject
we know that there are some fires which are worth keeping bright
and everyone who doesn't, is furthered by nothing more than our efforts
https://youtu.be/-ULOzaDoxZs
>>
File: 20210722_203630.jpg (198 KB, 1208x671)
198 KB
198 KB JPG
Thank you everyone for being so dutiful and so heroic in times of great loss and against all the odds.

You have performed beautifully. Your efforts directly saved much of our history from the brink of collapse. Thank you all for fighting this fight.
>>
File: 119329.png (339 KB, 3911x3265)
339 KB
339 KB PNG
>>37292664
>this is probably one of the last times we'll be able to watch these videos and its brethren
damn man..

you know what, fuck it
anyone that wants to, get in here and we'll watch them together one last time

https://derpy.me/OdpD1
https://derpy.me/OdpD1
https://derpy.me/OdpD1
>>
>>37292664
this cuts.
>>
>>37292693
will start at 2:45 ET so about 11 minutes
>>
>>37292684
so the big question, how many thousands have been saved?
>>
>>37292705
At least 23,000 videos, according to the spreadsheet.
>>
What should you all feel if YouTube changed their mind last minute. Or in a week or two
>>
>>37292710
and that's just what we found and saved. it should come to around ~435k total for all videos saved in TPA
>>
>>37292713
Happy. All of this this had to have happened sooner or later.
>>
File: 1614357780135.jpg (233 KB, 1280x960)
233 KB
233 KB JPG
This last stretch of time is an interesting feeling.
Like we all grabbed everything we could in a mad rush and evacuated a city, to now stand silently on a nearby hillside and watch the incoming meteor just fucking obliterate it.
>>
>>37292719
>it should come to around ~435k total for all videos saved in TPA
As in, 435k unlisted pre-2017 videos, or 435k total videos saved in TPA to date so far?
>>
This is what I have gathered so far from the spreadsheet (Saved at 00:06 AM)
Pony videos: 24795
Kinda pony videos: 590
Uncategorized: 26985
Criteria:
Pony videos = Y
KInda pony = K
Uncategorized = Anything else
Btw, what's the X for? Deletion?
>>
>>37292724
The latter
>>
>>37292693
yay
>>
https://youtu.be/pV30xIV7WCk
>>
>>37291849

https://www.youtube.com/watch?v=3ujqUW4SB-c&t=2s

https://www.youtube.com/watch?v=P-pZNjme5po&t=2s

https://www.youtube.com/watch?v=EvtfkM4qOgk&t=1s

https://www.youtube.com/watch?v=lxfzzAl6wT4&t=1s
>>
>>37292757
>>37292767
saved
>>
>>37291849
someone shared these videos i thought i would post them here

https://www.youtube.com/watch?v=IbfHL62vOLQ&t=5s

https://www.youtube.com/watch?v=AJP4X3VjDIA&t=1s

https://www.youtube.com/watch?v=pp0u7vHLqTs&t=1s

https://www.youtube.com/watch?v=XA3j7Qxso4U&feature=emb_title

https://www.youtube.com/watch?v=-VqER8WLEFM&feature=emb_title

https://www.youtube.com/watch?v=j151VzbdMno

https://www.youtube.com/watch?v=w1zbGFGQTq0

https://www.youtube.com/watch?v=sQI0kux0EKw
>>
File: 1578865191133.gif (251 KB, 350x417)
251 KB
251 KB GIF
So. When does it all go silent?
>>
>>37292774
thanks anon
>>
It’s just so sad to think how much will be lost, but at least thanks to you guys we have a lot secured and saved, thank you to everyone who helped save our ponies!
>>
File: 176912.jpg (22 KB, 500x558)
22 KB
22 KB JPG
>>37292787
thinking about keeping the stream going until they break
>>
>>37292787
the pain is that we dont know
>>
>>37292795
where can we go to watch it
>>
Keep submitting stuff guys
We keep saving until it's over
>>
>>37292798
the archived stuff i mean
>>
>>37292798
https://theponyarchive.com/
>>
>>37292798
There was a userscript in the previous thread which will automatically search for the video on hobune when viewing an unavailable video on youtube. I'd link it but I'm quickly trying to save stuff too.
>>
>>37292775
these are definitely saved too
>>
File: 1576633507497.gif (1.69 MB, 1200x689)
1.69 MB
1.69 MB GIF
So wait, do we have all the listed videos saved or what, because I'm goin crazy thinkin here
>>
>>37292814
they are only targetting unlisted ones i think
>>
>>37292814
There's a lot of videos in the spreadsheet that aren't sorted yet. You could still pick out the pony ones from there if you want.
>>
>>37292823
i would but i don't know how to
>>
>>37292823
>https://derpy.me/4RApE
should I just go on the spreadsheet copy it over then what do i do to the spreadsheet after
>>
>>37292828
No no don't copy
You can add the pony status to the spreadsheet online
>>
>>37291849
https://www.youtube.com/watch?v=XvtP3TsxKss
I know this one isn't pony related, unless you count joinedtheherd liking it, but I can't seem to get it downloaded on my computer.
>>
Just checking if I did this right. I had some time to grab several videos from the spreadsheet earlier tonight with youtube-dl [URL]. The .mp4s downloaded fine. Do I just mark the videos as green myself in the spreadsheet? Looks like YouTube hasn't taken anything down yet.
>>
Ok, I did this from the spreadsheet I downloaded and using the ids anon >>37292572 gave me
These are the 1611 ids left to download based on my copy of the spreadsheet
https://u.smutty.horse/mcftaetiqag.txt
It's only ids and maybe the bot already got some, who knows
But worth checking out
Ok, now you have that, I'll rest a little and post one with the full info of these ids
Btw, none of the category kinda pony have been downloaded (590) afaik
>>
I forgot to mention all of those are pony
>>
>>37292845
Anyway I got get a list of kinda pony? Granted, I am in the mist of a cascade failure, but I'll try
>>
>>37292833
ask and you shall receive
https://theponyarchive.com/archive/tantabus/quickchive/_-galawaille-20160925-youtube-1280x720-XvtP3TsxKss.mkv
>>
>>37292851
here you go
most are only kinda pony because of their length (so fucking long), but the rest only have a little bit of pony to them: https://u.smutty.horse/mcftbvhcuju.txt
>>
File: 1446602239523.jpg (11 KB, 256x256)
11 KB
11 KB JPG
So, what's the next step? Uploading all of them back to Youtube?
>>
File: 1501485400592.png (116 KB, 1000x1000)
116 KB
116 KB PNG
>>37292862
>>
>>37292862
That's gonna be quite the number of uploads for one channel
>>
Okay, more info:
Pony yet to be downloaded: 1611 / 24785
Criteria: id marked green based on >>37292572
Only ids: https://u.smutty.horse/mcftaetiqag.txt
Full info: https://u.smutty.horse/mcftbwxowbm.txt (Use only as reference! Some characters are borked and the sqlite separator is "|"!)

>>37292851
Kinda pony yet to be downloaded: 589 / 590
Criteria: saved = 0 (Where saved = 1)
Only ids: https://u.smutty.horse/mcftdangryf.txt
Full info: https://u.smutty.horse/mcftcwybmqc.txt (Use only as reference! Some characters are borked and the sqlite separator is "|"!)
>>
Database, tomorrow I'll post how I got it
https://u.smutty.horse/mcftdmyjtnq.db
Remember is static, no new changes in the next hours will be reflected here
>>
>>37292836
if you have them uploaded then mark them green anon
>>
>>37292862
>>37292867

no do it on tons of different channels thay way youtube will not know what to do
>>
>>37292877
Uploaded where? Or did you mean downloaded to my PC?

Also, I just realized the menus on the Google doc are faded out and I don't think I can actually change the color. Am I misunderstanding how this works or is the doc not actually public for all anons?
>>
>>37292894
I think we're only meant to edit the pony column
>>
>>37292895
Ah, I see it now. I thought there was some way to note which videos had already been grabbed so that time isn't wasted getting a video that's already been downloaded.
>>
>>37292899
As far as I'm aware they should be on track to downloading everything marked as Y in the next few hours. It's the unsorted stuff that's the problem.
>>
File: 1626556191497.gif (429 KB, 1000x625)
429 KB
429 KB GIF
I made a GIF out of one of the videos we saw in the stream.
>>
>>37292905
Yes you can, yes you can!
>>
>>37292901
everything that was sorted was saved as of >>37290293 today at least

>>37292905
fukken saved
>>
>>37292860
>https://u.smutty.horse/mcftbvhcuju.txt
Thanks a billion. They will be low quality downloads but it is better then nothing!

Provided my computer don't just straight up fail, been having an odd level of techical failiure on unrelated machines all at once
>>
>>37292899
It's meant for TPA specifically
>>
>>37292572
I can't stay awake any longer I'm gonna request these before I crash.
https://www.youtube.com/watch?v=oC7B0JXIHGY
https://www.youtube.com/watch?v=1U9hb_5JzwA
https://www.youtube.com/watch?v=WabH5TBy3mU
https://www.youtube.com/watch?v=wlrrGMFhfDU
https://www.youtube.com/watch?v=vihB1fohOjo
https://www.youtube.com/watch?v=R9prJHAjMYY
https://www.youtube.com/watch?v=uVNzxuJR16Q
https://www.youtube.com/watch?v=P62MJgamfdY

Thanks for all the work you guys have done whether it was a little or a lot, I certianly wish I could have contributed more, but I do feel pretty good about ensuring that a dramatic reading of my little lobster will survive.
>>
>>37292958
meant for
>>37291849
definitely time for bed
>>
>>37292958
still gottem in, night anon
>>
File: 1614419645176.jpg (161 KB, 800x800)
161 KB
161 KB JPG
Since we've got most everything backed up now that we can (and no longer have to worry about vengeful creators intentionally privating/deleting videos once being reminded they exist) it couldn't hurt to start emailing/messaging the channel owners to bring the Youtube thing to their attention/ask them to opt out of the change.
Who knows, maybe even accounts that haven't uploaded anything for years still have their emails being read/notifications being checked.
>>
>>37292397
The problem is I’ve found some pretty good pony videos in there that have very generic titles
>>
https://www.youtube.com/watch?v=MRKgKZ8_U7E
Just noticed this one is unlisted too, lol.
>>
File: 1623516154278.png (162 KB, 600x600)
162 KB
162 KB PNG
I just wanted to say that you fags are awesome. I feel proud knowing I share the board with anons like (You). All of this is incredible work. Just amazing.
>>
File: 35592f0efd[1].png (16 KB, 1029x148)
16 KB
16 KB PNG
Saw this on an old video.
Seemed fitting for the occasion.
>>
>>37291849
Never even heard of Alpheus Soar until today
https://www.youtube.com/watch?v=eg6mno6eSAQ
https://www.youtube.com/watch?v=ynzNXUIkixI
https://www.youtube.com/watch?v=aptXP7YOQiE
https://www.youtube.com/watch?v=F_iK3dmnJXU
https://www.youtube.com/watch?v=drz0wad97q8
https://www.youtube.com/watch?v=DVV_e5omW1k
https://www.youtube.com/watch?v=sHT_uJyOrpk
https://www.youtube.com/watch?v=GbXHAWaRheA
www.youtube.com/watch?v=lhNkZGqh2d4
https://www.youtube.com/watch?v=DOppAIAuL_8
https://www.youtube.com/watch?v=QwUNcSZw1c0
https://www.youtube.com/watch?v=CUwUeOwbkBo
https://www.youtube.com/watch?v=VySwtXT8xHg
https://www.youtube.com/watch?v=hNWKuayEjEI
https://www.youtube.com/watch?v=tN1x5pSDsgo
>>
>Bump limit hit
Holy shit, I didn't even notice.
Does a new thread make sense at this point?
>>
File: 61108.png (735 KB, 945x945)
735 KB
735 KB PNG
>>37292872
I'm not going to be able to get'em all. Heck, I will be lucky to even get a hundred with the way my internet is going right now. I'll see if I can get the stuff that is actually import to our history over the extended long videos.

>>37292664
I understand were you are coming from, but this >>37293102 Over the past two years I have seen a spark of life that I have not seen elsewhere. From the PPP to the alternate boorus, from the archiving effort here to the pony videos that are still being put out and getting quite good views in spite of being against the algorithm on YT: https://www.youtube.com/watch?v=avJwsG6MahE Or heck, our largest thread being in 2020 I am not saying things are perfect and all things going are positive, but I am far more opportunistic of the future then I ever was before.

God bless y'all, I wish you the best. Hopefully we will have time to get the rest. I will get to work on my final run.
>>
>>37293152
saved now

>>37293158
I dont think so, I bet you they'll go dark before the thread dies.. which is really sobering to think about
>>
>>37293158
Yes, to discuss where to reup.
>>
>>37293158
Yes. I'll need to share the videos and metadata I've downloaded.
>>
>>37293159
>our largest thread being in 2020
Which thread was that?
>>
File: 1603005559596.png (259 KB, 563x771)
259 KB
259 KB PNG
>>37293193
>>
>>37292872
i dont know what is already archived by someone else but i have about 400 of the pony ones, should i continue that list or do the Kinda pony first?
>>
>>37293211
Focus on pone but do some of the kinda pone list if you see anything worth it here
>https://u.smutty.horse/mcftbwxowbm.txt
>>37292872
A lot of them are BIG but just extented stuff that wouldn't be worth it.
>>
>>37293243
Ok, thanks.
Where is everyone uploading the stuff they saved?, im starting to worry about drive space
>>
>>37291894
>>37292684
>>37292722
>>37293159
We tried and did our very best, and that's all anyone or anypony could have asked.

>I am not saying things are perfect and all things going are positive, but I am far more opportunistic of the future then I ever was before.
Same here, and wouldn't want it any other way.
>>
>>37293152
>>37291849
I think there's a few more in here
https://www.youtube.com/playlist?list=PLudBqUUc38WCh-z7Q1sVep4-bD9LT2x8b
>>
Yo /mlp/ just want to say stellar work on the efforts to archive all these unlisted videos and especially the scripts you guys made. I stayed up last night archiving ~80 videos from my YouTube favorites after using your script to search playlists. I don't watch mlp but glad you guys were putting in effort to do this stuff.
>>
File: 354962.png (697 KB, 1028x1064)
697 KB
697 KB PNG
>>37293569
ur welcome my mare
>>
Videos are still around, but as mentioned previously it may kick in around 9am pst, which is just under an hour from now
If there are any last minute submissions, now might be the time.
>>
>>37285318
Well this is interesting. The comments section have now been disabled. It wasn't like this several hours ago.
>>
>>37293079
This is a gem, one of favourite videos.
>>
>>37293158
Yes at least one more thread is necessary. A lot of questions about where to find videos, share metadata etc. Don't want this to be another "yeah it's archived somewhere but I can't find it" episode.
>>
>>37293793
There's still much possibilities in what to do in the means of archiving. Now that we had organized and got familiar with our tools we could go further discovering, classifying and archiving other stuff while it's not in immediate danger. Probably we could make it semi-regular thing like there were MLP Music Archive torrents.
>>
>>37293917
Still got all those post 2017 unlisted videos to sort too right?
>>
>>37293934
That's right. If find_unlisted* script anon still here it would be a good idea to update repository to add option to not filter upload date anymore. Also, after nukes finally drop it'll be interesting to recheck youtube against archive database to find how many channels actually opted-out from change. Yeah, and it'll probably be good to rerun this check on monthly basis to see if authors unprivate or opt-out late after all dust settled.
>>
i did my part
>>
I'm stuck inside with a bad ulcer watching all of these vids. Bringing me back to 2011 when the net and TV were still both worth tuning into.
>>
One final thread for the day of reckoning?
>>
>>37291849
Requesting these if they haven't been saved already.
https://www.youtube.com/watch?v=HlE0qm7fpOI
https://www.youtube.com/watch?v=J-LmptgG1GQ
https://www.youtube.com/watch?v=SBoCM1Hsuro
https://www.youtube.com/watch?v=Vk4Til_jfVM
https://www.youtube.com/watch?v=P_GAly7xGfE
https://www.youtube.com/watch?v=mDe1eVui4A4
https://www.youtube.com/watch?v=31pDXI8xyE0
https://www.youtube.com/watch?v=GXQwLJzwZGM
https://www.youtube.com/watch?v=mkyt5vQ4NA0
https://www.youtube.com/watch?v=D2TmXN2CV0w
https://www.youtube.com/watch?v=tkwgVQZ-HCI
https://www.youtube.com/watch?v=RvRRKoPMeUA
https://www.youtube.com/watch?v=0J4T13PkntQ
https://www.youtube.com/watch?v=ZXkpH_Mc9L8
https://www.youtube.com/watch?v=H8TekIbdy0Y
https://www.youtube.com/watch?v=8B7b2HIeu5k
https://www.youtube.com/watch?v=EQD0GMRAEaI
https://www.youtube.com/watch?v=byjF_DbEN5M
>>37293159
>I'm not going to be able to get'em all.
I feel pretty confident we got the cream of the crop, we're probably missing a few hidden gems but I don't feel too bad about losing the fanfic readings and random noise videos.

>>37292862
I noticed https://www.youtube.com/watch?v=Zh5NVsLuWyM got reuploaded along with https://www.youtube.com/watch?v=kIaM7q4OrIw by the same person, you can search for the video titles to find them.

>>37293793
I kinda like the idea of reuploading them to youtube, at least we could let some of the existing archive channels know about our efforts.
>>
>>37294243
It'd have to be the more popular ones that would get uploaded. Reuploading over 20k videos isn't too feasible. That userscript that was shared earlier to redirect the unavailable page to a video archive like hobune was pretty cool. You'd also get to display all the metadata and comments that were saved.
>>
Well fuck. I've been running youtube-dlp for several hours but only now decided to check logs. Apparently some videos throws
>[download] Skipping fragment N
How fucked am I? Will it end in corrupted video files or it is normal operation?
>>
>>37294243
Ok so these currently aren't in:
HlE0qm7fpOI
J-LmptgG1GQ
SBoCM1Hsuro
Vk4Til_jfVM
P_GAly7xGfE
ZXkpH_Mc9L8
H8TekIbdy0Y
byjF_DbEN5M
I'm going to get them saved real quick.
>>
File: 1624760568205.png (461 KB, 773x842)
461 KB
461 KB PNG
>>37294263
its past the deadline and we are still finding videos
>>
>>37294257
If it skips segments it may be working on a video playlist
To have videos load, some sites split the video in pieces then produce a list (playlist) with all those fragments. As the video progresses, the player reads this list and plays the segments.
Check the video manually, but imo I don't think the files are corrupt.
youtube-dl may be reading the same playlist and skipping a (possibly) empty fragment or skipping on some criteria I don't know
>>
>>37285318
>https://www.youtube.com/watch?v=OB03ufqDQsE
>ERROR: This video contains content from Hasbro Studios, LLC, who has blocked it on copyright grounds.
>>
>>37294275
Well the next deadline I can think of is within the hour (12pm PST).
>>
>>37294288
Yeah it has been that way for years. Unfortunate that the set will remain incomplete.
>>
File: 1598056663274.png (79 KB, 500x500)
79 KB
79 KB PNG
Suggest next OP pic and text.
>>
>>37294301
Is there a way to see in which regions exactly video was blocked? Maybe it could be bypassed by using right VPN?
>>
>>37294318
I think there is a website that checks that but I can't remember what it is.
>>
>>37294318
https://polsy.org.uk/stuff/ytrestrict.cgi
>>
>>37294327
>No Antarctica
Hm...
>>
>>37294276
Thanks! I tried to check some videos with that message but I don't have anything to compare with. Probably should try to redownload some of them and see the difference.
>>
>>37292872
I have 1451/1611 pony videos the rest failed or no longer exist
>>
>>37294327
Can you please check in which regions this video is banned?
https://www.youtube.com/watch?v=qDC_jGYN3xk
>>
>>37294336
Well gentlemen looks like we have to found the Brony's Republic of Antarica.
>>
>>37294353
The website's right there dude, why don't you do it? It's blocked worldwide.
>>
>>37294361
>The website's right there dude, why don't you do it?
I am retarded
>>
>>37294318
I use another site: https://watannetwork.com/tools/blocked/

You can use it as a boomarklet too: javascript:void(open("https://watannetwork.com/tools/blocked/#url="+new URLSearchParams(new URL(location.href).search).get('v')))

Make that as a bookmark on the bookmarks bar. Clicking it on a youtube video page will open up the region checker in a separate tab.

Unlike >>37294327, this site doesn't have an extra button to prevent bots, so the site occasionally won't return anything due to its quota running out. Also Somaliland, Kosovo, N. Cyprus are just...there. They'll always show as unblocked, but practically it's blocked everywhere.
>>
Okay, here's how I made the database >>37292876
First download the table from https://derpy.me/FkhNl as .ods
Next open it in your spreadsheet software, I used LibreOffice
Then click "Save as" and there choose .csv (Comma Separated Values)
Check the options before saving for the "Delimiter" box. Make sure it is ";" (without quotes)
Also check the box "Quote all values"
If you did this right and open the new csv file it will look like this:
>"https://www.youtube.com/watch?v=IpdrNz2855I";"IpdrNz2855I";"Y";"[SFM Ponies] [PMV] Feeling This";"2:30";"kennyoung";"UC7dViO9ogcMViU7hYAxKjbw";"2013-11-16";"82572";"{}";
Now, download these two scripts:
Rename to "todb.py": https://u.smutty.horse/mcfybpxxapz.py
Rename to "updb.py": https://u.smutty.horse/mcfybsvkddl.py
It is important that you rename them for one creates the database and the other updates it
From here on out, I'll assume you use gnu/linux
Open a terminal on the folder where you have all these files
Step 1
touch be.db
Step 2
echo 'CREATE TABLE pony_videos(id TEXT NOT NULL, title TEXT NOT NULL, approx_length TEXT, channel TEXT, published_on TEXT, views TEXT, adult_content TEXT, livestream TEXT, saved TEXT, channel_id TEXT, UNIQUE(id) ON CONFLICT IGNORE);' | sqlite3 be.db
echo 'CREATE TABLE kinda_pony_videos(id TEXT NOT NULL, title TEXT NOT NULL, approx_length TEXT, channel TEXT, published_on TEXT, views TEXT, adult_content TEXT, livestream TEXT, saved TEXT, channel_id TEXT, UNIQUE(id) ON CONFLICT IGNORE);' | sqlite3 be.db
Step 3
Open todb.py with any text editor and change on line 11 the name of the file to open
>with open("change_this_filename.csv", "r") as csv:
Step 4
Open your csv file and delete the first line, the one that reads "Id,"Channel"... it's useless now
Step 5
Run
chmod 755 todb.py
chmod 755 updb.py
./todb.py
That should populate the database and give you an extra text file where the non-categorized videos will appear

To update the database
Download: https://u.smutty.horse/mcfrzhmytzt.txt
Don't rename it
Run
./updb.py

I'll see if I can write an improved version that doesn't require so much meddling with the files later, but for now this will have to do.
>>
>>37294344
If its possible, pass me a list of ids or the filenames
If its the latter:
On a terminal: ls > lof.txt
>>
>>37294336
I mean Antarctica has an IP range:
https://lite.ip2location.com/antarctica-ip-address-ranges
The question is when will bighckintosh break into the computer of some researcher on antarctica to download pony videos.
>>
>>37294424
Sorry for the delay, was having trouble getting the list from my ftp, i also have the descriptions and thumbnails.
https://u.smutty.horse/mcfygeeasqi.txt
>>
Okay two minutes before the new hour. Going over the new videos in the sheet to scrape for extra api stuff. Hopefully it finishes right on the hour.
>>
File: 6y65825689.png (11 KB, 117x104)
11 KB
11 KB PNG
>>
Guys. . . . ?
>>
>>37294559
Oh good. Looks like another hour possibly...?
Okay then I'll grab the comments too.
>>
we are now on borrowed time
>>
>>37292872
>>37294344
I ended up downloading the full list of 1611 pone last night. Ended up being around 100gb, so it should fit into a handful of Megas.



Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.