[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / r / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip / qa] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
Settings Mobile Home
/g/ - Technology

[Advertise on 4chan]


Thread archived.
You cannot reply anymore.


[Advertise on 4chan]


File: 1313.jpg (957 KB, 2400x2400)
957 KB
957 KB JPG
>Links
Rentry: https://rentry.org/dhg

>What is /dhg/
In this thread we discuss and create technology and software for data-hoarding, archiving, scripts, and more.

>gallery-dl - scrape images, manga, videos and more from many websites
https://github.com/mikf/gallery-dl

>Hydrus Network
https://hydrusnetwork.github.io/hydrus/

>Stash
https://github.com/stashapp/stash

>SmartImage
https://github.com/Decimation/SmartImage
>>
>>85222041
based OP
>>
>>85222041
Based
Thanks OP.
Should amend Danbooru downloader to the list as well.
>>
You didn't trigger the auto delete so this should be good
>>
File: 1vzy93.png.jpg (114 KB, 1200x859)
114 KB
114 KB JPG
>>85222041
Pepperoni...
>>
>>85222041
i'd eat that
>>
File: file.png (494 KB, 1030x888)
494 KB
494 KB PNG
WOAH
>>
>>85222137
this one?
https://github.com/Nandaka/DanbooruDownloader
>>
>>85222181
That's the one

Anyone utilize anything with Archive Team?:
https://wiki.archiveteam.org/
>>
File: 1540085108629.jpg (1.28 MB, 1440x1080)
1.28 MB
1.28 MB JPG
>>85222163
>>70239223
insane we just went back in time
>>
>>85222198
alright, added to the rentry
>>
I know that cumg posters are here, you are not fooling anyone.
>>
>>85222322
the cum- what?
>>
>>85222322
We don't know what you're talking about.
>>
this thread is sus
>>
>>85222322
This general or the people in it does not have any involvement with CUMG (Children's University Medical Group), we do not know why you're saying this.
>>
>>85222041
>no lolisnatcher
>>
>>85222363
HDDs have gone up in price dramatically in the past year, but the 8TB Seagate Barracuda is somewhat reasonable
If you're your only client, it should be plenty
>>
>>85222322
who? we're excited for this new general which focuses on data hoarding and software to tag it, save it, and view it.
>>
File: 1502077471321.gif (985 KB, 500x211)
985 KB
985 KB GIF
>>85222322
This thread is about the preservation, archival and storage of media
>>
>>85222453
I'm happy that CHIA didn't take off to the point.
Now you can get a 12tb drive for under 300 bucks. Shit is comfy man.
>>
File: 1639240519537.png (95 KB, 240x240)
95 KB
95 KB PNG
>>85222322
>cant stop thinking about cum
>>
>>85222322
imagine posting on 4channel and all you see is cum(g)
>>
>>85222363
Yes. Only if you have an RAID as well as an FS that can checksum and do bit rot checking like ZFS.
Tape is still expensive and can only be accessed in a linear fashion.
>>
>>85222322
>uh-uh-t-take meds schizoid!
>t. seething coomer
>>
>>85222536
you election tourist puritans are a funny bunch
>>
File: 82670979_p0.png (1.83 MB, 1000x1050)
1.83 MB
1.83 MB PNG
It's nice to see this general back from the dead
>>
I don't care about porn.
How do you organize your content?
I have everything hashed into a SQLite DB. I burn all files on BD-Rs (uncompressed if possible) and each file has the disc number associated in the Files table.
>>
>>85222574
>election tourist
>a whole year after Joseph Biden was lawfully and fairly elected by the American people
brainlet. being against cooming has nothing to do with being a 'puritan' or 'moralfag'. It's just called being reasonable.
>>
>>85222627
don’t care
we’re picking up where we left and you may cry yourself to sleep now
>>
>>85222618
Files Folders and numbers for referencing
All stored on drive array and a 12-bay synology NAS
It needs to be cleaned out and reorganized.
>>
>>85222618
I throw everything in a secondary PC and I use a HTTP server (that I coded myself so I can make sure it doesn't glow and just werks) to consume the content
>>
Hot pockets are actually pretty good
>>
>>85222704
Preserving tape becomes an issue since they're not particularly resilient
>>
>>85222041
Janny edition
>>
File: violent sex.png (479 KB, 811x393)
479 KB
479 KB PNG
Why would you want to hoard data????
>>
File: m8hrd.png (13 KB, 307x317)
13 KB
13 KB PNG
>>85222425
Spread your legs, miss. Trust me, I'm a doctor.

M8HRD
>>
i have a question here
which system actually good for hoarding stuff ?
and should i use a server even as VM ?
>>
MOOOOOODS
>>
File: 1637241295374.png (54 KB, 708x800)
54 KB
54 KB PNG
>MOOOOOODS
>>
File: cumg_is_kill_cope.jpg (119 KB, 601x508)
119 KB
119 KB JPG
Hello fellow data hoarders.
I want a local mirror of my distribution's repository (Ubuntu), any tips on how to do it?
>>
>everyone is already dumping porn images
I swear, coombrains have no self-control. You can't go a single second without having your plaque-filled tiny dicks erect at all times.
>>
>>85222041
thanks for resurrecting old general
>>
File: 1624326000112.jpg (377 KB, 1920x1080)
377 KB
377 KB JPG
>>85222874
No guarantee anything is going to stay on the internet forever
>>
>>85222041
If I have raid 0 SSDs do I need to plug them both in to recover data or can I recover the data on the drives just by plugging them in individually.
>>
>>85222041
https://github.com/NO-ob/LoliSnatcher_Droid
>>
>>85223060
Pic related? kek
>>
>>85223087
The admin of kiwifarms did a livestream like half an hour after the ChCh shooting where he was spreading the torrent around
That shit got wiped from the internet and I think I'm the only one that still has a copy
I found it looking in my videos folder for something to post
>>
>>85223131
you can share it using ipfs ?
>>
File: 164202266445.jpg (132 KB, 1200x1200)
132 KB
132 KB JPG
>>85223036
What you're talking about anon? I don't see any images that would hurt the advertisers in this thread.
>>
>>85223174
I'll make a throwaway mega
gimme a sec
>>
Based coomchads
>>
hello mods this is the new /cumg/, just letting y'all know
>>
>>85222322
Who's that? Some youtuber?
>>
>>85223193
>mega
ok i guess
>>
File: 164160616880.jpg (335 KB, 1280x1707)
335 KB
335 KB JPG
>>85223249
No, this is /dhg/. Nothing wrong with data hoarding now is it?
>>
we need a new report category for embedded data...
>>
>>85223310
It's absurdly easy to just block this shit or clean it from the image, they didn't do this yet because they are incompetent or maybe the programmers just aren't paid enough.
>>
>>85223332
they did block it, their devs had to do work for once and they were not pleased
>>
File: 1338932679718.jpg (81 KB, 800x400)
81 KB
81 KB JPG
Can't believe there are people unironically seething over script or two this hard when they can just not use them.
>>
File: g.jpg (154 KB, 1046x1048)
154 KB
154 KB JPG
Nice try but you're not fooling anyone. To make this truly a /g/ you're going to have to be more clever.
>>
>>85223393
>just don't use them
>but we'll keep raiding other blue boards with offtopic porn dump threads, you have no reason to complain :)
literally /qa/ 2.0
>>
>>85223289
here you go, don't forget to like and subscribe
https://mega.nz/file/RdxQQDwT#Eu349zmbXxbtgC16JCxZlfqnqUocRX2bbh_z2u0qNpc
>>
can an booru client actually tag pictures ?
something like a booru server where it tags pictures automatically ?
>>
>>85223440
that sounds like a /v/ issue
>>
>>85223249
>ya'll
>>
>>85223445
thanks
>>
>>85223445
Bless you anon
>>
>>85223131
what shooting?
>>
>>85223567
>american moment
>>
>>85223730
i'm not american
>>
lynx -listonly -nonumbers -dump "$1" | grep 4cdn | sort -u | xargs yt-dlp

Is there a way to get a curl equivalent of this? Arg 1 is just a url, mostly used for threads on wsg. This just downloads all the images for a given thread. Seems like curl would be able to do this except I use sort -u to filter out duplicates. Idk if curl can do that.
>>
>>85223440
What if someone made an userscript that replaced every file posted on 4chan with random porn images with no input from the poster, would you consider that a raid on your so-called innocence and prudity too?
>>
easy way to organise doujins ?
should i use lanrargi ?
and how do you tag doujins right ?
>>
File: 164157214231.jpg (1.07 MB, 1000x1414)
1.07 MB
1.07 MB JPG
>>85223462
Not just scraping tags, but tagging the images by itself?
>>
>>85224068
yes something like hydrus but simpler and MORE EFFICIENT, doesnt require mass import
>>
>>85224008
>should i use lanrargi
Yes.
>and how do you tag doujins right
Lanraragi has an autotagging tool.
>>
I truly mean it wholeheartedly and without irony when I say that everybody in this thread--no, this entire website is a fucking faggot.
>>
One mechanism I have of organizing is using TOSEC and RomVault
Roms go in, they get checked and organized, compressed and autosorted.
Comfy as hell.
>>
>>85224242
thanks for coming out anon :)
>>
>>85223032
This is a good question, bump.
>>
>>85223396
assuming the assumption?
>>85223440
>literally
you are literally blaming us for existence of tourists
>>
Can you recommend good web scraping tutorials.
Usually I'm just downloading a page, selecting what I want with structural regex and using it as input for curl or wget.
I want to be more elaborated than that.
If it's on python it would be better since it's a good complement to learning it. Common Lisp is fine too, because it's a language I ways wanted to learn but never had a reason to.
>>
>>85223032
rsync another mirror or the originating mirror.
https://help.ubuntu.com/community/Rsyncmirror
>>
>>85224738
https://www.crummy.com/software/BeautifulSoup/bs4/doc/
>>
>>85222041
Is gallery-dl better than RipMe?
>>
>>85224919
I heard about BeatifulSoup before but I forgot about this name because of how weird it sounds. I'll check it out, thanks.
>>
>>85224934
Check this.
https://github.com/Bionus/imgbrd-grabber
>>
does anyone know of a good method to categorize / check / deduplicate retro game roms?
due to nintendos retardation ive long wanted to keep a local archive, although i dont necessarily want to limit it to nintendo roms

also, does anyone backup gog installers? do you use lgogdownloader, and if so, what is your archive structure and how is it treating you?
>>
>>85225317
>retro game roms
I don't know, I just downloaded this.
https://archive.org/details/no-intro_romsets

It is already well organized.
>>
>>85222041
So this is what it took to get the general renamed? Interesting...
>>
Not sure if this is the right thread to ask.
But dose anyone know if mega does promos on their subscriptions?
>>
Don't know where else to put this but uh...
It's still christmas on >>>/qa/
>>
>>85226709
>>>/trash/44690583
>>
>>85226731
Nice.
>>
What are people's formats for naming folders?

Do you use:
dots: Movie.stills/
hyphens: Movie-Stills/
underscores: Movie_Stills
>>
How do you guys archive youtube comments? I've got the json scraper working from yt-dlp, but viewing it is a bitch.
>>
>>85227247
Underscores, dots and hyphens can mess up your code and scripts if you want to automate something later.
>>
>>85227247
Movie Stills
>>
>>85227338
How so?
>>
File: 164181529792.jpg (95 KB, 600x845)
95 KB
95 KB JPG
any alternatives to hydrus? i checked localbooru, seemed like an abandonware
>>
>>85222692
They're nasty, processed garbage. Even the smell of them is off-putting.
>>
>>85227474
Are you implying that hydrus is spyware or something with your pic?
>>
>>85228143
no? what gave you the idea
>>
File: b0r2vv.webm.png (6 KB, 333x142)
6 KB
6 KB PNG
>>85227474

anyone?
>>
Fuck MEGA, i think they decreased their free download limit.
What do you guys use to bypass this shit? Or should i just use proxies?
>>
>>85229632
depends on what are you trying to do
pcloud is an alternative
>>
>>85229659
>depends on what are you trying to do
I want to bypass the free download limit, not find an alternative site to upload.
>>
>>85229728
unironically autistic? i was asking what are you trying to do with the mega
now I don't care though
>I want to bypass the free download limit
retards are meant to pay
pay, retard
>>
>>85229754
Wow, this general is really useless, no wonder the mods banned the last one.
>>
>>85229787
pretty sure last /dhg/ thread was like 3 years ago and it wasn't banned
>>
Anyone know how to rip the ATRAC files from a minidisc? Some enterprising Jap has to have pulled it off, but good luck crossing the japanese internet barrier to find that how-to written in english

I know about MZ-RH1's ability to backup tracks, but I dont think it can do that for anything copied from a digital source - which is like 90% of recordings.
>>
File: ecgq6l.ogg.jpg (72 KB, 907x778)
72 KB
72 KB JPG
>>85222322
>>
>>85222041
hey guys, i have favorite stuff on pixiv, sankakucomplex, deviantart, e621, etc. I was curious about a way to serve my own booru that automatically downloads my favorites and also my favorite artists.
Oh wait, this isn't the coomer tech thread? Sorry guys
>>
>>85229214
I want to pet that chen
and then I want to pet that ran

>>85230228
gallery-dl in the OP?
>>
>>85230367
Yes but I would like to run a local booru (a web site) to browse it all
>>
>>85230367
>>85230588
it would be great to have a single place to view my favorites, especially if it fetches all the tags that are available (sites like gelbooru, sankaku complex, e621, etc. all tend to have detailed tags on art)
>>
>>85230597
Danbooru Downloader is the closest.
Otherwise you'll need your own website and you're going to run into XSS issues
>>
File: 1616732502764.png (7 KB, 273x60)
7 KB
7 KB PNG
How does this work? I am using my PC and want to clone my "Main Drive" it has my OS on it but I'm using it literally and creating an image backup.

I don't know how that works when I'm literally using it. How can it clone something that I'm using and is active?
>>
>>85230597
Hydrus?
>>
>>85224987
>>85224919
selectolax is better
>>
>>85222198
>Anyone utilize anything with Archive Team?:
yes
>>
>>85222673
>a HTTP server (that I coded myself so I can make sure it doesn't glow and just werks)
bros just imagine the amount of vulnerabilities this has
>>
>>85230702
Macrium reflect can do full clone for free from a bootable usb
>>
>>85230702
anyone know the answer as to how this works?
>>
>>85230958
Files don't have a fundamental "it's being used" property. It's just copying 1s and 0s.
>>
>>/trash/ doesn't state that GR 17 applies, maybe you should post where you belong coomtrannies
>>
>>85226709
Weeeeeeeeeeeeeeeee
>>
>>85227474
write your own
>>
>>85222041
I leave for a few days and you faggot retards have killed /cumg/ with your retarded fucking embed scripts fuck you. NIGGERS
>>
Hello /hoarders/. Don't mass download from danbooru, got me permanently b&.
>>
>>85232239
the fuck
how
>>
>>85232387
>be me
>want to update my metadata dump
>download with 8 threads in 200 post batches, json
>start getting empty results
>update script to just try again again after a short period of waiting (10s + random(10s))
>doesn't work
>cannot even connect anymore
>see this (pic related)
>ok no panic, restart router, delete cookies
>different IP
>same error
>they probably blocked the whole country, sorry fellow cunnysseurs
>one month later
>still b&
>>
>>85232442
Did it ever occur to you that rate limits exist for a good reason you fucking goober?
>>
>>85232217
Completely new to cumg, what are those embed scripts about? I really don't get what they are talking about
>>
File: 1642250868.png (53 KB, 968x265)
53 KB
53 KB PNG
>>85232461
I'm ESL and thought pic rel, from the API documentation, meant "no rate limit for reads".
>>
>>85232442
What country, if you don't mind me asking?
>>
>>85222041
I'm working on an image database, much like hydrus but as a webapp.

https://gitlab.theswissbay.ch/theswissbay/4id

Features so far:
- images, folders, tags
TBD:
- search, multi-file upload, moderation, duplicate detection, edit history, linking related images/folders/threads, search on SEs, OCR, etc
>>
>>85232500
That's exactly what I'm reading...
>>
>>85232442
>8 threads
anon come the fuck on
>>
>>85232500
yeah, but what you did falls under ddos, not API reads
also, if youre still banned after a month id guess its permanent
>>
File: 1620055650435.png (64 KB, 420x420)
64 KB
64 KB PNG
>>85232492
answering it will make jannies very upset
if only there were some websites that archive previous threads that had clues in them hmmmm
>>
>>85232551
not him but is that a lot?
>>
>>85232527
Don't want to get lynched, small country, few ip ranges.

>>85232555
The dreaded single-client d-dos who does not even have gigabit. But yes, Cloudflare definitely thinks so as well.

>>85232567
I know, still good metadata.
>>
>>85232614
Do you even know how ddos works?
>>
>>85232614
>Don't want to get lynched, small country, few ip ranges.
In that case it might be possible since you are probably one of the few people that use danbooru from there, you can always use a VPN if you still need to access danbooru.
>>85232596
I assume that means that he saves 8 images concurrently and yeah some servers might see that as abuse. I currently do the same thing for another website (8 threads) but I always use a VPN so my main IP won't get banned if they find out, luckily they don't seem to care.
>>
>>85232638
For the torrent: It's outdated. As for their database: I tried, but could not figure out how to download from there.

>>85232647
Yes

>>85232661
>8 images
I just downloaded json, no images.

>VPN
I guess I could, but for now I will just do without danbooru.
>>
>>85232596
i'd say making more than 2 big (>10KB) requests per second to a free website that tries to push paid API usage is indeed irresponsible
>>
Thread is kill. What do you hoard and why. How much do you have? Which drives, case, connections?
>>
>>85233098
cumg has cummed its last cum
>>
>>85233120
So you're saying it has ED?
>>
>>85233120
Had it cumming, advertisers hate certain letter groups and Hiro got to eat.
>>
File: nas-case.jpg (45 KB, 700x303)
45 KB
45 KB JPG
>>85233098
a truenas instance with 12x8tb drives, 5 mirrored pairs and two hot spares. case is picrel and im using a reflashed thinkraid 530-8i
as for what to hoard, not too much yet, primarily using it for backups currently.
>>
>>85232239
Hydrus doesn't have this problem :-)
>>
File: 1641879280004.png (1.29 MB, 900x650)
1.29 MB
1.29 MB PNG
>>85233098
>thread is kill
that's because most cumg posters were rangebanned in janny rampage trying to create the thread
many such cases!
>>
>>85233660
Madness.

>>85233569
:^)
>>
my strat is hdds on nas, backing up to external hdds every 2 weeks, and backing up to another set of external hdds every couple months. will we ever go beyond hdds? It feels like the SSD industry sees that 1tb is affordable for most people and has called it "good enough".
>>
>>85232194
you might as well say 'suck your own cock'
>>
>>85227338
So does spaces. In fact, underscores, dots and hyphens are a thing BECAUSE of programs and OS's not handling spaces in filenames. Nowadays, it's not an issue except your programs have to account for all the different word seperators.

>>85230702
>>85230958
Shadow copy and tracking changes.
>>
>>85234295
Nobody does it better
Makes me feel sad for the rest
Nobody does it half as good as you
Baby, you're the best
>>
>>85222627
"cooming" is a discord tranny meme
fuck off.
>>
File: 1614572193491.jpg (24 KB, 400x400)
24 KB
24 KB JPG
>>85235138
>>85235138
>Nobody does it better
>>
okay back
>>
>>85236302
kys coomdev
>>
>>85236329
I'm not him.
>>
>still not a single utilty like hydrus without importing
sad life
>>
File: 1636086318715.png (25 KB, 270x252)
25 KB
25 KB PNG
I tried hoarding images from boorus in the past, but I got bored trying to organize them. How do you have the strength to do so?
>>
>>85236601
hydrus
>>
>>85222041
based, someone actually did it! is there any benefit to using WD red hard drives if all you do it throw them in an external 3.5" enclosure?
>>
>>85233098
>what you hoard
music scores, older music, old/vaporware gaymes I've played
>and why
at risk of being impossible to find in the future. I'm not gonna "own nothing and be happy"
>how much you have
not much, only a couple hundred GB right now. still building my setup
>>
>>85237122
how many albums you got
>>
>>85237165
probably 100 or so, mostly 70s jazz stuff
>>
>>85222493
yummy yummy cummy all in my tummy
>>
>>85229632
>What do you guys use to bypass this shit?
IPv6
>>
I keep porn/doujins,movies/tv shows, music, software,game rips, photos, various document formats, and client backups on my server. Nothing fancy. When in the future all that's left is the low res crap (or if it even is still around) I'll be glad my copy is not shit in the case of my porn/doujins and movie,music rips (Freenas + backups/ups). I take the long view - I want it all to be there intact when I'm 90+.. Hey way medical science is going you could be 90+ and still pop a boner,etc you know..
>>
>>85222336
>>85223174
>>85223462
>>85224738
>>85224987
>>85236456
Good thread
>>
>>85237352
>90+
I'd rather be dead than have some uninterested useless nurse wipe my ass
>>
https://github.com/qarmin/czkawka

good tool to get rid of duplicates
>>
>>85237352
Make sure whatever you're storing is not in a lossy, compressed format... Have you heard of rotational velocidensity?
>>
>>85237404
I convert all my files to 64 kbps wma
>>
only way to auto tag is hydrus ?
>>
>>85237390
Joke's on you. I'm literally into that shit!
>>
>>85237391
Neat, this seems promising
>>
is nas setup worth it or can I get away with a bunch of hard drives and a couple enclosures?
>>
>>85237391
Thanks anon
>>
>>85232239
>>85232442
Danbooru is garbage anyway, it was for the best.
>>
Is there something like gallery-dl to download Telegram channels?
>>
>>85237511
Can Hydrus actually tag on it's own via some partly functioning recognition methods or do you "auto tag" in it via importing and saying the software "This batch of imports is X tag"?
>>
>>85237511
>>85238995
If you want hydrus to autotag, you have to download this massive public tag database which hydrus will match your image to. Otherwise, there is some addon that generates tags for you for hydrus, but I never looked into it.

For manual batch tagging, there are a few schemes hydrus can do, like tagging via folder/file name and stuff.


>>85238006
NAS is worth it if you need some small, low power form factor. Otherwise, nothing wrong with PC's.
>>
>>85237511
does importing tags from booru count as autotagging? if not then you are doing something wrong. ""ai"" was a mistake.
>>
>>85239473
*boorus
>>
>>85223131
are you retarded?

Null was pretty adamant about keeping it up
>>
>>85223887
just make it xargs -n1 curl -O
>>
>>85238995
>>85237511
https://github.com/Zweibach/text/blob/master/Hydrus/PTR.md
>>
>>85237511
h-hot
>>
File: 94156712_p0.jpg (1.1 MB, 1447x1976)
1.1 MB
1.1 MB JPG
>>85224738
Reasking this question, because sometimes the data I want to hoard is hidden behind Javascript. For example, take this page: https://www3.nhk.or.jp/nhkworld/en/tv/traincruise/
I want to set up a cron to scrape this site and download the episodes of this program every so often. I can do it manually, since youtube-dl supports NHK VODs. However, if I try to do it with Python, the HTML page I receive is different from the HTML page I can see with Inspect Element in my browser. So somehow the site is seeing that I am trying to download from my Raspberry Pi and denies me the links to download from.
It's not my useragent, either. This particular site blocks me if I try to connect without a useragent. If I send a useragent, it gives me the gimped HTML document I talked about earlier.
I guess my question is, is there any way to run the Javascript on my Pi in a lightweight and automated manner?
>>
>>85240007
The his youtube livestream not the shooting video retard
>>
>>85242144
Requests -html or Selenium
>>
>>85242144
If you're doing this correctly, it should be literally impossible for the website to know what you're really using to send a request. Take a look at how your browser is sending it's request (through the Developer Tools) and try to replicate it.
>>
>>85237391
Thanks. Looks cool.
>>
File: 164151701873.jpg (19 KB, 551x550)
19 KB
19 KB JPG
>>85237391
>rust
>>
>>85243230
Do you have a point?
>>
>>85243230
politics aside Rust does more things right than almost any other language
>>
So the steganography part and the data hoarding part got split after all. The former seems to have moved to /dev/null, though...

>>85223393
The problem is that legally speaking, if someone embeds CP into an image and you download it without knowing it contains CP, you're in violation of the law for possessing CP. It doesn't matter if it was intentional or not or if you knew or not; you're guilty until proven innocent. This also goes for the 4chan servers, which could get legal issues when people embed CP.

>>85230020
You may want to look into the PSP's UMD discs, I'm not very familiar with the subject but I think they're related to MiniDiscs and PSP games use ATRAC for audio.
>>
Does gelbooru somehow block explicit cute and funny? I know it doesn't show up in search results unless you change the settings, but I'm using direct image links. I can't download them with szurubooru, though when I try using wget it works so maybe it's the useragent.
>>
Now that this thread covers the broader subject: I have four HDDs which I originally intended to put into a RAID array, but stuff happened and now I'm already using one and it has quite a lot of data on it. Can I create a software RAID (on Debian, Mint specifically) using that disk and the three others with the data automatically being kept, or do I need to find a way to back up the data first because wiping is inevitable?
>>
>>85244197
it hides them from search results unless you have an account
if you have direct link you can still view the image
that's by design
>>
>>85244197
just use the API, it doesn't hide shit
>>
>>85244239
>>85244268
I am using direct links with the img1 subdomain which I assume is gelbooru's CDN and still getting 403'd. It doesn't happen with regular pictures nor other boorus that's why I can't figure out if its gelbooru's or szurubooru's fault.
>>
>>85244197
No. Direct links work.
>>85244319
If you're running a script on gelbooru that can cause a 403. Try changing the user agent.
>>
I was trying out Hentoid/Hendroid because I want a manga gallery app on android specifically for hentai.
Besides some annoying UI bugs, why the fuck is the image quality so shit? I opened my downloaded galleries on MiXplorer and they look really sharp, but the text looks like blurry shit on Hentoid unless I fit the image vertically and that's obviously not a solution
The main tachiyomi app doesn't support exhentai, right? I can only see an e-hentai extension
>>
>>85244444 (nice!)
>Try changing the user agent.
This fixed it, thanks. I found out szurubooru has a convenient useragent setting in its config file.
>>
>>85244239
don't even need an account. just tick the show all results checkbox and save settings
>>
How do I download gelbooru. Like all of it.
>>
>>85244773
very slowly
>>
>>85244777
So there's no torrents out there?
>>
>>85244773
Start with danbooru2020, it's a torrent and already gets you to the ~70% mark.
Then download the metadata for gelbooru, filter out all "apparent hashes"* you already have and download the rest.
*not the actual file hash but what it says in "hash" or "md5" in the JSON, it's not always the same.
>>
>>85244773
Why do you need all of it?
>>
>>85244803
thanks
>>85244872
just in case
>>
>>85244882
>just in case
There are other boorus with stuff that is forbidden on dan-/gelbooru (e.g. western artists) and of course all the doujinshi sites like exhentai.
The collection is never going to be complete.
>>
shill the server in the official discord
>>
>>85244192
>It doesn't matter if it was intentional or not or if you knew or not; you're guilty until proven innocent
no you aren't, but you'll likely be found guilty. There is some degree of plausible deniability with things like this, but not very much because it is hard to trace data. It wouldn't go to court 99.99% of the time anyway, mind you

this entire conundrum is why I have a hard time justifying laws that ban digital content. No matter how good the intentions are I'm absolutely convinced the aggregate chilling effects seriously outweigh any possible benefits
>>
>>85245031
>stegano is unpatchable
*gets b&*
>>
>>85244895
just download everything off sankaku, baraag, pixiv, e621, inkbunny and ATF.

The only thing you'll be missing is twitter shit, which is impossible to sanely rip.
>>
>>85245579
>ATF
Uohhhhhhhhh
>>
>>85244895
>The collection is never going to be complete.
such is life of a data hoarder
many such cases!
>>
>>85245616
Baraag is far better because the actual artists post there, so by following & boosting their content you are supporting them. ATF is just re-uploads from pixiv mostly.
>>
>>85244212
Anytime you setup a RAID you have to format all the disks involved as far as I know.
>>
>>85245579
I use
ATFBooru
Kemono
pixiv
nijie
Danbooru
Gelbooru
yande.re
konachan
>>
>>85244212
With mdadm it is possible to start with degraded arrays (e.g. RAID1 with just one disk) and then grow/"replace" (the "failed" air disk with a real one).
It does need to rebalance after every step though, so it can be quite time consuming.
Generally speaking you should have your backups in order before you think about raid, so maybe allocate your disks accordingly.
>>
>>85242144
Most of the website is generated on the client side using javascript to request data from the server. A list of all the videos in the show including the URLs are found in this json file https://nwapi.nhk.jp/nhkworld/tvepisode/v6b/list/traincruise/all.json
>>
going to try hydrus PTR
wish me luck anon
>>
>>85247829
I've been procrastinating on tagging for the longest time
maybe the next time my internet dies I'll get to it out of sheer boredom
>>
>>85248216
actually i dont need PTR that much
i use Grabber which saves tags in txt file beside image which hydrus can import when i add files
>>
File: 1641686957443.jpg (150 KB, 1024x1024)
150 KB
150 KB JPG
>>85248238
most of my stash is shitty memes and art from tw*tter
so it has to be tagged manually
>>
>>85248665
>shitty memes
>twitter art
guess good luck
>>
>>85249372
>twitter art
wdym
https://twitter.com/Tsubasachyan
>>
Are the Toshiba MG08 drives any good? Have a MG07 and it hums/vibrates too much.
>>
>>85249417
UOOOOOOOOOOOOOOOOOOHHHHHHHHHHHHHHHHHHH
>>
>>85249455
>vibrates too much
why would be that a problem ?
>>
>>85249614
Don't want a humming PC. 12TB Ironwolf is fine in terms of vibration, but they are way overpriced and consumer shit with lower workload rating.
>>
File: 432157132839.jpg (25 KB, 480x375)
25 KB
25 KB JPG
>>85222041
I'm kinda hungry now
>>
>>85249417
you can tag this guy by looking at tsubasa_tsubasa tag on any booru

i mean art like pic related
>>
>>85222041
>100% real cheese as marketing point

oh say can seeeeeeee
>>
>>85249659
in my case i need a humming pc since it helps with my tinnitus
>>
so is the /cum/ general permanently rip now?
>>
related to compression technology
anything that can compress data to absolute limit ?
-images
-videos
-different data
any help regarding that since 1tb here cost 70$
>>
>>85249847
not even 3 days has passed, they can't even appeal their bans yet
>>
>>85249850
>1tb here cost 70$
Grim. Don't you have friends who can smuggle cheap hard drives into your country?
>>
>>85249883
what will appeal do if the bans were abusive in the first place?
are appeals handled by a more reasonable person?
>>
you guys think catbox is down because of PEE shenanigans?
>>
>>85249900
those are mostly used and cost about 40$
>>85249939
maybe
>>
>>85249883
Who exactly was banned?

>>85249939
Shouldn't be. It was only like 5 people max using it. Nobody posted anything illegal
>>
File: 164148550549.jpg (32 KB, 720x546)
32 KB
32 KB JPG
>>85250034
embedposters and people who were trying to (re)make the thread, knowingly or not.
six gorillion cumg posters were banned that day, the numbers don't lie!
>>
>>85249939
I remember Catbox having issues even before PEE was a thing, I doubt PEE or any of those other userscripts affected Catbox in any significant way.
>>
>>85249850
>>85249983
>about 40$
Grim! Do you live on an Island in SEA or the Caribbean?
Anyways to answer your question:
>-images
Webp, jpeg and png are fine, but also look into lossless image optimizers.
>-video
h265, AV1 is a meme. If you re-encode anime, be aware that many, especially TV shows, are actually upscaled 720p or 810p. So test 1280x720 and the original resolution, if there is barely or no difference you can save 20 to 50% at the same quality.
>different data
7zip
>>
>>85249850
>>85250267
Actually forget png, resave as jpeg if you want to lower size significantly and can accept some loss.
>>
>>85250262
People who posted the forbidden link were also banned.
>>
>>85250267
its middle east kek
>loseless image optimizers
where ? any tool recommended ?
>video
not anime but 3d SFM files? , any recommdation for those?
>different data 7zip
here a question i have many of archived Game clients each client in size about 60gb
i have almost 1 tb of those
is there anyway to compress game files? , (textures....etc) ?
i tried to look into how fitgirl compress games but didnt get to that far
>>85250300
i would prefer png
>>
>>85250262
i posted the webm with the thick loli butt and i was not banned kek
>>
>>85250344
pinga
or
image optimizer for the max
>>
>>85250344
>where ?
https://github.com/tjko/jpegoptim
https://github.com/yumeyao/pngoptim

>SFM
I think that's highly compressible, if you just use 7z. Unless you actually mean rendered output, then still h265.

>i would prefer png
Just try it. If you have images with many gradients then jpeg at quality factor >95 is basically indistinguishable, but 1/10th the size.

>Games
Probably not, the downloadable installers are already compressed.
>>
File: FJAXNZUXMAktACY.jpg (174 KB, 2048x1753)
174 KB
174 KB JPG
>>85230717
How is selectolax better than beautiful soup? Running faster doesn't make much difference when it comes to web scrapping because the bottleneck will always be site you access.
>>
File: rocket1.png (3.93 MB, 2500x1667)
3.93 MB
3.93 MB PNG
>>85250406
More like 1/4 in 4:4:4.
>>
File: image001.jpg (1.14 MB, 2500x1667)
1.14 MB
1.14 MB JPG
>>85250548
irfanview 95 quality/no chroma subsampling
>>
File: rocket2.jpg (1.12 MB, 2500x1667)
1.12 MB
1.12 MB JPG
>>85250562
Hmm, GIMP 95 quality floating point.
>>
>>85249850
This is more for backups bit it might help you out either way: https://github.com/borgbackup/borg
>>
File: 1642354652219.jpg (164 KB, 1680x1050)
164 KB
164 KB JPG
need to display a large amount of stored images. what should i use for it?
>>
>>85250406
i use pinga and it makes my laptop with 5800h sound like a jet engine but it can keep up
any good parameters for JPG files ? (using jpegoptim)
>SFM
i have a bunch of artists collection i need to compress which consist of SFM 3d mp4 files
got any advice ?
>games
guess i have no luck with games ?
>>85250653
>linux only
man wish it works on windows
>>
File: 1596793762200.jpg (972 KB, 1600x2260)
972 KB
972 KB JPG
>>85222041
So what is the safest/quickest to repair 4 disk configuration? As far as i am seeing this, 10 can allows only two specific plates fail at best, while 6 allows for any two plates to fail, but takes very long to restore? I plan to build 4x 10TB.
>pic unrelated
>>
>>85250518
I was going to say css selectors which is the reason i use it but then realised bs4 has them. now idek why i use selectolax kek
>>
>>85251129
>As far as i am seeing this
Correct. Longer rebuilds incur their own risk, but raid6 is pretty solid. Raid 10 has the better performance (4x read speed, 2x write speed).
>>
>>85251259
So pretty straightforward. A more practical question to add on, if i'm repairing a raid6 and another plate, fails during, will the repair fail at whole or continue as is for the original broken plate?
>>
>>85249850
You're more or less stuck with what you have unless you recompress/reencode your files at a loss. Windows offers compressed drives, and you can stuff other files into archives, but there's little to gain as most images and videos are compressed a lot as it is.
>>
>>85251355
Single additional failure should not affect the rebuild for the drive you replaced, but make it a lot slower.
>>
>>85250731
display how
elaborate
>>
>>85250790
>>85250406
>games
but why hoard vidya at all
>>
>>85251403
Thanks again! I think that is enough reassurance for me to go the Raid6 way.
>>
>>85251429
game client
WOWS in particular because certain replay files wont play on newer clients so you need old clients around
>>
>>85222874
Habitual
>>
>>85251411
to look at?
I've started using Plex's photo library. but it takes forever to scan everything
>>
Advice on buying memory cards?
Every search for them seems to turn up a mine field. I'm seeing memory cards with tones of storage space but suspiciously cheap, yet have good reviews.
>>
>>85252008
Don't buy at Amazon, Ebay or Aliexpress. Then you get more legitimate offers.
>>
>>85252064
>>85252008
i got a lexar 128gb from ali and its working fine
>>
I got a new 10TB WD Red, and am paranoid that it is fucked up. How normal is it to hear occasional sounds that aren't normal vibrations come from a hard drive? Last one I received was beyond fucked up and now I feel I might just be paying too close attention. Either way I should get another one and back my shit up just in case.
>>
>>85252180
Unsolicited clicking is normal with WD drives, they call it "preventive wear leveling".
>>
I noticed that Third Eye Sankaku image/video swaps fail after a while. It seems the source links have an expiry time, and Third Eye does not refresh it afterwards; you need to manually clear the cache to see now-broken Sankaku images/videos.
>>
>>85252230
Interesting. It seems like that is a newish feature and none of my old WD drives do that. Thanks.
>>
>>85252281
>sankaku
i know those kikes are really forcing their paywall
>>
>>85252180
You should run an extended SMART test just to be on the safe side. I hope you don't have a singular 10TB HDD. That's like going raw in a hooker. Either RAID it up or have a mirror backup incase it fails.
>>
>>85249883
I'm not banned, glad the embedding retards were though.
>>
Catalogizing is cool, but you need to store that shit somewhere. As a greedy poor nigger, I don't want to buy extra disks to store my terabytes. What if I abused big tech free space?
Is there any program that allows me to create a virtual disk out of many free space providers (like Google Disk, OneDrive, etc.), with duplication if possible?
>>
>>85253035
Multicloud, Odrive, AirExplorer maybe
>>
>>85253035
I don't know the situation in your country, but really not worth the hassle, considering how cheap hard drives are.
It's not just an organization nightmare, but you always have your internet connection as bottleneck inbetween you and your data.
>>
>>85253035
They’ll pull the plug on you sooner or later anon, stop being a stingy faggot and pay for the hardware
>>
>>85222041
Why Hot Pockets?
>>
>>85253135
Not OP, but Pizza and Cheese. Cheese Pizza, if you will.
>>
[spoiler]Does embedding still work?[/spoiler]
>>
File: 1619807105853.jpg (62 KB, 301x729)
62 KB
62 KB JPG
>>85253185
Okay that makes sense. Seems like a stretch, but it can make sense. I have nothing else to contribute to this thread, here's a penguin.
>>
File: cunny investigation.webm (1.92 MB, 711x400)
1.92 MB
1.92 MB WEBM
>>85253135
>>85253185
>>
>>85253197
Fucking retard
>>
>>85223462
bump, good question.
>>
Why are you cunny retards trying to get this thread nuked, fuck off to /b/
>>
>>85253243
PPP are Kemono Friends, there is no age of consent for animal girls.
>>
>>85252928
based, third eye is the way
>>
>>85252008
I pretty much only get these. Amazon exclusive branding.
>>
uh oh jannies gonna freak
>>
>>85249850
Videos: av1 (svt-av1 encoder) but obviously you'll lose visible quality.
Images: jpeg-xl. Can compress jpegs losslessly for ~20%. For other images use -d 0.5 and below for visibly lossless results (at 400% zoom). -d 0.8 would work if your eyes aren't great.
I would suggest deleting stuff that you don't care about.
>>
>>85253197
It works with catbox links. However the jannies caught on and are using it now, so if they catch you, they will ban you according to the mod who cancelled /cum/
>>
>>85253035
hard drives are fucking cheap
what the fuck man stop beign a nigger
>>
>>85252525
That's right. You RAID1 on hookers. If one gets infected, you can fall back on the other. RAID5 or 5 is ideal, but it's costly and you might suffer bandwidth issues.
>>
>>85253583
>RAID5
What the hell? Nobody uses raid5 anymore. The only raids you should be using are 10 or 6.
>>
>>85253089
Thanks, anon.
>>85253119
Yeah, so I was thinking if it's maybe possible to do the organization automatically. The internet bottleneck is real, but most of the time, I don't need all of my data at once anyway.
>>85253134
If I duplicate it, I'll be able to download the data if they won't pull their plugs all at once.
>>85253535
Wouldn't it be cool to get the most out of free space providers? It's not even about the money, it's like, a fun thing to do.
>>
File: 164155518121.jpg (252 KB, 1200x880)
252 KB
252 KB JPG
>>85252230
Thanks anon, I always wondered why my HD was noisy
>>
>>85253709
>Wouldn't it be cool to get the most out of free space providers?
No, because they'll randomly delete all your shit whenever they feel like it.
>>
Thread hit the bump limit.

>>85253468
>>85253243
They never freaked out about /cumg/ before people started moving towards embedding CP in images, and they won't freak out again as long as we don't make the entire thread about violating GR17 again.

>>85253511
>according to the mod who cancelled /cum/
Was there some sort of semi-official statement?

>>85253709
>it's like, a fun thing to do.
It's like, why we can't have nice things. Shit like this is half of why shit gets locked down all the time. (The other half being corporate greed, but at least it gets harder to justify if people aren't abusing the system.)
>>
>>85253197
yes, i've seen embeds on other boards stay up until archive.
>>
>>85253511
There are embeds in this very thread, and they aren't deleted. The extension dev was mad in the matrix channel and said he planted a backdoor in the extension that targets jannies and mods, so maybe they're scared to update?



Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.