Hey /g/, I'm looking for a specific type of backup solution.I'm going to be backing up from time to time to external HDDs. Ideally I want something automated or semiautomated to sync the new data when the HDDs are plugged in. The thing is, I will be backing up very large files and I want something smart like syncthing that will only add the changed parts of the files instead of coping the entire files again every time.Something cross-platform is preferred because I currently use windows but plan on moving to GNU fully soon. If not possible, two similar solutions (one for each OS) is acceptable.And last but not least, I would really rather a free software option, or at least open-source. Suggest proprietary software only as a last resort, please.Any ideas? Also, does anyone here do backups in a similar manner?
>>70566801Sounds good, but not exactly what I'm looking for. I want my backups to be fully accessible directly from the drives, without the overhead of restoring from the backup first.Also, it seems like this would also bloat excessively with time. I want to record the changes made to the files, not additions that will change the files when restoring.I might consider using this for cloud backup, though, so thanks.
>>70566309Absolutely no warranty, but if you feel lucky enough you can use my implementation of a backup tool for big files.I use it to backup my 2 tb big files to external drives. The current implementation is windows only but i think wine should make it work on linux without any problems. Good luck in finding a solution. https://github.com/Cloudnaut/HFFDC
>>70567274>tfw trusting all your data AND your backups to the hacker known as 4channel.Seriously, though. It sounds interesting. Thanks for the share.
>>70567536It's open source, no need to worry
>>70566309A shell script.
>>70567592I'll take it if you have one.
>>70567608Why not write one yourself?
>>70566309>>70567274* You want diff backup* You want the files to be directly accessible* You are storing large files* You want to do this on LinuxA solution that werks on Loonix and isn't some gentooman's github is ZFS snapshots + send/recv. Have all your drives formatted as ZFS and use send/recv to push snapshots to your backup.
>>70567548lolwut?>>70567671If I knew how to do that I wouldn't need to ask /g/ for assistance. I don't even know what tools would be necessary for my use case. I will take nudges in the right direction if you have them.>>70567684Unfortunately zfs is not an option. My hard drives are already formatted as ntfs and they'll sadly have to stay like that for the rare occasion I might need to access them on windows. Also, on GNU I'll be using btrfs, but zfs. But thanks.
Does anyone else have some ideas?
>>70569218I don't know, man. How big are the files that we're talking about here?
>>70569386Nothing extreme like the 2 TB this anon >>70567274 uses their program with, but some can be upwards of 20 GB which already would be very overkill to move back and forth every week on a USB 3 connection. Not to mention the unnecessary writes to the HDD. That's why I need to send only the changes to the files, which apparently is called delta.
rsync --inplace --no-whole-file
>>70569675Thanks. After researching for a bit it does seem like rsync would probably be the logical choice. Unless anyone else has a better solution.
>>70569675This might be a solution if you don't expect any speed improvements. It will write less to the hdd but it will also take much longer than a regular copy, just saying it. Rsync has to read the section from the hdd before it can compare their contents. This will lead to increased copy times.
>>70566309cron and rsync - google it, it'll take about 2 min to setup if you've got half a brain.
>>70571112Why would reading from the external hard drive take longer than writing to it?
>>70571192Because if you copy over you just override the bytes without knowing if anything had changed.Rsync has in addition to writing the bytes the overhead of reading all bytes, also the bytes which didn't change and calculate the checksum for each chunk
>>70571135I'm actually reading about udev and rsync now. Apparently with that I can make the backup happen when I plug the drive in, which is the level of automation that I wanted. Of course it's no a solution for windows right now, but I think it will come in handy when I move to GNU.