hate these ads?, log in or register to hide them
Results 1 to 18 of 18

Thread: (Data) backup / synchronization / replication

  1. #1

    Join Date
    May 31, 2011
    Posts
    4,168

    (Data) backup / synchronization / replication

    I paged back 'til 2015 threads and didn't see a fitting one, so here's a new one.

    I'm in the process of recovering from a HD crash. Git and regular backups prevented the worst case scenario, but I still lost a few files/newer changes. But not much, thankfully.

    Still, that leaves me a bit dissatisfied. The issue with my backup/sync job is that it's a) a manual process and b) it takes a bit of time (mostly for traversing the folders and finding out which files have change). That's why I don't do it every evening. So now I'm looking for a real-time sync option, that monitors specific folders for change and then syncs those.

    In theory OneDrive does this, if i would only let you add multiple/different folders to sync. I've tried to beat it with junctions, but apparently that doesn't work any longer (my experience confirms the article, though using OneDrive for business)

    Can anyone recommend a real-time sync tool? I'm well aware that the initial process might take quite some time. Which is fine as long as it does it jobs afterwards without too much system load.

    I of course tried asking aunt Google, but it comes up with tons of results. I tried a few, but none does actually fit the bill/work at all. E.g. DirSnyPro crashed with a NullPointerException during its discovery/preparation phase.

    And although I prefer a client only solution (sync'ing to a network share) I wouldn't shy away from a local "cloud" solution, if that's the way to go and someone here has experience with a working solution. I'm currently looking at Seafile for this, of which I personally use the client already for some time with cloud storage from https://en.yoursecurecloud.de/ for this.

  2. #2

    Join Date
    April 10, 2011
    Posts
    308
    I've been using SyncToy: https://www.microsoft.com/en-gb/down....aspx?id=15155

    It takes a little while to set up, but once it is done it is pretty quick to run. Can have it mirror one way or both ways if you so choose. It will replace any files that have changed from the paired folders.
    Last edited by Mythago; October 23 2018 at 05:06:56 PM.

  3. #3
    Daneel Trevize's Avatar
    Join Date
    April 10, 2011
    Location
    T L A
    Posts
    12,403
    Quote Originally Posted by Hel OWeen View Post
    a) a manual process and b) it takes a bit of time (mostly for traversing the folders and finding out which files have change).
    You don't mean totally manually finding the different files by personally checking creation/modified timestamps, right? But more like running rsync, yeah?
    Quote Originally Posted by QuackBot View Post
    Idk about that, and i'm fucking stupid.

  4. #4

    Join Date
    May 31, 2011
    Posts
    4,168
    Quote Originally Posted by Daneel Trevize View Post
    Quote Originally Posted by Hel OWeen View Post
    a) a manual process and b) it takes a bit of time (mostly for traversing the folders and finding out which files have change).
    You don't mean totally manually finding the different files by personally checking creation/modified timestamps, right? But more like running rsync, yeah?
    Yes to the later, manually running the application (which is MS SyncToy Mythago mentioned above). Sorry for not being clear there.

    BTW, I just tried the "junction magic" again with OneDrive and it did work for a couple of folders. I do not know why it works with one folder, but not with another one, but I suspect it might be related to the folder's size/amount of files/(sub-)folder depth of the root folder. So that only solves a 'lil part. OneDrive, being a pure sync tool, also doesn't allow to specify how it should operate, i.e. accidental deletions might happen.

    [Added]
    So here's a couple of restrictions that apply to OneDrive.
    Last edited by Hel OWeen; October 24 2018 at 02:58:10 PM.

  5. #5
    Movember 2011Movember 2012 Nordstern's Avatar
    Join Date
    April 10, 2011
    Posts
    9,484
    Can you not use the backup tool in Windows 10 to do a backup onto another drive?
    "Holy shit, I ask you to stop being autistic and you debate what autistic is." - spasm
    Quote Originally Posted by Larkonis Trassler View Post
    WTF I hate white people now...

  6. #6

    Join Date
    May 31, 2011
    Posts
    4,168
    Well, to be honest: I've never ever used Windows Backup before. I give it a try and see if that's sufficient.

    Besides - using a builtin Windows tool? That's too obvious and too easy.

    [Added]
    OK, I did a test run with one folder, ~ 100 MB, consisting of files and a few subfolders. It's a bit ... awkward. It does too much in the sense that it adds a couple of meta information files and folders to the backup so one can restore via the backup itself. Which makes sense for the way it's supposed to be used, but which isn't necessary for what I'm looking for.

    It also composed the backup in weird ways such that it created two ZIP files from the content of that one folder. Both of them contain parts of the original folder. So in a real life emergency case, I would've to wade through the ZIP files, trying to figure out which folder contains what part of the backed up data, and from there copy the missing files back.
    Last edited by Hel OWeen; October 25 2018 at 01:29:30 PM.

  7. #7
    Daneel Trevize's Avatar
    Join Date
    April 10, 2011
    Location
    T L A
    Posts
    12,403
    I can't get past a DIY approach to this:
    if you want truely synchronised (blocking-while-writes-complete) data between drives, why not use a filesystem that does that internally, such as Btrfs/a RAID setup;
    if you don't mind drives being out of sync for a bit, why not cronjob an rsync call every minute/hour, to sync 2 local or remote paths?
    I'm missing some benefit of a GUI, or the network complexity, or multiple sources of writes?
    Quote Originally Posted by QuackBot View Post
    Idk about that, and i'm fucking stupid.

  8. #8

    Join Date
    May 31, 2011
    Posts
    4,168
    For one: it's a Windows workstation (laptop even), so while a decent fileystem or RAID would be nice, it's not gone work. That also rules out rsync, although currently I see myself heading in a scheduled robocopy direction.

    I don't need a GUI, I'm more than fine with a command line tool.

    As for the tool itself: the way OneDrive or Google Drive work is what I'm looking for, except for the target location being the cloud, but something local. I point to a couple of folders as the source, a drive/folder as the target and the tool keeps monitoring the source(s) and copies over the new/changed files/folders. As simple as that. Bonus points for providing some decent filters.

  9. #9
    Daneel Trevize's Avatar
    Join Date
    April 10, 2011
    Location
    T L A
    Posts
    12,403
    The network share you mention, can that not host the rsync process, and sync your remote laptop share with its local storage, reversing the perspective?
    Quote Originally Posted by QuackBot View Post
    Idk about that, and i'm fucking stupid.

  10. #10

    Join Date
    May 31, 2011
    Posts
    4,168
    Hmm, nice line of thought to "reverse" the process... food for thought.

  11. #11
    Lana Torrin's Avatar
    Join Date
    April 13, 2011
    Location
    Bonding around
    Posts
    19,035
    https://www.idrive.com/

    It took over (for me) from where crashplan stopped.

    Windows backup is 'good' but like all Microsoft products it will just suddenly stop one day with no indication as to why, and there is a 'better than 50%' chance it wont even be kind enough to tell you.

    You could also get almost any NAS made in the last 10 years as they ALL come with backup software of some sort.
    Quote Originally Posted by lubica
    And her name was Limul Azgoden, a lowly peasant girl.

  12. #12
    Mallet Head Donor 56k Lagman's Avatar
    Join Date
    May 5, 2011
    Location
    Who fucking knows
    Posts
    4,689
    I just use google drive sync, it's not pretty but I don't have high expectations for enterprise grade backups on consumer devices

    Quote Originally Posted by Duckslayer View Post
    I should be home.now but I keep stopping to post. I'm in need of a mega poo. so much so that I'm tempted to leave slurry across one of these gardens and deal with the wiping later. gonna toss a coin

    phoneposting

  13. #13

    Join Date
    May 31, 2011
    Posts
    4,168
    I should have mentioned in the OP that it's a company machine/data that I wish to "tighter" back up, so suggestions like idrive.com or Google Drive (we're with MS) won't do.

    For my personal needs, yes, I use Google Drive Snyc/Backup, too. In addition a combined Seafile (the software) and https://www.yoursecurecloud.de/ (Cloud storage). The later is the one I settled with after Wuala closed shop. It is similar to Wuala as it doe end-to-end encryption.

    Quote Originally Posted by Lana Torrin View Post
    You could also get almost any NAS made in the last 10 years as they ALL come with backup software of some sort.
    NAS ... hold on, we've got a couple of those running here. Haven't tought about that one. Thanks for the reminder!

  14. #14
    GeromeDoutrande's Avatar
    Join Date
    April 10, 2011
    Location
    Fakefrenchistan
    Posts
    2,080
    I use SyncBack Pro and Resilio Sync, which let me synchronise files between machines automatically (SynchBackPro lets you intervene manually when you want, Resilio Sync is closer to fully automated with side-effects such as accidental deletes being propagated). I use Windows Backup and Acronis True Image for backup. That all involves a laptop, home pc and a NAS (backing up to cloud storage is possible with all of those products). At work we synchronise some data to a Synology NAS using their synchronisation tool (works like Dropbox but goes to the NAS).

  15. #15
    Mallet Head Donor 56k Lagman's Avatar
    Join Date
    May 5, 2011
    Location
    Who fucking knows
    Posts
    4,689
    Are you just flying in the face of whatever back-up solution your organisation has implemented because I wouldn't recommend that

    Quote Originally Posted by Duckslayer View Post
    I should be home.now but I keep stopping to post. I'm in need of a mega poo. so much so that I'm tempted to leave slurry across one of these gardens and deal with the wiping later. gonna toss a coin

    phoneposting

  16. #16
    GeromeDoutrande's Avatar
    Join Date
    April 10, 2011
    Location
    Fakefrenchistan
    Posts
    2,080
    Quote Originally Posted by 56k Lagman View Post
    Are you just flying in the face of whatever back-up solution your organisation has implemented because I wouldn't recommend that
    I would never dare.

  17. #17

    Join Date
    May 31, 2011
    Posts
    4,168
    Quote Originally Posted by Hel OWeen View Post

    Quote Originally Posted by Lana Torrin View Post
    You could also get almost any NAS made in the last 10 years as they ALL come with backup software of some sort.
    NAS ... hold on, we've got a couple of those running here. Haven't tought about that one. Thanks for the reminder!
    To wrap this up, Lana's suggestion above (NAS) is the winner. Another example of didn't see the wood for the trees. We've got a couple of Synology NAS running, one which I already use as the backup destination for SyncToy. So all I had to do was to install Synology's Clooud Station Server on the NAS + Cloud Station Backup on the client. There's also a sync client, if that fits ones need better.

  18. #18
    Donor erichkknaar's Avatar
    Join Date
    April 10, 2011
    Posts
    11,350
    Quote Originally Posted by Hel OWeen View Post
    Quote Originally Posted by Hel OWeen View Post

    Quote Originally Posted by Lana Torrin View Post
    You could also get almost any NAS made in the last 10 years as they ALL come with backup software of some sort.
    NAS ... hold on, we've got a couple of those running here. Haven't tought about that one. Thanks for the reminder!
    To wrap this up, Lana's suggestion above (NAS) is the winner. Another example of didn't see the wood for the trees. We've got a couple of Synology NAS running, one which I already use as the backup destination for SyncToy. So all I had to do was to install Synology's Clooud Station Server on the NAS + Cloud Station Backup on the client. There's also a sync client, if that fits ones need better.
    Yeah, the segment of products you are looking for are called "Personal Cloud Storage" and its basically a NAS with a cloud like web service that will deliver your files over DAV like protocols to clients, including smartphones, etc, and with some fairly typical consumer level NAT work, you can access your home NAS this way from anywhere in the world.

    I know of the course of many posts, this won't work for you, but I'll detail it for others who might get mileage out of it. What we did at our current place was enforce the "documents" folder (windows or macOs, no one is using Linux as a line desktop) as the place to save your all work, then actually have Google drive serve the documents folder and to mount that way at boot.

    https://support.google.com/a/answer/7644837?hl=en

    We're also enforcing VPN to every endpoint that terminates in Googles Network, so our client -> Google Services traffic is effectively behind the firewall. It's protected by TLS anyway, but, layers of the onion and all.

    It's nice, has worked like a charm for 1 and 1/2 years, and its eliminates the need for traditional backup agents.
    meh

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •