The original post: /r/datahoarder by /u/rez0n on 2024-07-27 17:13:28.
I'm software developer and a bit data hoarder, I keep all historical project files and documents for any client. Please give your recommendations on files syncing if you have similar experience.
For many-many years I syncing all my files and projects using cloud providers, I started it since Dropbox has out. It is convenient and partially replaces backups in sense that I can completely format or destroy my computer and do not lost any data.
Latest few years I struggling with this and continuously migrating from one provider to other because each have different issues.
What I have
- ~4 millions files
- ~150GB total size
- Most of files are Python / JS projects
- Half of directories are git repositories
- Some of projects contains node_modules / Python venv (it contains a lot symlinks)
What I want
- Selective (smart) sync. I need to keep only active projects loaded on disk on per-file basis.
- Fast syncing as it possible, taking into account crazy amount of files
- Correct behaviour in syncing of git repos in regard of branch switching and so on.
- Ideally support of .ignore file or alternative feature to ignore specific dirs from sync
My experience:
Right now I use MacOS on all computers, so cross-platform no more in a topic.
Dropbox - Before it migrated to MacOS "File Provider" it worked amazingly and mostly suit any my demands. Now sync are sluggish, many sync errors appears, when I set directory as "Available offline" it downloads partially and finish downloading some files only after access attempt.
No more many of Dropbox features like (LAN sync), defined location of files and access speed to files though File Provider are very slow.
Google Drive (File Stream) - Last time tried it two years ago and has been disappointed. Sync are fast, but it start full re-sync every week or two (which lasts few days).
Resilio Sync - Not a cloud, but also fine. Sync are fast and reliable, but no "Selective (smart) Sync". You can only completely remove directory from the computer.
Seafile - I was amazed by the speed of syncing, but I ceased testing almost on the start, because it can't sync all files, Python venv being broken on syncing due linked files.
iCloud - I made few attempts and right now I moved to it. It fast on initial sync, but then it becomes very slow when you need load remote files or after creating node_modules. Another critical issue, when I switch git branches it creates a lot of duplicate files as it seems sees two versions of the same file at a time.
Of course you can say that I can use git for this, but there are also a lot files that are should be ignored and did not pushed to the repository, but better if these files are same on all workstation for seamless work.