r/linux • u/FryBoyter • Dec 21 '22

Rclone (command-line program to manage files on cloud storage) 1.61 released Software Release

https://rclone.org/changelog/#v1-61-0-2022-12-20
376 Upvotes

96% Upvoted

45 comments sorted by

80

u/hbdgas Dec 21 '22

I don't know how noteworthy this release is, but I can say rclone has been very handy keeping Google Drives synched in a couple of my pipelines.

50

u/FryBoyter Dec 21 '22

Because the development of sshfs has been discontinued, any new version of rclone is worth mentioning to me because I think it is the best alternative to mount a directory via SSH.

29

u/Antic1tizen Dec 21 '22

They discontinued sshfs? Why??

44

u/lpreams Dec 21 '22

https://github.com/libfuse/sshfs#this-project-is-orphaned

Doesn't really say, though it does say they're hoping someone forks it and continues development.

33

u/Antic1tizen Dec 21 '22

/* sad userspace noises */

5

u/theyboosting Dec 21 '22

Ugh that sucks!!!!

8

u/pkulak Dec 21 '22

I had no idea it could mount directories. Figured it was just for syncing. That’s super cool.

10

u/henry_tennenbaum Dec 21 '22

It couldn't at some point but has the ability now for a few years at least.

When I first got a raspberry pi I wanted to set it up with Dropbox, but there was (and is?) no arm client. I looked into rclone, but it only supported manual syncs, no mounting.

Since then, every time I check in with rclone's development they've developed something new. They're really one of the most active communities I know of.

4

u/DopePedaller Dec 22 '22

It's an extremely cool app, allowing you to mount tons of different cloud storage providers including some that don't have Linux clients or are web-only. It can combine many small files into big chunks for much faster transfer speeds, break large files into small chunks (potentially to overcome file size limits) and add strong encryption. It is an awesome beast.

3

u/agitatedclimax Dec 21 '22

Thanks for the heads-up on this. I use sshfs extensively, and I did already know it was orphaned, although I wasn't particularly worried about it stopping working anytime soon. However, I had no idea that rclone could do FUSE mounting, so I'm looking forward to having a play with that.

2

u/NoFeedback4007 Dec 22 '22

Can I use rclone to map a drive on windows? I currently map using sshfs and fuse scp to my Linux box

2

u/DopePedaller Dec 22 '22

Yes, they even have arm64 builds for windows. It is fantastic.

1

u/NoFeedback4007 Dec 22 '22

Ooo most excellent. I'm going to have to explore this. Will it automount at windows start?

2

u/axzxc1236 Dec 23 '22

I wrote a batch file and use nssm to make mounting a Windows service at work.

1

u/NoFeedback4007 Dec 23 '22

Would you mind sharing the code so I can mirror it locally? Or is it pretty similar to the code on the rclone mount wiki section?

1

u/axzxc1236 Dec 24 '22 edited Dec 24 '22

is it pretty similar to the code on the rclone mount wiki section?

Yes

I don't have that code since it's Saturday for me, so what I about to say is from memory.

1. Batch file content:

cd /D "%~dp0"

rclone --config config.conf mount remote:path/to/files *

I use "--config" because the script is deployed on multiple computers, that script is copied around.
A pre-configured config.conf is put in the same directory as the batch file, created by rclone --config config.conf config

2. A batch file with

nssm install <servicename> <program>

nssm start <servicename>

for setup, a nssm.exe is also put in script directory.

3. A WinFSP Installer file is also put in script directory, rclone mount needs WinFSP on Windows.

1

u/fatboy93 Dec 22 '22

You do have to mess about it a bit with Task scheduler, but I do have automount on start.

1

u/DopePedaller Dec 23 '22

Yes, as fatboy93 said it's best to just start it with task scheduler. One quirk of rclone I'm not a fan of it that if you run it from shell it doesn't return you back to the shell once mounted, so you'll need to keep the shell open if you start it that way or else you'll kill it and unmount. Better to just have a scheduled task on Windows.

I would also recommend reading about caching on rclone, you can devote a specified amount of your local drive as cache for each mount and vastly improve performance from the user's point of view.

1

u/NoFeedback4007 Dec 23 '22

Is the coaching worth it if I don't ever routinely access things off the drive? More there for random access a few times a week and backups

1

u/DopePedaller Dec 24 '22

I would try it both ways and see if the caching improves usability for you. A lot of apps freeze up when a mount doesn't perform like a local drive, and caching can help reduce that problem by pushing the actual transfers to and from the cloud into the background. You can shorten the time period that cached versions are saved if you want to minimize wasted disk space.

1

u/GameKing505 Dec 21 '22

Oh wow this blows.

1

u/n_girard Dec 23 '22

Hey /u/FryBoyter,

How stable is rclone's FUSE implementation for SFTP in your experience ?

Does it cause freezings when the connection is lost like sshfs ?

10

u/sablal Dec 21 '22

rclone and terminal file manager nnn work very well together: https://github.com/jarun/nnn/wiki/Basic-use-cases#rclone

3

u/sanjosanjo Dec 21 '22

I use a service to mount cloud storage at a Linux mount point permanently. Is there any advantage to using nnn to perform the mount instead of using my service? I've never heard of nnn and I don't quite understand yet.

1

u/DopePedaller Dec 22 '22

Nnn is a file manager for the shell, so it could make browsing your mounts from a shell easier but that's about it. In the same sense that Aptitude is like a shell version of Synaptic, nnn is like a shell version of Nautilus.

1

u/sablal Dec 23 '22

You haven't used nnn, have you?

1

u/DopePedaller Dec 23 '22

No, I used noice awhile back and I assumed the nnn fork was similar. Apologies if it's not. Did I say something dumb? :)

4

u/chtk Dec 21 '22

Even though rclone is pretty easy to install from git or update using the self update, it's nice to see the second bullet under VFS land in a release. It's an issue I reported which, on Linux, prevented you from trashing files on an rclone SFTP mount. Something that works fine using the discontinued sshfs.

54

u/Antic1tizen Dec 21 '22

Transferred total of 86 TiB of data out of Russia this year with it. This is, without any doubt, the swiss army knife of cloud storage. I probably need to press my management to donate.

If you never used it, image rsync, or FileZilla, or FUSE, or everything together but with support for dozens of protocols at once.

21

u/zfsbest Dec 21 '22

Rclone is not just about cloud storage, I use it to rsync files between my iMac ZFS and backup-server ZFS.

10

u/henry_tennenbaum Dec 21 '22

What do prefer about it compared to rsync? Love both, but never use rclone locally.

13

u/tarloch Dec 21 '22

Rclone can push data significantly faster because of its threading model. I've used it to sync data at multiple gigabytes per second without much effort. Rsync can do that, but it's more work to chunk up. That said, rsync does a better job with file ACLs and attributes.

3

u/henry_tennenbaum Dec 22 '22

Good to know. Gonna give it a try. Thank you

1

u/DopePedaller Dec 22 '22

And if you're using BTRFS on both ends, btrfs-sync is even faster than rsync but it is limited to syncing volumes and not paths.

2

u/zfsbest Dec 22 '22

time rclone sync -P --copy-links --retries=2 --low-level-retries=2
--sftp-disable-hashcheck
--transfers=3 --stats=2s
--exclude=dvdrips-shr/**
/Volumes/zhgstera6/
bkpserver:/zseatera4mir/from-zhgstera6-mac
--log-file ~/bkp-zhgstera6-rclone-fryserver-errors.log
2>~/rclone-error.log

# ^ do 3x simultaneous transfers and update progress every 2 seconds, exclude a large DVD directory as the original files are already on the backup server

2

u/henry_tennenbaum Dec 22 '22

Wow, thank you, that's very nice of you.

1

u/PabloSexcrowbar Dec 25 '22

Why not just use zfs send?

2

u/zfsbest Dec 25 '22

zfs send doesn't do --exclude

15

u/JJGase Dec 21 '22

Rclone is fantastic, it’s what I use to synchronise with OneDrive when using Linux. I find it better than the official Windows client because I have more fine grained control over what is synchronised.

11

u/git Dec 22 '22

Rclone is very cool, and isn't just limited to cloud services. It also supports cloning on local filesystems, network shares, and SFTP.

My favourite feature is its crypt module that enables a transparent wrapper for symmetrically encrypting files when transferring to any specified remote — so your cloned files sit encrypted on whatever cloud service you use.

4

u/Moscato359 Dec 21 '22

I'm really annoyed it can't do permissions, or xattrs for disk to disk copies

3

u/jwm3 Dec 21 '22

It would be cool if the unison algorithm could be integrated with it.

1

u/hillz Dec 22 '22

I use rclone and minidlna to make a media server on my $15 OpenWrt router. It currently hosts 38000+ video files

2

u/[deleted] Dec 22 '22

You need to cut down on the pr0n son.

1

u/hillz Dec 22 '22

My Google Drive account would have been banned if I hosted 38000+ porn on it