rclone is a life saver. You might think that’s hyperbole, but it’s not. Rclone is this file storage Swiss army knife it speaks many protocols between the various online storage providers allows transfer between them. I just helped my sister download a massive set of photos shared with her from Google Drive that were refusing to be zipped and downloaded. We were waiting for a while to download these files, until I said let’s just go and use rclone. It’s not exactly a tool that is friendly to expose to the average nontechnical user, but when you need it you need it. So I went ahead and set it up to connect to her Google Drive.
She uses a mac and I use a Linux laptop, but teamviewer soon bridged that gap and I remoted in to help her. (Yes I know it’s old and not recommended anymore it’s still great for cross platform uses). I open a terminal after downloading rclone, and extract the files. Next I try to run it. The app is unsigned and Apple doesn’t like unsigned binaries. So I open the README, and there’s a command listed about halfway through to have mac run it. We’re back in business.
Next we needed to configure a remote. So it’s ./rclone config, since I’m not
installing it into her system since she doesn’t need it all the time, so we use
the binary in place. We add a new remote and walk through the default
configuration. All is properly configured. I look at the help menu a couple
times, and see how to list directories using the ./rclone lsd remote:
command. The shared folder we want isn’t there. So I start searching online for
a solution. I come across the excellent first party documentation. At
first we tried “team drives” which sounded like it might be what we want, but
it was actually organization file shares. So I had to reconfigure the remote
since I couldn’t change it back after it was set as a “team drive.”
After that I read the documentation some more there’s a flag to use to
access “shared with me” items, --drive-shared-with-me. Reading StackOverflow,
I come across another useful flag, the progress -P. Any file transfer tool is
worthless and frustrating without telling you how much work its done. This
progress screen is very advanced it tells you how much data is left to go, the
number transferred, and which file is currently being transferred.
So the final step is to actually start the download. There’s many verbs to use here, but the most obvious one is to sync it. Especially cause a sync will resume where it left off if interrupted. The overall command ended up looking a bit like the below. It took just under 20 minutes to download the 2400 items summing up to 2GB. Especially considering that google wasn’t able to generate a zip for the shared images.
mkdir downloaded
./rclone -P --drive-shared-with-me "gdrive:Shared Folder" "./downloaded"Don’t forget to make a folder to store the downloaded contents. We don’t want to make a mess to clean up later.
This certainly isn’t the first time I’ve used rclone, and it won’t be the last. In the past I’ve used it to back up folders to google drive, and transfer things between cloud providers. Everyone I’ve introduced rclone to has been very impressed with it, and found uses I couldn’t even consider. There are a couple concerns to be aware of.
- OAuth Authorization Scope - RClone comes with a default client id to access google drive, this gives the author access to your google drive when you authorize it. You can and should provide your own key. Rclone even encourages you to (just goes to show how ethical the author is).
- Encrypting the credentials - rclone provides options to encrypt the various keys it uses to talk to the cloud providers. I’ve never bothered with it because I’m often using it for one off jobs where I’m deleting it when I’m done.
- Revoking access - if you use the default client id, you should definitely revoke access when you’re done using https://myaccount.google.com/permissions
I also want to make clear that rclone supports more than just Google Drive. It supports about 50 different backends at this point and allows arbitrary transfers between the services. It nice to find a solution that is stable for speaking to Google Drive. I’ve tried a bunch of different uni-taskers in the past, and none of them work as well as rclone.
All in all, I hope this PSA serves to help you. I find rclone really valuable to unlock my personal cloud data and take back ups. Also this is the 100th post I’ve written here, and here’s to many more. Thank you for reading and following my work.