Using Rclone for File Transfer
Rclone is an open source file transfer tool to make transfering files to and from various cloud resources such as Box, Amazon S3, Microsoft OneDrive, and Google Cloud Storage and your local machine a simpler task. Guides on how to set up a variety of resources to transfer to and from can be found at rclone’s webpage.
This tool can be used to transfer files between Beocat clusters and outside cloud providers, such as OneDrive. This tool may allow you to perform backups of critical data onto remote systems.
Setup RClone
- You must be able to access your KSU Office365 account before beginning this process. Contact your local campus IT support if you need help with initial account setup.
- Open a browser on your local machine and navigate to the OnDemand portal for the cluster of your choice. We use Beocat for this example: https://ondemand.beocat.ksu.edu. Select Desktop under Interactive Apps in the menu at the top of the page to get a virtual desktop on the cluster.
Scroll down to the bottom of the next page, and click on the blue Launch button. When the resource is ready, click on the blue Launch Desktop button that appears on the next page.
On the virtual desktop, click on the Terminal Emulator icon at the bottom of the window to open up a command shell.
- At the command prompt in the shell that opens, load the rclone module by entering the command below at the prompt:
Load the Rclone module
[mozes@gremlin01 ~]$ module load rclone
- We will need to start the basic configuration for OneDrive. To do this run rclone config:
Load the rclone config
[mozes@gremlin01 ~]$ rclone config
- In a new configuration, you will see no remotes found. Enter n to make a new remote and name it a name you will know. In our example, we will use “myOneDrive”. Select Microsoft OneDrive by entering in the corresponding number, in our case 27. Hit Enter for the client_id, client_secret, and Edit advanced config. When you are prompted for auto config, select n. The terminal will stop at a result> prompt. Proceed to the next step.
Configure OneDrive
[mozes@gremlin01 ~]$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> myOneDrive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
27 / Microsoft OneDrive
\ "onedrive"
Storage> 27
** See help for onedrive backend at: https://rclone.org/onedrive/ **
Microsoft App Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Microsoft App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Option region
Choose national cloud region for OneDrive.
Enter a string value. Press Enter for the default ("global").
Choose a number from below, or type in your own value.
region> 1
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine (same rclone version recommended) :
rclone authorize "onedrive"
Then paste the result below:
result>
- Now open up another terminal window on the virtual desktop by clicking again on the Terminal Emulator icon at the bottom of the window.
In the new shell that opens, load the rclone module like you did in the first shell with module load rclone, and then run rclone authorize "onedrive" at the command prompt. You will be prompted to go to a 127.0.0.1 address in a web browser.
Authorize OneDrive from the terminal
[mozes@gremlin01 ~]$ module load rclone
[mozes@gremlin01 ~]$ rclone authorize "onedrive"
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize Rclone for access
Waiting for code...
Got code
Paste the following into your remote machine --->
{"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
<---End paste
Right-click on the link and select open link from the menu. A browser window will open on the virtual desktop. On the Microsoft Office sign-in page that opens enter in your KSU e-mail address and click “Next”. You will be taken to your eID single sign-on page where you can sign in using your KSU eID credentials. If login is successful, you should be redirected to a page that says “Success!”
Return to the terminal window where you ran the authorize command. You should see a message instructing you to paste a line of code into your “remote machine” which is the first terminal you opened to run rclone config.
Copy the text by highlighting it, right-clicking, and selecting copy. Then, return to the first terminal with the waiting result prompt and paste the text by right-clicking and selecting paste, then press enter. Next, select option 1 for OneDrive Personal or Business and then 1 for OneDrive (business). Press y at the next two prompts to confirm and q to exit.
Authorize OnDemand on cluster
result> {"access_token":"XXXX","token_type":"bearer","refresh_token":"XXXX","expiry":"XXXX"}
Choose a number from below, or type in an existing value
1 / OneDrive Personal or Business
\ "onedrive"
2 / Root Sharepoint site
\ "sharepoint"
3 / Type in driveID
\ "driveid"
4 / Type in SiteID
\ "siteid"
5 / Search a Sharepoint site
\ "search"
Your choice> 1
Found 1 drives, please select the one you want to use:
1: OneDrive (business) id=b!laCd4ZJ54U-[...]
Chose drive to use:> 1
Found drive 'root' of type 'business', URL: https://ksuemailprod-my.sharepoint.com/personal/mozes_ksu_edu/Documents
Is that okay?
y) Yes (default)
n) No
y/n> y
--------------------
[myOneDrive]
type = onedrive
token = {"access_token": ...}
drive_id = b!laCd4ZJ54U-[...]
drive_type = business
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
myOneDrive onedrive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Now test the connection by running the ls command. You should see a listing of your OneDrive files.
List contents of OneDrive
[mozes@gremlin01 ~]$ rclone ls myOneDrive:/
- To upload or download files, use the rclone copy command. For example:
Transferring files
OnDemand
Once you have defined an endpoint, you can use that endpoint in the Files manager in OpenOnDemand.
First you'd select the endpoint and files/folder you would like to copy or move. Then you would click the copy/move button near the top.
The you would traverse to the location where you would like to copy/move the files/folders to and then you would click the copy or move button on the left.
This would trigger the file transfer to start. Once it was done, you would see the files available in the path that you requested them.
Command Line
[mozes@gremlin01 ~]$ rclone copy myOneDrive:/SomeFile.txt ./ [mozes@gremlin01 ~]$ rclone copy ./SomeFile.txt myOneDrive:/
- To download directories, use the rclone copy command and use directory names over file. This copies the contents of the folders, so you need to specify a destination folder.
Download a directory from OneDrive
[mozes@gremlin01 ~]$ rclone copy myOneDrive:/my_beocat_dir ./my_beocat_dir
To upload a directory named my_beocat_dir to OneDrive, use rclone copy.
Upload a directory to OneDrive
[mozes@gremlin01 ~]$ rclone copy ./my_beocat_dir myOneDrive:/my_beocat_dir
- Rclone also supports using sync to transfer files, similar to rsync. The syntax is similar to rclone copy. This would only transfer files that are updated by name, checksum, or time. The example below would sync the files of the local directory to the remote directory on OneDrive.
Sync changed files
[mozes@gremlin01 ~]$ rclone sync ./my_beocat_dir myOneDrive:/my_beocat_dir
Potential issues
If you've setup an rclone remote previously and it no longer works, it may be that rclone's connection to your cloud storage has expired.
You may need to follow the procedures to get a desktop ondemand session above, then reauthorize your connection:
rclone config reconnect $whatever_you_named_your_endpoint:
note the trailing :
