From Beocat
Jump to: navigation, search
(Initial create)
 
(No difference)

Latest revision as of 09:53, 22 November 2019

Background info

  • Uploading to Google Drive is limited to 750 Gigabytes/day. There's also a rate limit of 2 files/second
  • A single file is limited to 5 Terabytes

Getting Started

  • You need to have a Google account through ITS. Note that Beocat staff cannot create or reset passwords for ksu.edu Google accounts and have no control over this part of the process.
  1. Send an e-mail to est-request@ksu.edu stating that you need to have a ksu.edu Google account for archiving data on Google Drive.
  2. When this account request is approved, you will get an email from gsuite-noreply@google.com with instructions on how to activate your account. It's quite likely this email will end up in your Spam, Junk Mail, or Clutter folders. Follow the instructions in that email to activate your Google account. Note that passwords to your Google account are NOT synchronized with your eID.
  • Login to the FIONA (Flash I/O Network Appliance) node.
  1. While logged into Beocat, you can 'ssh fiona'
  2. Alternatively, you can also ssh using your favorite ssh client (the same one you use to get to Beocat) to fiona.beocat.ksu.edu
  3. The username is your eID username and the password is your eID password
  • While on the fiona, setup rclone
  1. Run the command 'rclone config'
    1. Create a (n)ew remote
    2. Give it a name. For my example, I'll use 'remotegdrive', but you can change this to whatever you want (just be sure to make this change throughout)
    3. The type of connection is "drive" (at the time of this writing, it is option #12, but this may change)
    4. You will be asked for Google Application Client Id and Google Application Client Secret. For these two prompts, just type Enter as you don't need to provide them
    5. Select '1' for full drive access
    6. For the next two prompts, which ask for the Root Folder ID and Service Account Credentials, you can type Enter at each prompt
    7. Type 'n' to skip the advanced configuration
    8. You are working on a remote machine, so choose 'n' for Auto Config
    9. You will be shown a link/URL that you need to copy and paste in a browser (keep your terminal running the rclone config open). Once you access that URL, you will be asked to log in. Log in using your the ksu.edu Google account username and password from above. You will then see a window that rclone wants to access your Google account. Click "allow".
    10. A code will show up on your web browser. You will need to copy this code
    11. Go back to your rclone setup and paste the verification code there
    12. Type 'n' to the prompt on setting this up as a team drive
    13. Type 'y' to confirm the setup
    14. You should now see that you have a 'remotegdrive' (or whatever you named it) remote
    15. Type 'q' to quit rclone setup

Using rclone

Copying from Beocat to Google Drive

Below is an example of how to copy a file to the remote Google Drive. Note that for the rclone copy command, you need to follow this convention: rclone copy <path_file_you_want_to_move> <remote_string_name>:<destination_directory_you_want_to_move_into>. Don't forget the colon character that is in between of the remote name and the destination path. The -v flag is optional. It gives you a verbose output that includes progression of the upload.

[user@fiona ~]$ touch movethis.txt # Create a test file to copy
[user@fiona ~]$ rclone copy movethis.txt remotegdrive:main -v
2018/10/17 11:34:41 INFO : movethis.txt: Copied (new)
2018/10/17 11:34:41 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 1.8s

Copying from Google Drive to Beocat

To copy from the remote to local, make sure the source and destination paths are correct

[user@fiona ~] rm movethis.txt # delete the file we just created above - not necessary if this is a new file
[user@fiona ~] rclone copy remotegdrive:movethis.txt . # Copy movethis.txt to current local directory

Advanced copies

  • To copy files from, say, your bulk space to the remote (here the remote name is remotegdrive; substitute to what you named yours):
[user@fiona ~]$ rclone copy /bulk/your-username/somefile remotegdrive:destination_path/ # Copy somefile to remote destination
  • To transfer large files in parallel, you can use the options shown below with rclone copy. For details, see the documentation at https://rclone.org/commands/rclone_copy/ .
    • --transfers int Number of file transfers to run in parallel. (default 4)
    • --checkers int Number of checkers to run in parallel. (default 8 )
    • --drive-chunk-size SizeSuffix Upload chunk size. Must a power of 2 >= 256k. (default 8M)
    • --drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8M)
    • Eg: rclone --transfers=32 --checkers=16 --drive-chunk-size=16384k --drive-upload-cutoff=16384k copy source:path remote:path


  • Viewing remote content

Of course, you can always use a web browser ( https://drive.google.com/ ) to view your files, but you can also use the rclone command to view files and directories.

List remote directories:

rclone lsd remotegdrive:

List all your files:

rclone ls remotegdrive:

List all the files on your remote 'mydirectory'

rclone ls remotegdrive:mydirectory


  • Forgot which remote you setup?

If you changed 'remotegdrive' to something else in the above example, but don't remember what you changed it to, you can view your remote setup name with

rclone listremotes

Acknowledgement

Many thanks to our friends at the University of Kentucky Center for Computational Sciences from whom we copied large swaths of setup instructions.