Texas Tech University

Transferring Data

Transferring files/data to and from HPCC resources can be done using the HPCC Globus Connect data transfer service, powered by Globus Connect. Many large research labs as well as computing centers at other universities also have their own Globus Connect server endpoints that you can use to move data directly between sites.

Note: Please refrain from using scp, sftp, direct ssh connections or any other data transfer tool to transfer data.  These take up considerable HPCC resources while also running slower than using Globus Connect.

 

Table of Contents:

  1. Transferring data between your computer and the HPCC
  2. Transferring data between sites
  3. Sharing data with collaborators

 

Transferring data between your personal computer and the HPCC Cluster

To prevent unnecessary load on the cluster head nodes, we ask that all users refrain from using scp, sftp, direct ssh connections or any other data transfer tool. These types of data transfers are likely to be killed by cleanup scripts or our staff if they become intrusive. Instead, we suggest that you use Globus Connect to transfer data into and out of the HPCC. There are several reasons to prefer this service:

  • The Globus Connect service is well connected to the campus network and therefore the outside world with a high-speed connection.
  • The machine that runs our Globus Connect endpoint is newer, more robust, and has more processors than the cluster login node.
  • The Globus Connect service eliminates the load of data transfer from the cluster login node, which is used by many people for other functions.
  • There are Globus Connect personal endpoint clients available for Linux, Mac or Windows.​

 

Transferring data between your computer and the HPCC can be done by making your personal computer a Globus Connect endpoint.  This requires you to install the Globus Connect Personal Endpoint software, which you can do by following one of the guides below:

 

Once you have a personal endpoint set up, you can now use the Globus Connect interface to transfer data between your computer and the HPCC. For detailed instructions on how to do this, please see the guide "How To Log In and Transfer Files with Globus" written by the Globus team. When going through these instructions please keep in mind the following:

  • To reach your data on the HPCC Lustre storage area (/home, /lustre/work, /lustre/scratch), set the endpoint to "ttuhpcc#TTUTerra"
  • To reach your data on your personal computer, set the endpoint to the name you selected when you created the endpoint.

Transferring data between sites

Many research organizations and universities have established Globus Connect endpoints, making the transfer data between these sites and the HPCC a fast and simple process. To prevent unnecessary load on the cluster head nodes, we ask that all users refrain from using scp, sftp, direct ssh connections or any other data transfer tool. These types of data transfers are likely to be killed by cleanup scripts or our staff if they become intrusive. Instead, we suggest that you use Globus Connect to transfer data into and out of the HPCC.  

The first step you will need to take is to determine the name of the endpoint used by the outside organization.  For example, TACC prefixes all of their endpoints with the name "tacc"Once you have a personal endpoint set up, you can now use the Globus Connect interface to transfer data between your computer and the HPCC. For detailed instructions on how to do this, please see the guide "How To Log In and Transfer Files with Globus" written by the Globus team. When going through these instructions please keep in mind the following:

  • To reach your data on the HPCC Lustre storage area (/home, /lustre/work, /lustre/scratch), set the endpoint to "ttuhpcc#TTUTerra"
  • To reach your data at the external your personal computer, set the endpoint to the name you selected when you created the endpoint.

Sharing data with collaborators

The Globus Connect interface can also be used to share data you have stored on HPCC resources with outside collaborators. For detailed instructions on how to do this, please see the guide "How To Share Data Using Globus" written by the Globus team.  When going through these instructions please keep in mind the following:

  • To reach your data on the HPCC Lustre storage area (/home, /lustre/work, /lustre/scratch) set the endpoint to "ttuhpcc#TTUTerra"
  • Be careful in who you give "write" permission to. You will be held responsible for anything uploaded to your account on your behalf.

 

High Performance Computing Center