HPCC Data Transfer
For data transfers to and from the HPCC clusters, please refrain from using scp, sftp, or direct ssh connections and instead use the HPCC data transfer service based on Globus Connect described below.
There are several reasons to prefer this service:
- The Globus Connect service is well connected to the campus network and therefore the outside world with a high-speed connection.
- The Globus Connect Server endpoint service at the HPCC eliminates the load of data transfer from the cluster login nodes, which are used by many people for other functions.
- There are Globus Connect Personal endpoint clients available for Linux, Mac or Windows.
- Many large research labs and computing centers at other universities also have their own Globus Connect server endpoints that you can use directly within the service.
- The university is in the process of extending the Globus Connect data transfer service to other centrally-managed resources on campus.
It's very easy to set things up for user-driven data transfer using the HPCC data transfer service. Here are the instructions to get started.
Step 1: Go to https://app.globus.org/endpoints, select "Texas Tech University" from the drop down list to sign in if necessary and then click on the "Continue" button. On the endpoints webpage, click on the "Create a Personal Endpoint" link and follow the instructions to download and install Globus Connect Personal on your local machine if you have not already done so.
Step 2 (optional): When you finish installing Globus Connect Personal, you can change the default settings, for example, you can change the local accessible folder to an ideal location for doing the data transfer in the future, for example, it could be "D:\download" on a Windows machine, or a locally selected folder or directory on a MacOS or Linux one.
Step 3: Start Globus Connect Personal on your local machine; then you can start the data transfer. If you want to transfer data from/to your Lustre storage area on Hrothgar, you currently need to enter "ttuhpcc#TTUTerra" as the other endpoint. You can then use the file directory navigation shown for accessing your /home, /lustre/work or /lustre/scratch folders.
Once you have done this, in the future you can simply connect again to the service on the web to use it again at http://globus.org.
More information about Globus Connect Personal endpoint installation and usage can be found at http://www.globus.org/globus-connect-personal.
Note: This is the only supported mechanism for moving large amounts of data to and from the HPCC machines! Extended data transfers such as scp, sftp, ssh, or other similar tools through the cluster login nodes by other means, either directly or though terminal connection programs, are subject to being terminated without warning, as such tools put undue load on the login nodes, can interfere with their normal usage and function, and are far less efficient than using the data transfer service. Instead, please use the recommended Globus Connect data transfer service for moving data to and from the HPCC.
If, after reviewing the above information, you would like additional assistance from HPCC staff in moving large or special-case data collections, please send an email with the following information to email@example.com
*Location of files/directories:
Comments: (optionally include any additional details)
(* = Mandatory fields)
Click here to review HPCC operational Policies.
Click here to see the HPCC Data Policy.