HPCC Data Transfer
For data transfers to and from the HPCC clusters, please refrain from using scp, sftp, or direct ssh connections and instead use the HPCC data transfer service based on Globus Connect described here.
Direct data transfers through the cluster login nodes are subject to being terminated without warning, as this puts undue load on the login nodes and is far less efficient than using the data transfer service. Instead, please use this service.
There are several reasons to prefer this service:
- The Globus Connect service is well connected to the campus network and therefore the outside world with a high-speed connection.
- The Globus Connect Server endpoint service at the HPCC eliminates the load of data transfer from the cluster login nodes, which is used by many people for other functions.
- There are Globus Connect Personal endpoint clients available for Linux, Mac or Windows.
- Many large research labs and computing centers at other universities also have their own Globus Connect server endpoints that you can use directly within the service.
- The university is in the process of extending the Globus Connect data transfer service to other centrally-managed resources on campus.
It's very easy to set things up for user-driven data transfer using the HPCC data transfer service. Here are the instructions to get started.
Step 1: Go to https://www.globus.org/app/endpoints/create-gcp, select "Texas Tech University" from the drop down list then click on the "Continue" button, next on the webpage follow the instruction to download and install Globus Connect Personal on your local machine.
Step 2 (optional): When you finish installing Globus Connect Personal, you can change the default settings, for example, you can change the local accessible folder to an ideal location for doing the data transfer in the future, for example, it could be "D:\download" on a Windows machine, or a locally selected folder or directory on a MacOS or Linux one.
Step 3: Start Globus Connect Personal on your local machine, then you can start the data transfer. If you want to transfer data from/to your Lustre storage area on Hrothgar, you currently need to enter "ttuhpcc#TTUTerra" as the endpoint for accessing your /home, /lustre/work or /lustre/scratch folder. We plan to add other transfer endpoints in the future and will publish a list.
Once you have done this, in the future you can simply connect again to the service on the web to use it again at http://globus.org.
More information about Globus Connect Personal endpoint installation and usage can be found at http://www.globus.org/globus-connect-personal.
If, after reviewing the above information, you would like additional assistance from HPCC staff in moving large or special-case data collections, please send an email with the following information to firstname.lastname@example.org
*Location of files/directories:
Comments: (optionally include any additional details)
(* = Mandatory fields)