Remote Access

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons.
When you use a VPN-connection direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection.
When you don't use a VPN you should connect through a Bastion server, which is described below.

Bastion

The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.

• linux-bastion.tudelft.nl for employees or guests.
• student-linux.tudelft.nl for students.

Access from the commandline

The explanations below are the simplest way to connect to a console on an HPC. Advanced configs describes more complex configurations which save time when regularly using the connection.

Command line

When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:

[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
  NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
  NetID@hpcXX.tudelft.net's password: 
  Welcome
[NetID@hpcXX]$

• Replace <NetID> with your NetID (also remove the <>)
• When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
• Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

Advanced configs

Making OpenSSH more user-friendly

The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

Define connections

For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used.
Employees:

 Host bastion
 Hostname linux-bastion.tudelft.nl
 User <NetID>
 PreferredAuthentications password

Students:

 Host bastion
 Hostname linux-bastion.tudelft.nl
 User <NetID>
 PreferredAuthentications password

You can then connect simply by ssh bastion. For example:

 [user@home]$ ssh bastion
   NetID@linux-bastion.tudelft.nl's password:

SSH Proxy Support

Since this configuration depends on Define connections configure that first.
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

 Host hpcXX
 Hostname hpcXX.tudelft.net
 ProxyCommand ssh -W %h:%p bastion
 User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

Session multiplexing

OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

 Host *
 ControlMaster auto
 ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

PuTTY

When your desktop/laptop doesn't support ssh from the console, you can use PuTTY (it may already be installed).

• Session Settings
  Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22):
  
• OPTIONAL: Connection>SSH Settings
  Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session):
  
• OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host)
  
  • Source Port: 1080
  • Dynamic & Auto
  • Press "Add"
• Session Settings
  To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name:
  

Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:

  [NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
    NetID@hpcXX.tudelft.net's password: 
    Welcome
  [NetID@hpcXX]$

File Transfer

The best method for transfering files depends on where you want to store your files:

1. Files in your home directory on HPC are only available within HPC
2. Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters)
   Copying files to these shares is best done using the manuals on webdata.tudelft.nl

When you want to copy files to the HPC you can copy them from Network Shares

(Windows) Network Shares

Authenticate

It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:

kinit

This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

Basic access

Now just type (depending on which share you want to use):

cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need. You can access your Windows home directory as well. The subdirectories of the home directories are followed by the first letter of your surname and then your NetID.
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

Important note

The $USER part in the commands above is a shell variable that contains your NetID. You can also substitute $USER for your NetID yourself. If your NetID would be jsmith, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute $USER with anything else than your own NetID!
If you use anthing other than your own NetID you'll create a recipe for potential disaster!

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths.
So /tudelft/jsmith/staff-group/ only works if your NetID is jsmith and /tudelft/pbrown/staff-group/ only works works if your NetID is pbrown.
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

Data on HPC

If you want to transfer data directly to the HPC you will need some extra configuration

Command line

Please first configure SSH Proxy Support.
To transfer files you can use scp, which is very similar to ssh

• Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.

scp <NetID>@hpcXX:~/projects.txt ~

• Copy an entire directory with all its content, use the '-r' option
  This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:

scp -r <NetID>@hpcXX:~/projects/ ~

• Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.

scp projects.txt <NetID>@hpcXX:~/

• Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.

scp -r projects/ <NetID>@hpcXX:~/

WinSCP

WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from https://winscp.net/eng/download.php

• Session > New Session

• Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students)
  Fill in your NetID
  Click the "Advanced" button

• Connection > Tunnel
  Select "Connect through SSH tunnel"
  Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students)
  Fill in your NetID
  Click the "OK" button

• Click "Save"

• Save session as site
  You can accept the suggested name

• You can now "Login"
  You will be asked to supply your password twice: first for the bastion, second for the HPC
  You may be presented with a question about accepting the server's host key (this is normal the first time)

MobaXterm

MobaXterm can also be used for file transfers. Installation and use is described in How to display graphical applications -> Windows

How to display graphical applications

Windows

MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from https://mobaxterm.mobatek.net/. The free version has enough features for most users. The Portable edition can be run without installing (just unzip).

• Click the Session icon
  
• Click the SSH icon
  
• Enter Remote host
  Check Specify username
  Enter your NetID
  
• tab: Network settings
  Click SSH gateway (jump host)
  
• Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students)
  Enter your NetID
  Click OK
  
• Click OK in the session settings to start connecting
  You will be asked for your password
  Enter your password on the console of hpcXX
  You may be presented with a question about accepting the server's host key (this is normal the first time)

Graphical

• Right Window
  Test X-forwarding (Graphics) by running xeyes
  You should see these eyes (following the cursor)
  

File transfer

• Left Window
  Download files with arrow down
  Upload files with arrow up
  

Mac OS X

For displaying graphical applications on a Mac OS X system you need a software package called XQuartz which can be downloaded from: https://www.xquartz.org/ This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option -X to the ssh command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing xeyes on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the -Y option:

ssh -Y <NetID>@hpcXX.tudelft.net

Linux

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option -X to the ssh command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing xeyes on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the -Y option:

ssh -Y <NetID>@hpcXX.tudelft.net