Remote Access

From hpcwiki
Revision as of 17:55, 11 October 2021 by Koen Mulderij (talk | contribs) (Koen Mulderij moved page RemoteAccess to Remote Access without leaving a redirect)
Jump to navigation Jump to search

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons.
When you use a VPN-connection direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection.
When you don't use a VPN you should connect through a Bastion server, which is described below.


Bastion

The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.

  • linux-bastion.tudelft.nl for employees or guests.
  • student-linux.tudelft.nl for students.

Console access

The explanations below are the simplest way to connect to a console on an HPC. Advanced configs describes more complex configurations which save time when regularly using the connection.

Command line

When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC: 

  [user@home]$ ssh <netid>@linux-bastion.tudelft.nl
    netid@linux-bastion.tudelft.nl's password:
  [netid@bastion]$ ssh <netid>@hpcXX.tudelft.net
    netid@hpcXX.tudelft.net's password: 
    Welcome
  [netid@hpcXX]$
  • Replace <netid> with your NetID (also remove the <>)
  • When your a student replace <netid>@linux-bastion.tudelft.nl with <netid>@student-linux.tudelft.nl
  • Replace <netid>@hpcXX.tudelft.net with the HPC you are connecting to.

Putty

When your desktop/laptop doesn't support ssh from the console, you can use Putty (it may already be installed).

  • Session Settings
    Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22):
    PuTTY-Hostname.png
  • OPTIONAL: Connection>SSH Settings
    Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session):
    PuTTY-ShareConnection.png
  • OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel trafic through the bastion host)
    PuTTY-Tunnels.png
    • Source Port: 1080
    • Dynamic & Auto
    • Press "Add"
  • Session Settings
    To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name:
    PuTTY-SaveSession.png

Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC: 

  [netid@bastion]$ ssh <netid>@hpcXX.tudelft.net
    netid@hpcXX.tudelft.net's password: 
    Welcome
  [netid@hpcXX]$

Advanced configs

Making OpenSSH more user-friendly

The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

Define connections

For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used.
Employees: 

 Host bastion
 Hostname linux-bastion.tudelft.nl
 User <netid>
 PreferredAuthentications password

Students: 

 Host bastion
 Hostname linux-bastion.tudelft.nl
 User <netid>
 PreferredAuthentications password

You can then connect simply by ssh bastion. For example: 

 [user@home]$ ssh bastion
   netid@linux-bastion.tudelft.nl's password:
SSH Proxy Support

Since this configuration depends on Define connections configure that first.
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer: 

 Host hpcXX
 Hostname hpcXX.tudelft.net
 ProxyCommand ssh -W %h:%p bastion
 User <netid>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<netid>/") from your local computer.

Session multiplexing

OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server: 

 Host *
 ControlMaster auto
 ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

File Transfer

The best method for transfering files depends on where you want to store your files:

  1. Files in your home directory on HPC are only available within HPC
  2. Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters)
    Copying files to these shares is best done using the manuals on webdata.tudelft.nl

When you want to copy files to the HPC you can copy them from Network Shares

(Windows) Network Shares

Authenticate

It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with: 

kinit

This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

Basic access

Now just type (depending on which share you want to use): 

cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need. You can access your Windows home directory as well. The subdirectories of the home directories are followed by the first letter of your surname and then your netid.
For example, if your name would be John Smith and your netid jsmith; 

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

Important note

The $USER part in the commands above is a shell variable that contains your netid. You can also substitute $USER for your netid yourself. If your netid would be jsmith, the command 

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute $USER with anything else than your own netid!
If you use anthing other than your own netid you'll create a recipe for potential disaster!

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths.
So /tudelft/jsmith/staff-group/ only works if your netid is jsmith and /tudelft/pbrown/staff-group/ only works works if your netid is pbrown.
If both netids have the same privileges for accessing the group volumes, then these commands give access to the same data.

Data on HPC

If you want to transfer data directly to the HPC you will need some extra configuration

Command line

Please first configure SSH Proxy Support.
To transfer files you can use scp, which is very similar to ssh

  • Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <netid>@hpcXX.tudelft.net:~/projects.txt ~
  • Copy an entire directory with all its content, use the '-r' option
    This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <netid>@hpcXX.tudelft.net:~/projects/ ~
  • Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <netid>@hpcXX:~/
  • Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <netid>@hpcXX:~/

Filezilla

For Filezilla to connect to HPC you will first need to configure and start a session to the bastion with Putty. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)

  • Edit > Settings

FileZilla01.png

  • Connection > Generic Proxy
    Type: Socks 5
    Host: localhost
    Port: 1080 (if you configured this in Putty portforwarding)
    Exit with "OK"

FileZilla02.png

  • File > Site Manager

FileZilla03.png

  • Click "New Site"
    Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students)
    Protocol: SFTP
    Port: 22
    Logon Type: Interactive (or "Normal")

FileZilla04.png

  • You can now "Connect"
    You may be presented with a question about accepting the server's host key (this is normal the first time)

WinSCP

WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from https://winscp.net/eng/download.php

  • Session > New Session

WinSCP01.png

  • Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students)
    Fill in your NetID
    Click the "Advanced" button

WinSCP02.png

  • Connection > Tunnel
    Select "Connect through SSH tunnel"
    Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students)
    Fill in your NetID
    Click the "OK" button

WinSCP03.png

  • Click "Save"

WinSCP02.png

  • Save session as site
    You can accept the suggested name

WinSCP04.png

  • You can now "Login"
    You will be asked to supply your password twice: first for the bastion, second for the HPC
    You may be presented with a question about accepting the server's host key (this is normal the first time)

How to display graphical applications

Windows

There are several options for displaying graphical applications on a Windows system. A basic option is Xming. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu.
Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running.
You also need to enable X forwarding in PuTTY for your connection to the bastion.

  • Connection>SSH>X11
    Check "Enable X11 Forwarding"

PuTTY 04.png

From putty on the bastion you can continue logging in to the hpc cluster, using -X to enable X-forwarding in that session. 

 ssh -X <netid>@hpcXX.tudelft.net

To check if X forwarding works, type xeyes on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer. Xeyes.png


An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is MobaXterm, this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

Mac OS X

For displaying graphical applications on a Mac OS X system you need a software package called XQuartz which can be downloaded from: https://www.xquartz.org/ This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option -X to the ssh command that you use to log in: 

ssh -X <netid>@hpc10.tudelft.net

You can test if this works by typing xeyes on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the -Y option: 

ssh -Y <netid>@hpc10.tudelft.net

Linux

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option -X to the ssh command that you use to log in: 

ssh -X <netid>@hpc10.tudelft.net

You can test if this works by typing xeyes on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the -Y option: 

ssh -Y <netid>@hpc10.tudelft.net
Retrieved from "https://hpcwiki.tudelft.nl/index.php?title=Remote_Access&oldid=223"