hpcwiki - User contributions [en]

Remote Access

2021-12-22T13:47:04Z

Koen Mulderij: /* Linux */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection or when you want to use [[#Linux|X-forwarding]]
=== Command line ===
When your desktop/laptop supports ssh from the console:
[user@home]$ ssh -J <NetID>@linux-bastion.tudelft.nl <NetID>@hpcXX.tudelft.net
NetID@linux-bastion.tudelft.nl's password:
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

If your SSH-client doesn't support Jump hosts, you can first connect to the bastion server and then to the HPC:

[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===
Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in 
The easiest way is:
[user@home]$ ssh -J <NetID>@linux-bastion.tudelft.nl -X <NetID>@hpcXX.tudelft.net
NetID@linux-bastion.tudelft.nl's password:
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

When you have configured [[#SSH_Proxy_Support|SSH Proxy Support]] you can connect with:

ssh -X <NetID>@hpcXX

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX

Remote Access

2021-12-22T13:42:43Z

Koen Mulderij: /* Command line */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection or when you want to use [[#Linux|X-forwarding]]
=== Command line ===
When your desktop/laptop supports ssh from the console:
[user@home]$ ssh -J <NetID>@linux-bastion.tudelft.nl <NetID>@hpcXX.tudelft.net
NetID@linux-bastion.tudelft.nl's password:
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

If your SSH-client doesn't support Jump hosts, you can first connect to the bastion server and then to the HPC:

[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

For a grahpical connection to an HPC server you will first have to configure [[#SSH_Proxy_Support|SSH Proxy Support]] 
Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX

Remote Access

2021-11-16T13:10:25Z

Koen Mulderij: /* Access to the commandline */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection or when you want to use [[#Linux|X-forwarding]]
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

For a grahpical connection to an HPC server you will first have to configure [[#SSH_Proxy_Support|SSH Proxy Support]] 
Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX

Remote Access

2021-11-16T13:02:44Z

Koen Mulderij: /* Linux */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

For a grahpical connection to an HPC server you will first have to configure [[#SSH_Proxy_Support|SSH Proxy Support]] 
Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX

Remote Access

2021-11-16T13:02:20Z

Koen Mulderij: /* Linux */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

For a grahpical connection to an HPC server you will first have to configure [[#SSH_Proxy_Support|SSH_Proxy_Support]] 
Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX

Remote Access

2021-11-16T13:01:50Z

Koen Mulderij: /* Linux */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

For a grahpical connection to an HPC server you will first have to configure [[#SSH_Proxy_Support|SSH_Proxy_Support]]
Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX

Remote Access

2021-11-09T16:46:49Z

Koen Mulderij:

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Main Page

2021-10-18T10:11:13Z

Koen Mulderij:

Welcome to the hpcwiki!

This wiki is intended as a guide for the hpc clusters managed bij the ICT Department. If you have never used one of our clusters before, this is the place to start. Please note that this wiki is a work in progress, it may not yet be complete.

[[Introduction]]

[[How to log in|Access]] (How to log in and how to copy files)

[[Remote_Access|Remote Access]] (How to connect from outside the TU Delft)

[[The queue system]]

[[More about queues and nodes]]

[[How to run a job]] (Examples)

[[Applications]]

[[Software Environments]]

[[Further reading]] (Where to find more information)

Remote Access

2021-10-18T10:10:07Z

Koen Mulderij:

Starting from 9-Nov-2021 access to the HPC-clusters will be limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T17:12:16Z

Koen Mulderij: /* MobaXterm */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has built-in support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T17:11:45Z

Koen Mulderij: /* MobaXterm */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T17:11:30Z

Koen Mulderij: /* Windows */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Installation and use is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Configuration is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T17:10:18Z

Koen Mulderij:

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Installation and use is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

== How to display graphical applications ==
=== Windows ===
MobaXterm can also be used for displaying graphical applications. Installation and use is described in [[#MobaXterm|Access to the commandline -> MobaXterm]]
*In the commandline windows (right) Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T16:54:54Z

Koen Mulderij: /* MobaXterm */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

=== MobaXterm ===
MobaXterm can also be used for commandline access. Installation and use is described in [[#Windows|How to display graphical applications -> Windows]]

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Installation and use is described in [[#Windows|How to display graphical applications -> Windows]]

== How to display graphical applications ==
=== Windows ===
'''MobaXterm''' is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)
====Graphical====
*Right Window Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]
====File transfer====
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T16:54:34Z

Koen Mulderij: /* Access from the commandline */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access to the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

==== MobaXterm ====
MobaXterm can also be used for commandline access. Installation and use is described in [[#Windows|How to display graphical applications -> Windows]]

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Installation and use is described in [[#Windows|How to display graphical applications -> Windows]]

== How to display graphical applications ==
=== Windows ===
'''MobaXterm''' is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)
====Graphical====
*Right Window Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]
====File transfer====
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T16:35:51Z

Koen Mulderij: /* PuTTY */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Installation and use is described in [[#Windows|How to display graphical applications -> Windows]]

== How to display graphical applications ==
=== Windows ===
'''MobaXterm''' is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)
====Graphical====
*Right Window Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]
====File transfer====
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T16:34:49Z

Koen Mulderij: /* Data on HPC */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)

==== MobaXterm ====
MobaXterm can also be used for file transfers. Installation and use is described in [[#Windows|How to display graphical applications -> Windows]]

== How to display graphical applications ==
=== Windows ===
'''MobaXterm''' is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)
====Graphical====
*Right Window Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]
====File transfer====
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T16:29:38Z

Koen Mulderij: /* How to display graphical applications */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==
=== Windows ===
'''MobaXterm''' is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)
====Graphical====
*Right Window Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]
====File transfer====
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T16:26:17Z

Koen Mulderij: /* Windows MobaXterm */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==
=== Windows MobaXterm ===
MobaXterm is a very versatile application. It can be used to connect to the commandline on HPC, display graphical applications and transfer files. It also has build support for using SSH-tunneling to connect through a bastion. You can download it from [https://mobaxterm.mobatek.net/ https://mobaxterm.mobatek.net/]. The '''free version''' has enough features for most users. The '''Portable edition''' can be run without installing (just unzip).
*Click the '''Session''' icon [[File:MobaXterm_NewSession.png]]
*Click the '''SSH''' icon [[File:MobaXterm_NewSSH.png]]
*Enter '''Remote host''' Check '''Specify username''' Enter your NetID [[File:MobaXterm_Hostname.png]]
*tab: Network settings Click '''SSH gateway (jump host)''' [[File:MobaXterm_NewJumphost.png]]
*Enter linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Enter your NetID Click '''OK''' [[File:MobaXterm_Jumphost.png]]
*Click '''OK''' in the session settings to start connecting You will be asked for your password Enter your password on the console of hpcXX You may be presented with a question about accepting the server's host key (this is normal the first time)
====Graphical====
*Right Window Test X-forwarding (Graphics) by running <code>xeyes</code> You should see these eyes (following the cursor) [[File:MobaXterm_Xeyes.png]]
====File transfer====
*Left Window Download files with '''arrow down''' Upload files with '''arrow up''' [[File:MobaXterm_SFTP.png]]

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

File:MobaXterm SFTP.png

2021-10-12T15:44:28Z

Koen Mulderij:

File:MobaXterm Xeyes.png

2021-10-12T15:40:43Z

Koen Mulderij:

File:MobaXterm Jumphost.png

2021-10-12T15:35:58Z

Koen Mulderij:

File:MobaXterm NewJumphost.png

2021-10-12T15:35:30Z

Koen Mulderij: Koen Mulderij uploaded a new version of "File:MobaXterm NewJumphost.png"

File:MobaXterm NewJumphost.png

2021-10-12T15:34:19Z

Koen Mulderij:

File:MobaXterm Hostname.png

2021-10-12T15:33:22Z

Koen Mulderij:

File:MobaXterm NewSSH.png

2021-10-12T15:31:42Z

Koen Mulderij:

File:MobaXterm NewSession.png

2021-10-12T15:31:02Z

Koen Mulderij:

Remote Access

2021-10-12T14:34:12Z

Koen Mulderij: /* How to display graphical applications */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==
=== Windows MobaXterm ===
=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T14:32:36Z

Koen Mulderij: /* Command line */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T14:30:24Z

Koen Mulderij:

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T10:54:53Z

Koen Mulderij: /* Command line */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.

=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T10:53:22Z

Koen Mulderij:

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Access from the commandline ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T09:17:28Z

Koen Mulderij: /* Mac OS X */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Console access ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T09:17:11Z

Koen Mulderij: /* Linux */

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Console access ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpc10.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpcXX.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpcXX.tudelft.net

Remote Access

2021-10-12T09:11:56Z

Koen Mulderij:

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Console access ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <NetID>@linux-bastion.tudelft.nl
NetID@linux-bastion.tudelft.nl's password:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
*Replace <NetID> with your NetID (also remove the <>)
*When your a student replace <NetID>@linux-bastion.tudelft.nl with <NetID>@student-linux.tudelft.nl
*Replace <NetID>@hpcXX.tudelft.net with the HPC you are connecting to.
=== PuTTY ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html PuTTY] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel traffic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[NetID@bastion]$ ssh <NetID>@hpcXX.tudelft.net
NetID@hpcXX.tudelft.net's password:
Welcome
[NetID@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <NetID>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
NetID@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <NetID>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<NetID>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your NetID. 
For example, if your name would be John Smith and your NetID jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your NetID. You can also substitute <code>$USER</code> for your NetID yourself. If your NetID would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own NetID! 
'''If you use anthing other than your own NetID you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your NetID is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your NetID is pbrown. 
If both NetIDs have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <NetID>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <NetID>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <NetID>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <NetID>@hpcXX:~/

==== FileZilla ====
For FileZilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#PuTTY|PuTTY]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in PuTTY portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From PuTTY on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <NetID>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpc10.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <NetID>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <NetID>@hpc10.tudelft.net

Remote Access

2021-10-11T16:55:43Z

Koen Mulderij: Koen Mulderij moved page RemoteAccess to Remote Access without leaving a redirect

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Console access ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <netid>@linux-bastion.tudelft.nl
netid@linux-bastion.tudelft.nl's password:
[netid@bastion]$ ssh <netid>@hpcXX.tudelft.net
netid@hpcXX.tudelft.net's password:
Welcome
[netid@hpcXX]$
*Replace <netid> with your NetID (also remove the <>)
*When your a student replace <netid>@linux-bastion.tudelft.nl with <netid>@student-linux.tudelft.nl
*Replace <netid>@hpcXX.tudelft.net with the HPC you are connecting to.
=== Putty ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html Putty] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel trafic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[netid@bastion]$ ssh <netid>@hpcXX.tudelft.net
netid@hpcXX.tudelft.net's password:
Welcome
[netid@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <netid>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <netid>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
netid@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <netid>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<netid>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your netid. 
For example, if your name would be John Smith and your netid jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your netid. You can also substitute <code>$USER</code> for your netid yourself. If your netid would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own netid! 
'''If you use anthing other than your own netid you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your netid is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your netid is pbrown. 
If both netids have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <netid>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <netid>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <netid>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <netid>@hpcXX:~/

==== Filezilla ====
For Filezilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#Putty|Putty]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in Putty portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From putty on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <netid>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <netid>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <netid>@hpc10.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <netid>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <netid>@hpc10.tudelft.net

Remote Access

2021-10-11T16:48:06Z

Koen Mulderij:

Access to the HPC-clusters is limited from outside the TU Delft network for security reasons. 
When you use a [https://www.tudelft.nl/ict-handleidingen/vpn VPN-connection] direct access from your desktop/laptop to the HPC is possible. You can use your application of choice for this connection. 
When you don't use a VPN you should connect through a [[#Bastion|Bastion server]], which is described below. 

== Bastion ==
The TU Delft offers two bastion servers. These servers can be used to tunnel your traffic to the HPC.
*linux-bastion.tudelft.nl for employees or guests.
*student-linux.tudelft.nl for students.

== Console access ==
The explanations below are the simplest way to connect to a console on an HPC. [[#Advanced configs|Advanced configs]] describes more complex configurations which save time when regularly using the connection.
=== Command line ===
When your desktop/laptop supports ssh from the console, you can connect to the bastion server and from there you can connect to the HPC:
[user@home]$ ssh <netid>@linux-bastion.tudelft.nl
netid@linux-bastion.tudelft.nl's password:
[netid@bastion]$ ssh <netid>@hpcXX.tudelft.net
netid@hpcXX.tudelft.net's password:
Welcome
[netid@hpcXX]$
*Replace <netid> with your NetID (also remove the <>)
*When your a student replace <netid>@linux-bastion.tudelft.nl with <netid>@student-linux.tudelft.nl
*Replace <netid>@hpcXX.tudelft.net with the HPC you are connecting to.
=== Putty ===
When your desktop/laptop doesn't support ssh from the console, you can use [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html Putty] (it may already be installed). 
*Session Settings Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students), and select the SSH protocol (port 22): [[File:PuTTY-Hostname.png]]
*OPTIONAL: Connection>SSH Settings Select Share SSH connections if possible (this means that you will not have to authenticate when you start a second session): [[File:PuTTY-ShareConnection.png]]
*OPTIONAL: Connection>SSH>Tunnels Settings (This allows other applications to tunnel trafic through the bastion host) [[File:PuTTY-Tunnels.png]]
**Source Port: 1080
**Dynamic & Auto
**Press "Add"
*Session Settings To store these session settings, go back to the session tab, and save the settings under a session name. To activate a stored session, double-click on the session name: [[File:PuTTY-SaveSession.png]]
Once you're connected to the bastion server you can use the locally available ssh to connect to a HPC:
[netid@bastion]$ ssh <netid>@hpcXX.tudelft.net
netid@hpcXX.tudelft.net's password:
Welcome
[netid@hpcXX]$
=== Advanced configs ===
==== Making OpenSSH more user-friendly ====
The OpenSSH clients can be configured using a configuration file in ~/.ssh/config (Linux, macOS) or ~\.ssh\config (Windows).

===== Define connections =====
For the OpenSSH clients you can predefine often used connections by placing a section like the following in the configuration file on your local computer. Authentication on the bastion with keys is not possible, so password is used. 
Employees:
Host bastion
Hostname linux-bastion.tudelft.nl
User <netid>
PreferredAuthentications password

Students:
Host bastion
Hostname linux-bastion.tudelft.nl
User <netid>
PreferredAuthentications password

You can then connect simply by ssh bastion. For example:
[user@home]$ ssh bastion
netid@linux-bastion.tudelft.nl's password:

===== SSH Proxy Support =====
Since this configuration depends on [[#Define connections|Define connections]] configure that first. 
To connect directly from your machine to an HPC cluster login node (without connecting to the bastion server first), create a connection via a proxy by adding the following lines to the configuration file ~/.ssh/config on your local computer:

Host hpcXX
Hostname hpcXX.tudelft.net
ProxyCommand ssh -W %h:%p bastion
User <netid>

You can then simply use ssh hpcXX (or scp -p '<local file>' "hpcXX:/scratch/<netid>/") from your local computer.

===== Session multiplexing =====
OpenSSH clients support multiple sessions over a single connection. This means you have to enter your password only once, to make the connection, and can then open multiple sessions (SSH, SCP and SFTP) to that computer over the same connection. This is especially convenient when connecting to the bastion server, where you have to type your password always.

Note: SSH public key logins won't work (reliably, (because Kerberos authentication is required to access your home directory).

Add the following section to the end of the configuration file on your local computer and on the bastion server:

Host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

Windows users may need to adapt the ControlPath location to match Windows.

You can use the exact same commands as above.

== File Transfer ==
The best method for transfering files depends on where you want to store your files:
#Files in your home directory on HPC are only available within HPC
#Files on Network Shares are available on other locations (such as a TU Delft workplace, weblogin.tudelft.nl or other HPC clusters) Copying files to these shares is best done using the manuals on [https://webdata.tudelft.nl/ webdata.tudelft.nl]

When you want to copy files to the HPC you can copy them from Network Shares
=== (Windows) Network Shares ===

==== Authenticate ====
It is possible to access the group and bulk shares of your research group. It is required that you have logged in using your password. If you logged in with an ssh public/private keypair instead of your password, you need to authenticate first with:
kinit
This will ask you to type your password. This authentication is valid for about eight hours, after that you have to authenticate again.

==== Basic access ====

Now just type (depending on which share you want to use):
cd /tudelft/$USER/staff-group/
cd /tudelft/$USER/student-group/
cd /tudelft/$USER/staff-bulk/
cd /tudelft/$USER/staff-umbrella/

and navigate from there to the directories that you need.
You can access your Windows home directory as well. The subdirectories of the home directories are followed by '''the first letter of your surname''' and then your netid. 
For example, if your name would be John Smith and your netid jsmith;

cd /tudelft/$USER/staff-homes/s/jsmith
cd /tudelft/$USER/student-homes/s/jsmith

==== Important note ====

The <code>$USER</code> part in the commands above is a shell variable that contains your netid. You can also substitute <code>$USER</code> for your netid yourself. If your netid would be <code>jsmith</code>, the command

cd /tudelft/jsmith/staff-homes/s/jsmith

would work just as well. Please do not substitute <code>$USER</code> with anything else than your own netid! 
'''If you use anthing other than your own netid you'll create a recipe for potential disaster!''' 

If you and a coworker need to access te same data on a group or bulk volume at the same time, you both use different paths. 
So <code>/tudelft/jsmith/staff-group/</code> only works if your netid is jsmith and <code>/tudelft/pbrown/staff-group/</code> only works works if your netid is pbrown. 
If both netids have the same privileges for accessing the group volumes, then these commands give access to the same data.

=== Data on HPC ===
If you want to transfer data directly to the HPC you will need some extra configuration
==== Command line ====
Please first configure [[#SSH Proxy Support|SSH Proxy Support]]. 
To transfer files you can use scp, which is very similar to ssh 

*Copy the file 'projects.txt' from your remote home directory on HPC to the local home directory on your desktop or laptop.
scp <netid>@hpcXX.tudelft.net:~/projects.txt ~

*Copy an entire directory with all its content, use the '-r' option This will copy the directory 'projects' from your remote home directory on HPC to the local home directory on your desktop or laptop:
scp -r <netid>@hpcXX.tudelft.net:~/projects/ ~

*Copy the file 'projects.txt' from your local home directory on your desktop or laptop to your remote home directory on HPC.
scp projects.txt <netid>@hpcXX:~/

*Copy the directory 'projects' and all its content from your current working directory on your desktop or laptop to your remote home directory on HPC.
scp -r projects/ <netid>@hpcXX:~/

==== Filezilla ====
For Filezilla to connect to HPC you will first need to '''configure and start''' a session to the bastion with [[#Putty|Putty]]. Make sure you have configured SSH portforwarding (Connection>SSH>Tunnels Settings)
*Edit > Settings
[[File:FileZilla01.png]]
*Connection > Generic Proxy Type: Socks 5 Host: localhost Port: 1080 (if you configured this in Putty portforwarding) Exit with "OK"
[[File:FileZilla02.png]]
*File > Site Manager
[[File:FileZilla03.png]]
*Click "New Site" Name & Host: linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Protocol: SFTP Port: 22 Logon Type: Interactive (or "Normal")
[[File:FileZilla04.png]]
*You can now "Connect" You may be presented with a question about accepting the server's host key (this is normal the first time)

==== WinSCP ====
WinSCP is a simple tool that has the internal option to create a tunnel through the bastion server. It can be downloaded from [https://winscp.net/eng/download.php https://winscp.net/eng/download.php]
*Session > New Session
[[File:WinSCP01.png]]
*Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "Advanced" button
[[File:WinSCP02.png]]
*Connection > Tunnel Select "Connect through SSH tunnel" Fill in the name of the TU Delft SSH server linux-bastion.tudelft.nl (student-linux.tudelft.nl for students) Fill in your NetID Click the "OK" button
[[File:WinSCP03.png]]
*Click "Save"
[[File:WinSCP02.png]]
*Save session as site You can accept the suggested name
[[File:WinSCP04.png]]
*You can now "Login" You will be asked to supply your password twice: first for the bastion, second for the HPC You may be presented with a question about accepting the server's host key (this is normal the first time)
== How to display graphical applications ==

=== Windows ===

There are several options for displaying graphical applications on a Windows system. A basic option is [http://www.straightrunning.com/XmingNotes/ Xming]. You can download it from: http://www.straightrunning.com/XmingNotes/ The site asks for a donation to use the latest version, but there is also a public domain version which can be used for free. Once you have installed it, you can find it in the usual place in the start menu. Before you log in to an hpc cluster and run graphical applications, you need to start Xming first. It will not show much, only a small X in the system tray once it's running. You also need to enable X forwarding in PuTTY for your connection to the bastion.
*Connection>SSH>X11 Check "Enable X11 Forwarding"
[[File:PuTTY_04.png]]

From putty on the bastion you can continue logging in to the hpc cluster, using <code>-X</code> to enable X-forwarding in that session.

ssh -X <netid>@hpcXX.tudelft.net

To check if X forwarding works, type <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that will follow the mouse pointer.
[[File:Xeyes.png]]

An alternative option for displaying graphical applications is Exceed, this is a commercial package for which the TU Delft has a license. It is not available by default on the standard Windows systems, but it can be installed upon request if you contact the servicedesk.

A second alternative is [https://mobaxterm.mobatek.net/ MobaXterm], this is an all-in-one solution containing an ssh client, X server and more. You can download a free version from: https://mobaxterm.mobatek.net/download.html The free version has some limitations, but it's still perfectly usable in our environment.

=== Mac OS X ===

For displaying graphical applications on a Mac OS X system you need a software package called [https://www.xquartz.org/ XQuartz] which can be downloaded from: [https://www.xquartz.org/ https://www.xquartz.org/] This will run on Mac OS X v10.6 and higher. Once it's installed, you end up with an application called XQuartz located in /Applications/Utilties. There is no need to start it first before you log in to an hpc cluster, it will start automatically as soon as it is needed. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <netid>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <netid>@hpc10.tudelft.net

=== Linux ===

Most Linux workstations come with a graphical environment. This works transparantly with the graphical applications on the hpc clusters. The only thing you need to do is add the option <code>-X</code> to the <code>ssh</code> command that you use to log in:

ssh -X <netid>@hpc10.tudelft.net

You can test if this works by typing <code>xeyes</code> on the command line, this will show a small window on your screen with two eyes that follow the mouse pointer. If it doesn't work, try the <code>-Y</code> option:

ssh -Y <netid>@hpc10.tudelft.net

Remote Access

2021-10-11T16:15:30Z

Koen Mulderij: /* WinSCP */

Remote Access

2021-10-11T16:14:53Z

Koen Mulderij: /* Filezilla */

File:FileZilla04.png

2021-10-11T16:06:19Z

Koen Mulderij: Koen Mulderij uploaded a new version of "File:FileZilla04.png"

File:FileZilla04.png

2021-10-11T15:54:58Z

Koen Mulderij:

File:FileZilla03.png

2021-10-11T15:54:49Z

Koen Mulderij:

File:FileZilla02.png

2021-10-11T15:54:40Z

Koen Mulderij:

File:FileZilla01.png

2021-10-11T15:54:22Z

Koen Mulderij:

Remote Access

2021-10-11T15:04:26Z

Koen Mulderij: /* WinSCP */

Remote Access

2021-10-11T12:26:16Z

Koen Mulderij: /* WinSCP */

File:WinSCP04.png

2021-10-11T10:25:36Z

Koen Mulderij:

File:WinSCP03.png

2021-10-11T10:25:19Z

Koen Mulderij:

File:WinSCP02.png

2021-10-11T10:25:03Z

Koen Mulderij:

File:WinSCP01.png

2021-10-11T10:21:32Z

Koen Mulderij: