More about queues and nodes - Revision history

Frank Everdij: /* MPI jobs */

2022-06-13T11:27:26Z

MPI jobs

← Older revision		Revision as of 13:27, 13 June 2022
Line 164:		Line 164:
	module load mpi/openmpi-1.8.8-gnu		module load mpi/openmpi-1.8.8-gnu
	cd $PBS_O_WORKDIR		cd $PBS_O_WORKDIR
	mpirun whatever_workload_there_is		mpirun -n $PBS_NP whatever_workload_there_is
	</pre>		</pre>

Paul Keekstra: /* MPI jobs */

2020-07-01T10:11:31Z

MPI jobs

← Older revision		Revision as of 12:11, 1 July 2020
Line 175:		Line 175:
	Do not enter a passphrase, just press the enter key three times.		Do not enter a passphrase, just press the enter key three times.

	Next type these two commands:		Next type (or copy/paste) these two commands:

	cat ${HOME}/.ssh/id_rsa.pub >> ${HOME}/.ssh/authorized_keys		cat ${HOME}/.ssh/id_rsa.pub >> ${HOME}/.ssh/authorized_keys
	chmod go-rwx ${HOME}/.ssh/authorized_keys		chmod go-rwx ${HOME}/.ssh/authorized_keys

	And finally type (or copy/paste) these ~~three~~ lines:		And finally type (or copy/paste) these two lines:

	HPC=$(hostname \| cut -c 4-5) ; \		HPC=$(hostname \| cut -c 4-5) ; \

Paul Keekstra: /* MPI jobs */

2020-07-01T10:11:02Z

MPI jobs

← Older revision		Revision as of 12:11, 1 July 2020
Line 173:		Line 173:
	ssh-keygen		ssh-keygen

	Do not enter a passphrase, just press the enter key three times. Next type these two commands:		Do not enter a passphrase, just press the enter key three times.

			Next type these two commands:

	cat ${HOME}/.ssh/id_rsa.pub >> ${HOME}/.ssh/authorized_keys		cat ${HOME}/.ssh/id_rsa.pub >> ${HOME}/.ssh/authorized_keys

Paul Keekstra: /* MPI jobs */

2020-07-01T10:10:04Z

MPI jobs

← Older revision		Revision as of 12:10, 1 July 2020
Line 153:		Line 153:

	== MPI jobs ==		== MPI jobs ==

			Some workloads need OpenMPI to run, typically on two or more nodes at once. For such a job your job script usually contains a line like this:

			module load mpi/openmpi-1.8.8-gnu

			And your actual workload would start with <code>mpirun</code> or <code>mpiexec</code> like this:

			<pre>
			#PBS -l nodes=2:ppn=20
			module load mpi/openmpi-1.8.8-gnu
			cd $PBS_O_WORKDIR
			mpirun whatever_workload_there_is
			</pre>

			OpenMPI uses rsh or ssh under water to communicate between the assigned nodes, in some cases this leads to <code>Host key verification failed</code> errors and a premature termination of your job. To prevent this, you need to prepare a few files in your home directory. You only have to do this once on the master node.

			First of all, if you have never done this before on the master node, generate an ssh private/public keypair:

			ssh-keygen

			Do not enter a passphrase, just press the enter key three times. Next type these two commands:

			cat ${HOME}/.ssh/id_rsa.pub >> ${HOME}/.ssh/authorized_keys
			chmod go-rwx ${HOME}/.ssh/authorized_keys

			And finally type (or copy/paste) these three lines:

			HPC=$(hostname \| cut -c 4-5) ; \
			printf "host n${HPC}-* hpc${HPC}*\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n\tLogLevel QUIET\n" >> ${HOME}/.ssh/config

			The first line will give you a temporary <code> > </code> prompt, this is normal behaviour.

	== A generic example of a job script ==		== A generic example of a job script ==

Paul Keekstra: /* Access to nodes */

2020-07-01T09:22:20Z

Access to nodes

← Older revision		Revision as of 11:22, 1 July 2020
Line 151:		Line 151:

	It is important to know that an interactive can only be run in the normal queues, '''you can not run an interactive job in the guest queue!'''		It is important to know that an interactive can only be run in the normal queues, '''you can not run an interactive job in the guest queue!'''

			== MPI jobs ==

	== A generic example of a job script ==		== A generic example of a job script ==

Paul Keekstra: /* Avoid over- and underutilization */

2019-07-16T19:01:30Z

Avoid over- and underutilization

← Older revision		Revision as of 21:01, 16 July 2019
Line 96:		Line 96:
	For example, Ansys has the <code>-np</code> switch:		For example, Ansys has the <code>-np</code> switch:

	ansys -nt N		ansys -np N

	and Fluent has the <code>-t</code> switch		and Fluent has the <code>-t</code> switch

Paul Keekstra at 14:25, 20 April 2017

2017-04-20T14:25:02Z

← Older revision		Revision as of 16:25, 20 April 2017
Line 151:		Line 151:

	It is important to know that an interactive can only be run in the normal queues, '''you can not run an interactive job in the guest queue!'''		It is important to know that an interactive can only be run in the normal queues, '''you can not run an interactive job in the guest queue!'''

			== A generic example of a job script ==

			The script below can be used as a starting point to create your own jobs. Feel free to copy, paste and modify it to your needs. Lines starting with # will not be executed and contain useful information.

			<pre>
			#!/bin/bash
			#
			# Torque directives (#PBS) must always be at the start of a job script!
			#
			# Request nodes and processors per node
			#
			#PBS -l nodes=1:ppn=1
			#
			#
			# Set the name of the job
			#
			#PBS -N name_of_job
			#
			#
			# Set the mail options (type 'man qsub' for more information)
			#
			#PBS -m bea
			#
			#
			# Set the email address where you want notifications sent to
			# By default mail will be sent to your TU Delft mailbox
			#
			#PBS -M $USER@mailboxcluster.tudelft.net
			#
			#
			# Set the rerunable flag, 'n' is not rerunable, default is 'y'
			#
			#PBS -r y

			# Make sure I'm the only one that can read my output
			umask 0077

			# create a temporary directory in /var/tmp
			TMP=/var/tmp/${PBS_JOBID}
			mkdir -p ${TMP}
			echo "Temporary work dir: ${TMP}"
			if [ ! -d "${TMP}" ]; then
			echo "Cannot create temporary directory. Disk probably full."
			exit 1
			fi

			# copy the input files to ${TMP}
			echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
			/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/

			cd ${TMP}

			#

			module load application1
			module load application2

			export OMP_NUM_THREADS=1

			# Here is where the application is started on the node


			# job done, copy everything back
			echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
			/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

			# delete my temporary files
			[ $? -eq 0 ] && /bin/rm -rf ${TMP}
			</pre>

Paul Keekstra at 14:20, 20 April 2017

2017-04-20T14:20:37Z

← Older revision		Revision as of 16:20, 20 April 2017
Line 129:		Line 129:

	This usually works best if you create a seperate directory in your homedir, move the necessary files and the job script to it and run your job from there. Otherwise you would end up copying your entire home directory to the node for no good reason.		This usually works best if you create a seperate directory in your homedir, move the necessary files and the job script to it and run your job from there. Otherwise you would end up copying your entire home directory to the node for no good reason.

			== Access to nodes ==

			All nodes are independant Linux machines and you could be tempted to log in to one of the nodes and work from there. This is however forbidden, any attempt to log in to a node will fail. There is one exception, you can log in to a node if you have a job running on it, this way you can check on the progress of your job and see if things are still working as intended. To check which node runs your job, type:

			qstat -u $USER -n1

			This will get you a list of all your jobs, in the last column you'll see the nodes in use. If you log in to a node, please do not run any additional CPU intensive programs to avoid overutilization.

			If you must log in to a node in order to run software that can not be run from a script, you can start an interactive job. This is done using the <code>-I</code> switch with qsub, like this:

			qsub -I

			As soon as a node is assigned to you (this may take a while), you'll get a new command line prompt, as if you just logged in with ssh. This will reserve only one processor, you should take care that if you start a CPU intensive program, it does not use more than one processor. If you need more processors or if you want to use a specific node, you can request this for your interactive job with the <code>-l</code> switch, for example, if you want to request 8 processors on node n10-08:

			qsub -I -l nodes=n10-08:ppn=8

			If you want to run a progam with a graphical interface on a node, you'll need to make sure that X forwarding works when logged in to the master node. Then you can use the <code>-X</code> switch start your interactive job with X forwarding enabled:

			qsub -I -X

			It is important to know that an interactive can only be run in the normal queues, '''you can not run an interactive job in the guest queue!'''

Paul Keekstra: /* Avoid excessive reads and writes on your homedir */

2017-04-20T12:59:15Z

Avoid excessive reads and writes on your homedir

← Older revision		Revision as of 14:59, 20 April 2017
Line 121:		Line 121:
	</pre>		</pre>

	Once your program is done you can copy the results back to your home directory and clean up by adding these two lines at the end of your job script.		Once your program is done you can copy the results back to your home directory and clean up by adding these two lines at the end of your job script:

	<pre>		<pre>

Paul Keekstra at 12:49, 20 April 2017

2017-04-20T12:49:51Z

← Older revision		Revision as of 14:49, 20 April 2017
Line 109:		Line 109:

	Of course, N must match the number of processors that you request in your job. Alternatively, you could also request an entire node (all processors) in your job and let your program use all available resources of that node.		Of course, N must match the number of processors that you request in your job. Alternatively, you could also request an entire node (all processors) in your job and let your program use all available resources of that node.

			== Avoid excessive reads and writes on your homedir ==

			Some programs read and write a lot of data to and from your home directory. This is not very efficient, on the nodes your home directory is a network share, so access is relatively slow and it keeps the master node unnecessarily busy. If you expect that your job will do a lot of reading and writing to disk, you can use the local disk on the node instead, which is mounted on /var/tmp on all nodes. You can do this by adding a few extra lines to your job script, right before the line that starts the program in your job, for example:

			<pre>
			TMP=/var/tmp/${PBS_JOBID}
			mkdir -p ${TMP}
			/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/
			cd ${TMP}
			</pre>

			Once your program is done you can copy the results back to your home directory and clean up by adding these two lines at the end of your job script.

			<pre>
			/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
			[ $? -eq 0 ] && /bin/rm -rf ${TMP}
			</pre>

			This usually works best if you create a seperate directory in your homedir, move the necessary files and the job script to it and run your job from there. Otherwise you would end up copying your entire home directory to the node for no good reason.