IRD Cluster Usage » History » Version 16

« Previous - Version 16/17 (diff) - Next » - Current version
ndomassi TANDO, 03/30/2017 01:03 PM

IRD Bioinformatic Cluster Usage

1) Introduction:

The IRD Biormatic Cluster is comprised of a pool of machines reachable through a single entry point. The connections to the internal machines are managed by a director node that tries to ensure that proper balancing is made across the available nodes at a given moment.

The cluster is composed of:

- 1 master
- 2 nas servers for a 77To data storage
- 21 nodes servers : 19 nodes with 12 cores, 1 node with 16 coeurs and with RAM from 48Go to 144Go

After your logged with your ldap account and your ird mail password, you can see the architecture details here:

2) Requesting an account:

Please go to
- Log on with your ldap account and your ird mail password
- Choose "Platform" --> "Ask for a Cluster Account"
- Fill the form

3) Accessing to the cluster:

From Windows:

Log on with your ldap account and your ird mail password to acces here:

From MAC/Linux:

Log on with your ldap account and your ird mail password to acces here:

4) Shared areas

The $HOME directories are shared between the master, the nas servers and the computing nodes. There is a projects shared area (located at /data/projects/) and a teams shared area (located at /teams/) , also accessible from the the master, the nas servers and the computing nodes.
The /opt and /usr/local containing scripts of software are also shared between each machine of the cluster.

4.1 Requesting a project:

Please go to
- Log on with your ldap account and your ird mail password
- Choose "Platform" --> "Ask for a Project Account"
- Fill the form

4.2. Usage
The shared directories are not intended for scratch, use the temporal areas of the local filesystems instead. In other words, instruct every job you send to copy the input from the shared directory to the local scratch ($TMPDIR), execute all operations there, then copy the output back to some shared area where you will be able to retrieve it comfortably from the master.
As mentioned above, the contents of $TMPDIR has to be removed after job execution.

4.3 Quota:

Disk quotas are enabled on both user and projects filesystems. If you need more quota on your project shared area, please contact explaining your reasons.

Partition | Quotas

/home --> 100Go
/teams --> 200Go
/data/projects --> 500Go

4.4 How to transfer data to cluster:

Log on with your ldap account and your ird mail password to access here:
Follow the following rules to avoid slowing the cluster network:

On Windows, you can use filezilla to transfer your files:

5) Sotfwares:

5.1 Sotware list:

All softwares of the cluster are contained in the /usr/local/ partition.
You can see the list of the software here:

5.2 Ask for a software:

Please go to
- Log on with your ldap account and your ird mail password
- Choose "Platform" --> "Ask for software install"
- Fill the form

6) Basic Sun Grid Engine Usage:

6.1 Lauching jobs:

The jobs launched on master are scheduled and order than to the Sun Grid engine Tools.

Sund Grid Engine jobs are submitted to the cluster by means of the qsub and qrsh programs:
qsub is used to submit batch jobs.
qrsh is used to submit interactive sessions.

All of these commands allow a wide range of arguments that control the job behaviour; and how the job is scheduled. These options can be specified as follows, from lower to high priority:

- Embedded in the job script.Flags are embedded in the script by specifying them at the very top of the script (SGE Configuration part) in the form #$ <-flag> <flag arguments, for example:

Options       descriptif                       exemple
#$ -N         Name the job                     #$ -N Cas1
#$ -1         Define ressources                #$ -l h_rt=03 :00 :00
#$ -q         Define the queue to use         #$ -q bioinfo.q
#$ -cwd       Specify the directory where to
submit the job

#$ -V Export Environnement variable to the job

#$ -pe        Define the parallel environment  #$ -pe ompi 10
and the number of slots
#$ -t          Submit a job array              #$ -t 1-10
#$ -M          Specify a mail address          #$ -M 
#$ -m          Indicate the events that trigger #$ -m bea
the mail
#$ -S          Specify the shell to be used     #$ -S /bin/bash

- Using the command line when submitting a job with the options above


$~ qsub -N job_name

6.2 Queues available:

Depending on the kind of jobs who are going to launch, you have to choose between the following queues:

Queue Role Nodes Nodes specs
bioinfo.q Normal use Node0 to 20 (except node3, node4, node5, node6, node7) 2 hexa core intel 2,4GHz 48 to 64Go RAM
longjob.q Long and numerous jobs> 1day and/or > 10 jobs Node0 and node1 2 hexa core intel 2,4GHz 48Go RAM
bigmem.q Memory jobs Node3 2 hexa core intel 2,4GHz 96Go RAM
highmem.q Memory jobs Node4 and node7 2 hexa core intel 2,4GHz 144Go RAM
alizon.q prior jobs for samuel alizon's team Node8, node9 and node10 2 hexa core intel 2,4GHz 48Go RAM
r900.q specific queue for DELL R900 Node5 4 quad core intel 2,93GHz 32Go RAM
dynadiv.q queue with nodes with bigger /scratch node2, node10 2 hexa core intel 2,4GHz 48Go RAM
dynadiv2.q queue with node20 node20 2 x10 core intel 2,4GHz 48Go RAM

6.1 Steps to follow:

- Prepare the files (scripts, data, source code, makefiles, etc.) for submitting the job.
- Copy then to cluster on the appropriate nas server ( see "4.4 How to transfer data to cluster" for details)
- Copy those files to the appropriate directory in the /scratch of the node via scp ( )
- Submit the job
- Monitor the job with

qstat -u your_username

- Get your results.
- Transfer them to the appropriate nas server via scp
- Erase data contained in the /scratch directory of the node

7) Use module environment:

It allows you to dynamically change your environment variable(PATH, LD_LIBRARY_PATH) and then choose your version software.
The nomenclature use for modules is <package_name>/<package_version>.

Commands to Use:

To know the available module

$~ module avail

You can have info on the version of a

To load a particular module

$~ module load module_name

example: module load bioinfo/picard-tools/1.119 for the version 1.119 of picard-tools

To list loaded module:

$~ module list

To unload module:

$~  module unload module_name

To purge all the loaded module:

$~ module purge