Introduction to HPC
High-performance computing (HPC) facilities allow researchers to log in remotely and submit compute jobs - essentially the programs that you want to run - from the command line. They are generally large clusters of many processors with a queue-based job submission system. They have many open-source and some commercial software packages installed, and are also often used to run users' own code. Researchers are generally allocated compute time through a merit-based allocation process resembling a mini-grant, but it is usually also possible to get a small amount of time to try the system out, without going through the full allocation process.
At the most basic level, the HPC systems available for your use are just large, very capable, Linux servers. If you're familiar with Linux, any flavour of Unix, or the command line on Mac OS X, you'll have no trouble getting up to speed with HPC. If you're a Windows user the learning curve will be a little steeper, but plenty of help is available to get you on your way.
HPC systems are most often useful for researchers:
- with parallelisable tasks (either using multi-threaded software, or just many tasks which can be submitted independently and simultaneously).
- with computing jobs needing more memory (RAM) than available on their local systems.
- needing access to software packages offered by the HPC facilities.
If you are running out of memory on your desktop system or can no longer process your data quickly enough, then HPC could be for you.
HPC works best for jobs that can be run in a non-interactive batch mode. These jobs can be inserted into a queue, and when the time comes for them to be executed they can run without any user intervention. This means that HPC systems rarely sit idle and it has the benefit that you can enqueue your jobs and get on with other things while you wait for them to execute. Many programs have a batch mode, but not all do. If you're accustomed to using your program interactively, you'll have to learn how to use its batch mode in order to use it for HPC.
Intersect offers courses on the HPC topics below. See http://www.intersect.org.au/training for more information:
- Introduction to Unix and HPC (HPC for Wimps): A two day introduction to our HPC computing facilities. A basic familiarity with Windows or Mac OS X is assumed. For further information please consult the course outline.
- Intermediate HPC (HPC for the CLUEy): A half day introduction to our HPC computing facilities for those with Command Line Unix Experience.
- Parallel Programming for HPC (Advanced): An introduction to Parallel Programming in C or Fortran (necessity, paradigms).
As a researcher you have access to a number of facilities:
- The National Facility at NCI (National Computing Infrastructure), which serves researchers nationally.
- Intersect's Orange facility, which serves researchers in NSW.
- Facilities local to your institution or faculty
For details about the last option, please contact your institution.
A few factors come into play when selecting between the National Facility and Intersect's Orange facility:
- The availability of "specialised facilities" tailored to your needs.
- The availability of your software prerequisites.
- The extent of your processing needs.
- The nature and extent of your memory needs.
To make the right choice, take the following steps:
- Have a look NCI's Specialised Facilities. At the time of writing, NCI offers specialised Bioinformatics and Imaging and Visualisation facilities. If your research is in either of these areas, then NCI could well be the right choice for you.
- Have a look at the software available at the National Facility and on Orange. The following links go to the same page - you can navigate between machines using the drop-down menu on the page:
- Don't be dissuaded if your required piece for software is not listed - researchers can request to have new software packages installed; however, requests cannot always be fulfilled.
- Have a look at the hardware available at the National Facility and on Orange.
- Gain a more detailed understanding of the hardware involved here:
- Consider the memory requirements of your research problem. Orange is a large-scale shared memory system while NCI’s facility is a distributed memory system. This means that Orange is suited to certain classes of research applications, where access to large amounts of memory is a key performance requirement.
Example decision paths
- Alice has a large amount of data she wants to process with Matlab. In her case, she cannot use Matlab's open source equivalent Octave because of compatibility issues. After checking the software listings, she chooses the National Facility because it supports Matlab.
- Bob has an extremely memory intensive application. The National Facility's cluster and Orange have the same architecture, but Orange has more memory per node, so Bob chooses Orange.
- Carlos wants to conduct a large-scale parametric search. His research problem is such that he can execute many instances of the same program at the same time, giving each a small allocation of the problem space to search. He wrote the program himself and it doesn't have an specific software dependencies. Although he could run his application on Orange, he opts for the National Facility's cluster because it has a larger number of cores.
- Dorothy wants to conduct computationally intensive gene sequencing. She chooses the National Facility so she can take advantage of the specialised bioinformatics facility.
So you've chosen your facility - how do you gain access to it? The easiest way is to get a start-up account on the system you want to try. All HPC accounts are limited in terms of the amount of computing resources (e.g. processing time) to which you have access. This is referred to as your quota, and when it runs out you can still run in the bonus queue. The mid-term adjustment may re-distribute un-used quotas.
Start-up accounts have a fairly limited quota, but it is enough for you to assess if HPC is right for you, i.e. to install your application, submit a job, and see if it all runs smoothly.
Once you've successfully completed your evaluation using a start up account, you can apply for a much larger quota. This application will be subject to appraisal along with other applications, and time will be granted on the basis of merit. Start-up accounts let you skip the full grant assessment process and give HPC a go.
A typical start-up account will give you 500 service units of computing resources. One SU usually corresponds to one CPU core-hour, so if you run a parallelised program that uses four processes and runs for one hour before finishing, you will have used 4 SUs of your quota.
To try out:
- the National Facility's cluster, apply for a start-up account by going to https://nf.nci.org.au/accounts/projects_new/APP_form.php and choosing 'Start-Up' from the Scheme/Partner menu.
- Orange, contact firstname.lastname@example.org.
Once your start-up account request has been processed, you should receive the details of the username and password required required to login to your account. You can then get onto connecting to the machine and submitting your first compute job.
When getting started on NCI, see the User Guide at https://nf.nci.org.au/facilities/userguide/ for instructions on how to log in and use the job-submission system.
Connecting to the machines
There are two key ways to access the HPC machines. The first is via an interactive terminal that allows you to issue commands, such as adding your compute job to the queue. The second mode of access is for transferring files to and from your computer and the HPC machine, for example the data your wish to process and, once the job is complete, the results files.
To connect interactively to Orange you'll need a Secure Shell (SSH) client. If you're on Linux or Mac OS X, chances are that you can just open a terminal and issue one of the following command:
When prompted, enter your username and password. If you're running Windows, you'll need to download an SSH client. A good free option is Putty.
To transfer files to and from your machine, you'll need a File Transfer Protocol (FTP) client that supports Secure-FTP (SFTP). You can do this from the command line on Linux, Mac OS X, and Windows:
But you might find it easier to use a graphical FTP program, such as FileZilla.
You've selected the right facility, you've got your start-up account, you've run a test compute job and everything looks fine. How do you get more computing resources (ie. service units) so that you can run a full-scale compute job and get some real results?
Irrespective of whether your using Intersect's Orange or NCI's cluster, computing resources are granted in merit allocation rounds held periodically. You can apply in the following rounds:
- Orange resource allocation rounds (one per year). Rounds open in October for the following calendar year. Further details can be obtained on the Intersect HPC Resource Allocation page.
- NCI merit allocation round. This occurs annually in October for allocations starting the following January. A mid-term call may be held in April for new projects for the July to December period. Further details and an application form can be obtained here.
- Intersect has a partner share arrangement with NCI. This provides a second means of accessing the NCI facilities. Consider applying directly through NCI in the first instance and making an application through Intersect if your request to NCI is not granted. Further details can be obtained on the Intersect HPC Resource Allocation page, and the application form can be obtained here.
It is recommended that you work through this checklist to help assess if HPC is of benefit to you:
- Estimate your computing needs. Work up a rough estimate of the number of CPU hours your compute job will require. As a guide a dual core desktop machine running flat out will provide you with about 1500 core hours (service units) per month. If your current computing infrastructure can handle your computing needs in the timeframe of your research project then skilling up in HPC might cost you time overall.
- Leave plenty of time. If you think HPC could be of use to you, then start investigating it early into your research project. It can take time to assess if your software will run on HPC and to become comfortable in using the command line, module, and queueing systems if you've never used similar systems before. Finally, because HPC is a shared resource, your job will most likely spend some time in the queue awaiting execution.
- Check for available software. Verify that your chosen facility supports the software your job depends on. This is particularly important in the case of commercial software which may not be available due to licensing costs.
- Consider the nature of your research problem. Can it inherently take advantage of the parallelisation of HPC? If not, can you run multiple instances at the same time with different parameters or data to process? If not, then your compute job may not be able to take advantage of HPC's capabilities.
- Check that your software can be used non-interactively. If your processing involves interacting with the program, for instance, clicking buttons or selecting input files using a file browser, then you'll need to find a way to do those things from the command line using a batch mode, if your program supports it.
If you have used resources on Orange or Vayu/NCI via the Intersect partner share, we ask you to acknowledge us. The proposed text is:
Computational (and/or storage) resources used in this work were provided by Intersect Australia Ltd.
The full policy can be found here: http://www.intersect.org.au/attribution-policy