'Are there clusters available to rent?

I am wondering if there are clusters available to rent.

Scenario:

We have a program that will take what we estimate a week to run(after optimization) on a given file. Quite possibly, longer. Unfortunately, we also need to do approximately 300+ different files, resulting in approximately 300 weeks of compute time(roundable to 6 wallclock years of continuously running job). For a research job that should be done - at the latest - by December, that's simply unacceptable. While we are exploring other options, I am investigating the option of simply renting a Beowulf cluster. The job is academic and will lead towards the completion of a PhD.

What would be ideal would be a company that we send the source and the job files to the company and then receive a week or two later the result files. Voila!

Quick googling doesn't turn up anything terribly promising.

Suggested Solutions?

hpc


Solution 1:[1]

Cloud computing sounds like what you need. Amazon, Microsoft and Google rent computer resources on a pay for what you use basis.

Amazon's service is the most mature, and there are several questions already about Amazon's service, EG here and here.

Solution 2:[2]

Amazon EC2 (Elastic Compute Cloud) sounds like exactly what you're looking for. You can sign up for one or more virtual machines (up to 20 automatically, more if you request permission), starting at $0.10 an hour per VM, plus bandwidth costs (free between EC 2 machines and Amazon's other web services). You can choose between several operating systems (various Linux distributions, OpenSolaris, Windows if you pay extra), and you can use pre-existing machine images or create your own. If you're using all open-source software and don't have much bandwidth costs, it sounds like it would cost you around $5000 to run your job (assuming that your 6 years of compute time was for something comparable to their small instances, with a single virtual CPU).

Once you sign up for the service and get their tools set up, it's pretty easy to get new virtual machines launched. I've even spent the $0.10 to launch a machine for a few minutes just to verify an answer I was giving someone here on StackOverflow; I wanted to check something on Solaris, so I just booted up an instance and had a Solaris VM at my disposal within 5 minutes.

Solution 3:[3]

I don't know where are you doing your PhD... Most of the Asian, European, and North American universities have some clusters. You can

  • meet directly the people at the lab which is in charge of the cluster.
  • ask your PhD director to arrange that. Maybe he/she have some friends that can handle that.

Also, the classical trick is to use the unused time of the computers of your lab/university... Basically, each computer run a client application that crunch numbers when the computer is not used. See http://boinc.berkeley.edu/

Solution 4:[4]

This lead may prove helpful:

http://lcic.org/vendors.html

And this is a fantastic resource site on the matter:

http://www.hpcwire.com

Solution 5:[5]

The thread has been replete with pointers to Amazon's EC2 - and correctly so. They are the most mature in this area. Recently, they've released their elastic map-reduce platform which sound similar (although not exactly) like what you are trying to do. Google is not an option for you as their compute model doesn't support the generic compute model you need.

Solution 6:[6]

Solution 7:[7]

There are several ways to get time on clusters.

  1. Purchase time on Amazon elastic cloud. Depending on how familiar you are with their service, it may take time to get it configured the way you want it.
  2. Approach a university and see if they have a commercial program to rent out the time to companies. I know several do. One that I know of specifically is private sector program at NCSA at UIUC. Depending on the institution, they also offer porting and optimization service for your code.

Solution 8:[8]

Or you could rent CPU time from a private provider.
I'm from Slovenia and, for example, here we have a great private provider called Arctur. The guys were helpful and and responsive when I contacted them.

You can find them here: hpc.arctur.net

Solution 9:[9]

One option is to rent the virtual resources equivalent of whatever number of PCs you need, and set them up as a cluster, using the Amazon Elastic Compute Cloud.

Setting up a beowulf cluster of those is entirely possible.

Check out this link which provides resources and software to do exactly that.

Solution 10:[10]

Go to : http://www.extremefactory.com/index.php True HPC cluster, up to 200 TFlops.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Community
Solution 2
Solution 3 Monkey
Solution 4 Anthony
Solution 5 argodev
Solution 6 PA.
Solution 7 powerrox
Solution 8 Josh Darnell
Solution 9 Hejazzman
Solution 10 Cyril

Related Questions