I'm using LIBSVM for regression analysis. Works like a champ. But a 3-parameter grid search to optimize parameters for the model maxes out all four cores on my 2.66 GHz Intel box, and I still have to wait a couple of hours to generate a single model.

This seems like a job for Amazon EC2.

I've seen plenty of tutorial开发者_Go百科s and introductory material on using EC2 for web-related tasks.

But what if you have a small compute-intensive custom ANSI-C program that you want to run multiple instances of on EC2? Can anyone provide pointers on how to do that (or even just buzzwords to search for)?

---

I don't think your quest is too different from that of a web application. Your stack is different of course, but regardless – the principles remain the same.

As someone commented on your question: Elastic Map Reduce might be what you're looking for the parallelize your work easily, etc.. If that is too limited, you could look into Cloudera. A ready-to-rumble hadoop distribution with support for EC2 as well.

If map-reduce is not to your liking, then you need to setup your own instance. Roughly speaking, the keypoints are as follows:

1. You want to figure out a way to start EC2 instances.
2. You want to figure out a way to bootstrap and configure them.
3. Cluster/network?

Starting EC2 instances

If you don't require something like auto-scaling or a custom interface, the AWS Console does an extremely good job. You have to select an AMI (Amazon Machine Image) suitable for your project. I'd probably look into either the official AMI or something Ubuntu-based (If I remember correctly, Ubuntu is the most used Linux on EC2).

But that is up to you and your liking. (And I don't know enough about your project.)

Once you figured out a setup that works for you, the easiest way to clone your work is to setup your own AMI and start instances with it, etc..

Bootstrapping

Bootstrapping can be using what EC2 calls user-script. It allows you to pass shell script to the instance, which would execute calls to setup your stack, etc.. I'm not sure what is required in this case, etc.. So in case you comment or extend your answer, I could go into detail here.

Cluster/Networking

This is a wild guess since I'm not sure what your code does, or how it works, etc.. If it's not necessary, I'd probably scale this out using a single instance first. You can get a lot of cores and RAM provisioned easily with EC2. Depending if your work requires more RAM or CPU, look into high-cpu and high-memory instance types.

You can start off with a t1.micro, which you can currently get for free even and go from there.

Let me know if this helps!