What "pieces" are needed in order to set up a cluster of physical servers?
- by Chris Dutrow
Background:
Currently, we use Rackspace cloud servers. We have no intention to stop using them, but would like to look into setting up a cluster of physical servers (probably desktop computers in the $400 range with 8gb memory each) to offset some of our load and work as a secondary, more powerful, less reliable system. To put things in perspective, we can buy comparable desktop computers for the same price as we pay in one month to rent them on Rackspace Cloud.
I understand that this is generally a dumb idea. However, in this particular instance, the server cluster is needed for its computation power. It is not mission-critical, it does not host a consumer-facing website, and if it goes down for a day or two, its not really a problem.
Currently, we have access to business class verizon fios. If I understand correctly, we can get at least 25 dedicated IP addresses with this service, this should be enough.
Requirements:
Each server runs Linux Centos 6.3
Some of the servers run Python and execute processes from a task queue (Redis or RabbitMQ)
Some of the servers are capable of serving static files and Python driven REST APIs
Some of the servers host a Cassandra database cluster
One or more of the servers are a Redis database servers
One or more of the servers are PostgreSQL servers
Questions:
What kind of router or switch is needed?
We would like the computers to be able to communicate effectively with each other via internal IP addresses. This is especially important for communicating with servers hosting Redis that need to be able to respond to requests very quickly. Are there special switches or routers that need to be used to connect the servers together?
Are Desktop computers ok for this?
We have found that we are mostly RAM-bottle necked, I understand that some servers have highly superior CPUs, but I'm not sure we need CPU power as much as we need RAM, which is cheap in Desktop computers.
Will we have problems with the WIFI cards in the desktops or any other unexpected hardware limitation?
What tools should be used to "image" the servers.
For example, when we get an installation right for a Redis server or Cassandra node, are there tools that come with Linux Centos 6.3 to image the server to a USB drive or something like that?
Or do we need to use some other software for this?
What other things are we missing that we should be concerned about?
Thanks so much!
