Project: Quad-CPU Server (Intro)

I've built my own computers for years, now, and I once set up a client/server lab in my home to research and practice distributed computing. But that was back in the dotcom bubble days and I'm feeling the urge to play with technologies on the bleeding edge again.

The convergence of multi-CPU motherboards, multicore CPUs, and the ability to access gobs of RAM memory (thanks to 64-bit technology) means that it is possible to cram an incredible amount of distributed processing power into a single PC case. (albeit a very large one) Companies are interested in this because it means they can consolidate many separate servers into far fewer physical boxes by using virtualization software. This software lets you boot up several instances of Operating Systems on a single computer. (these instances still seem like separate computers to the outside world but doing it this way saves money by using less space, hardware, power, cooling, etc.)

However, I have bigger fish to fry than playing with virtualization. I'm interested in working on the kinds of problems that only a small army of CPUs can solve by working together. To make this work, I need to create two software systems: the first one divides up the work, sends it out to the army of CPUs, and then receives and collates the answers from them. The the second system is the code that actually runs on the army of CPUs to solve the broken-down problem. Basically I'm talking about creating a small-scale supercomputer. If virtualization means treating one computer as if it were many computers, my research will involve treating many computers as if they were one.

My platform of choice will be the AMD Opteron 8000 series of quad-core server CPUs. This is the only mature "commodity" processor series with an integrated memory controller supporting four (or even eight) CPUs in a single case. Intel's Xeon 7300 platform supports 4 CPUs, but it seems like a bit of a hack to me since it goes through contortions to compensate up for its external memory controller, plus it is much more costly. Intel's new "Core i7" CPUs will have integrated memory controllers, but they won't be available for quad-CPU solutions before some time in 2009.

A reasonable question to ask is, "Why not just buy four (or more) regular PCs that each have a single multicore processor and network them together?" Indeed, it would be far cheaper to do it this way due to the price premium of the processors and motherboards that support quad-CPU configurations. But AMD uses a chip-to-chip interconnect called Hyper Transport that allows messages between processors on a single motherboard to happen at far higher speeds and lower latencies than Ethernet can handle. Also, each multicore CPU on the motherboard has it's own local bank of RAM memory. Accessing that local RAM is much quicker than being sloppy and borrowing some from a neighbor CPU's bank of RAM. (this is referred to as NUMA or non-uniform memory access) So, part of the research I intend to do will involve writing software that is aware of these things and able to organize the rest of the code to make the best use them.

Another good question would be, "Why go to the trouble of building such a complicated computer when you can buy one that just works from an established vendor who offers service and support?" Beyond all of the reasonable-sounding answers I could rationalize for this one, it boils down to the fact that I just like to scrape up my knuckles playing with hardware from time to time. Plus, I just know I'd open up and muck around with a pre-built server anyway, instantly voiding the warranty and ensuring myself a place in phone-system purgatory if I ever hypothetically called for support.

One great feature of these quad-CPU Opteron motherboards is that you can use them with one, two, or four CPUs. Nice! Instead of having bite the bullet and pay for all four processors at once, (as well as four banks of RAM) I'll be able to initially buy just one processor and start using the computer during the time that I'm still upgrading it on the installment plan.

So, lucky you! You get to follow along as I progress along the path of building my quad-CPU server. My plan for purchasing the components is to buy the foundational items first that tend to have stable prices (like the case, the power supply, hard drives) and then add the items later that tend to go down in price over time (the CPUs, the RAM).