Cheap Cell Processors and The USAF Condor Cluster

Image via Wikipedia

Last March, bloggers reported that Sony had released a firmware update to remove the OtherOS feature that made PS3's attractive to some users. With the OtherOS option, users could install linux on their PS3's in order to create a very cheap computer with a cell processor.

Some bright sparks hit on the idea of clustering several such customized PS3s, in order to make powerful computer systems. There is even a well written, straightforward guide explaining step-by-step how to do this.

Anyone could install Linux or FreeBSD and set up SSH, NFS and MPI to create a cluster. This is not something many gamers are likely to do, of course.

Seeking to put together a powerful supercomputer for research purposes, the US Air Force upped the ante by putting together 336 PS3s to create a pretty decent supercomputer.

Their reasons for doing this were two-fold:

1. The cell processor is very powerful 8-core CPU, consumes less power than processors of similar capabilities and its chipset lends itself to clustering because "MPI computations run much faster than on desktop workstation chipsets" while an "8 PS3 (i.e. 64 core) Cell cluster had comparable if not better performance to a 200 Node IBM Blue Gene system" (Gaurav Khanna, University of Massachusetts)
2. Cell processors are normally expensive, but Sony produce the PS3 at a loss, with the knowledge that they'll make a profit as customers buy games for their console

All was well until March (although the news of this only broke in May). Sony issued a firmware upgrade that removed the OtherOS option, meaning that anyone who upgraded the firmware would lost the ability to reload the (other) OS, should this need to be done. This made applying the latest firmware had to be avoided at all costs by the USAF, since it would have threatened their entire cluster should they apply it.
Presumably, the USAF weren't buying games and Sony were probably afraid that this would set a precedent, and that they could end up losing a lot of money in the long term, if every university, governmental department,

It seemed like game over for OtherOS, and while a class action lawsuit ensued, it was expected that the USAF would look to other means of building a supercomputer. However, 6 months later, news that they've created the largest cell cluster to date - consisting of 1760 PS3s - was a bit of a surprise (at least it was to me).

Hopefully, this latest lawsuit (and it's not the first Sony have faced) will have the desired effect and OtherOS will make a return. If that happens, I might even consider buying a PS3 myself.

Further Reading

• Air Force may suffer collateral damage from PS3 firmware update (arstechnica.com)
• This Stack Of PS3s Is The 33rd Biggest Computer In The World [Military] (kotaku.com)
• Sony files to dismiss 'Other OS' class action lawsuit (joystiq.com)
• Sony PS3 v3.50 Firmware Update Disabling Some USB Peripherals (slashgear.com)
• USAF Unveils Supercomputer Made of 1,760 PS3s (games.slashdot.org)
• US Air Force connects 1,760 PlayStation 3's to build supercomputer (physorg.com)

Storage solutions

External HDDs and Direct-attached Storage:

Given that many, if not most, of us consumers no longer use desktop PCs in their home, it's rare that we have more than 500GB of internal storage at our disposal. Online storage can provide us with somewhere to store our photos, and even documents and code, but this only adds a few GB (unless you opt for a paid service). As a result, the majority of users who need more storage opt for externall HDDs. They are a good short-term solution, those with eSATA connectivity are fast enough to run VM images, and a couple of 1.5TB disks can keep you going for a while. If you're a hoarder (like me), and/or you want some sort of redundancy, this isn't going to be enough.
Any old PC (not too old, though) can act as a decent file server, with minimal effort. I have an old Athlon that serves as a kind of DAS file server - not dedicated to the task, but it serves my modest needs and beats dragging external HDDs around the house. There are a couple of good NFS apps (or you can use SMB): FreeNAS (based on FreeBSD), and MS' own Windows NFS too.

Networked Storage - NAS and SAN:

Network-Attached Storage has been around a long time in small to medium sized businesses and corporate environments, but more recently it has become more popular with consumers too. It's possible to set up a NAS (using the NFS or SMB protocol), and is one way of expanding your external storage while making it accessible from anywhere on the network. However, ready-to-go NAS devices, ranging from ones nearly as small and portable as an standard external HDD, and not much more expensive, to racks, are completely self-contained: they have their own processor and embedded Linux. The most basic NAS enclosures can be picked up at a very reasonable price from resellers (well known brands available for under €80), and will network any SATA drive. Larger, more expensive NAS devices can network many more drives.

SANs are different to NAS, in that they don't provide any file system, but only storage. The way this storage is utilised is up to the client. SANs use protocols like fibre channel, and also iSCSI and ATA over Ethernet. The last two are a particularly good choice because they don't require dedicated channels of copper or more expensive optical fibre, but use the existing (and ubiquitous) ethernet network and a clever protocol that extends the normally local storage protocol over the network via IP. I don't really understand the ins-and-outs of the SAN controller, but once set up properly and well tuned, SAN performance should be indistinguishable from local storage.
For this reason it's an extremely popular enterprise solution. Many of the customers I deal with use it for storage for their application servers, DBs, LDAPs, mail servers and so on, and it rarely comes under suspicion these days (although it often did up until two or three years ago, so my impression is that they have become very reliable as time has passed).

Scalable storage solutions:

While combining storage with IP can allow you to share storage across a LAN or WAN, things can get messy as you try to scale up the system. Projects to upgrade or expand an existing SAN can be a real headache, and at the very least can involve a very stressful weekend for the IT team.
Using grid architectures, it's possible to build a small storage system first, and then expand it over time. Grid technology also provides redundancy and means upgrades can be carried out while the system is still live.

Related articles

• IDC: Growing Data Need Boosts Q3 Storage Revenue (pcworld.com)
• You Need More Data Storage: Network Attached Storage explored (brighthub.com)
• Excel Meridian Data Adds Expanded Storage Options with its NetStor WSS 8311 and NetStor WSS 8411 Network Attached Storage (NAS) Products (prweb.com)
• Server Virtualization's Encapsulation And Its Impact On Storage (informationweek.com)
• BridgeSTOR dedupes primary storage for small biz (go.theregister.com)
• G-Technology debuts slim G-Drive external HDD (slashgear.com)
• Updated Western Digital external HDDs offer up to 3TB and USB 3.0 (gizmag.com)
• iXsystems Releases FreeNAS 8 Beta (prweb.com)