New Linux system 4x Octal Core Xeon's 256GB RAM

[James]

Thought some of you might find this interesting (and some might not)

Just built a new workhorse server. Basic specs are below.

And as a bonus, a quick set of pictures from the setup.

Dell R910
4x X7650 2.26GHz 8-core Xeon CPU's w/ HT (64 threads)
256GB ECC/REG RAM'
3x 146GB 15k SAS 2.5" hdd's
45x 1TB SAS 3.5" drives
Supermicro SAS2 JBOD enclosure (24 drives in front, 21 drives in the back)
2x LSI 9280DE SAS2 RAID controllers
CentOS 5.5 x64

Simpleviewer pics of new server

[ua549]

That's pretty cool, James. What apps is it going to run? One large database?

[proxops-pete]

Yikes...

[James]

Sort of. It's a data processing box. Large data sets in excess of 100GB apiece. Some larger than 500GB. Some smaller ones too. It's made to be "the" processing box. I.e. we can run multiple jobs on it without slow downs.

So it needs not only the RAM and CPU's, but also fast disk access. As it stands right now, the arrays are close to pushing 1GB/s. Having some issues with write caching with Linux though (my thread in the OS forum).

Overall I'm pretty happy with the box, but I haven't fully stress tested it yet.

[OS-Wiz]

Sweet, but can you overclock it

[MrDigital]

They really need to make RAM smaller, that takes up WAY too much space.

[Nabby]

Sweet, but can you overclock it

You don't need to .

Seriously though, that is one loaded box.

[James]

They really need to make RAM smaller, that takes up WAY too much space.

No doubt! Cheaper would be nice too. The box is rated at 1TB RAM capacity maximum (64 x 16GB DIMMs). If you wanted to actually get the 1TB of RAM, it would have cost over $100,000 just for RAM.

[ua549]

I remember that mainframe memory cost about $100k per megabyte in 1980.
It was a big purchase for my "home" computer I had in a rented garage.

[Vindir]

Quick, install Quake before they start using it for productive applications!

<--Jealous.

[Nater]

Anyway you can run Cinema 4D on that beast? I saw it run on an eight socket Nehalem-EX system, I've never seen it run on a four socket.

Gotta love Supermicro. I was pretty disappointed with their range of Boxboro-EX systems. They're all four-sockets with only four DIMMs per socket. They need to offer some two and four socket fat memory systems (256GB and 512GB) and maybe an eight socket system too. I suppose you can do 512GB with their systems, but it requires that you use hyper-expensive 16GB DIMMs. Even in a system that expensive, I'm guessing most people would much rather use the much less expensive and more available 8GB DIMMs.

Maybe they will, they just brought out SBB and new Itanium systems, so maybe they're just the first rollout.

What RAID level are you running on the 1TB drives? Are they broken down into multiple arrays?

EDIT. I really like the idea of placing the DIMM slots on separate cards. I'm guessing this allows you to jam more slots in less board space. Apple needs to return to this with it's Mac Pro, four DIMM slots per CPU just isn't enough in a burly two socket workstation.

[James]

We are running RAID6 on the 1TB drives. 1 array of 24 drives, 1 array of 21 drives.

[OS-Wiz]

For those that haven't used RAID 6:
"RAID 6 extends RAID 5 by adding an additional parity block; thus it uses block-level striping with two parity blocks distributed across all member disks.
RAID 6 does not have a performance penalty for read operations, but it does have a performance penalty on write operations because of the overhead associated with parity calculations.
The definition of RAID 6 is: "Any form of RAID that can continue to execute read and write requests to all of a RAID array's virtual disks in the presence of any two concurrent disk failures."
http://en.wikipedia.org/wiki/Standard_RAID_levels