General Description
lightning is an AMD Opteron cluster with 592 processor cores (148 nodes) with each node having 8 GBytes of memory (1182 GB total memory on the cluster). Nodes are interconnected with a high performance InfiniPath HTX communication network for MPI communication and a Gigabit Ethernet switch for I/O, management, and PBS communication. 15 Terabytes of RAID-5 storage are integrated into the cluster. The peak performance of each processor is 4.8 Gflops so 376 processors gives a peak performance of 2,842 gigaflops (2.842 teraflops). The cluster was purchased from Atipa, see www.atipa.com
Front End Node
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache and 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with two 500 watt, hot swappable, redundant power supplies
Two mirrored, 160 GB SATA, enterprise quality, Western Digital (
·
144 Compute Nodes with each node consisting of the following:
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache and 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with a 500 watt, cold swappable power supply
One 160 GB SATA, enterprise quality, Western Digital (140 GB of temporary local disk space accessible as $TMPDIR under PBS and with 16 GB of swap space.
3 Storage Nodes with each node consisting of the following:
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache
· 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with 760 watt, hot swappable, triple-redundant power supplies
· 5 TeraBytes of usable RAID-5 storage with fifteen 400 GB Western Digital (WD4000YR) enterprise quality, SATA disk drives with one disk drive running as a hot spare.
Communication Networks
Interconnecting Nodes
· A Mellanox 144-port InfiniBand switch with InfiniPath HTX cards is used for MPI communication providing very low latencies (1.29 microseconds) and very high bandwidth (954 Mbytes/second).
· Three stacking, 48-port managed Gigabit Ethernet switches are used for I/O, management, and PBS communication.
· The cluster has a Gigabit Ethernet connection to the campus network.
Security
The cluster can be accessed only by ssh from machines that are on campus and not from off campus. To access lightning from off campus, one must first logon to a machine on campus and then ssh to lightning.