Titan (supercomputer)

Plans to create a supercomputer capable of 20 petaFLOPS at the Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National Laboratory (ORNL) originated as far back as 2005, when Jaguar was built.[3] Titan will itself be replaced by an approximately 200 petaFLOPS system in 2016 as part of ORNL's plan to operate an exascale (1000 petaFLOPS to 1 exaFLOPS) machine by 2020.[3][4][5] The initial plan to build a new 15,000 square meter (160,000 ft²) building for Titan, was discarded in favor of using Jaguar's existing infrastructure.[6] The precise system architecture was not finalized until 2010, although a deal with Nvidia to supply the GPUs was signed in 2009.[7] Titan was first announced at the private ACM/IEEE Supercomputing Conference (SC10) on November 16, 2010, and was publicly announced on October 11, 2011, as the first phase of the Titan upgrade began.[4][8]

Jaguar had received various upgrades since its creation. It began with the Cray XT3 platform that yielded 25 teraFLOPS.[9] By 2008, Jaguar had been expanded with more cabinets and upgraded to the XT4 platform, reaching 263 teraFLOPS.[9] In 2009, it was upgraded to the XT5 platform, hitting 1.4 petaFLOPS.[9] Its final upgrades brought Jaguar to 1.76 petaFLOPS.[10]

Titan was funded primarily by the US Department of Energy through ORNL. Funding was sufficient to purchase the CPUs but not all of the GPUs so the National Oceanic and Atmospheric Administration agreed to fund the remaining nodes in return for computing time.[11][12] ORNL scientific computing chief Jeff Nichols noted that Titan cost approximately $60 million upfront, of which the NOAA contribution was less than $10 million, but precise figures were covered by non-disclosure agreements.[11][13] The full term of the contract with Cray included $97 million, excluding potential upgrades.[13]

The yearlong conversion began October 9, 2011.[14][15] Between October and December, 96 of Jaguar's 200 cabinets, each containing 24 XT5 blades (two 6-core CPUs per node, four nodes per blade), were upgraded to XK7 blade (one 16-core CPU per node, four nodes per blade) while the remainder of the machine remained in use.[14] In December, computation was moved to the 96 XK7 cabinets while the remaining 104 cabinets were upgraded to XK7 blades.[14] ORNL's external ESnet connection was upgraded from 10 Gbit/s to 100 Gbit/s and the system interconnect (the network over which CPUs communicate with each other) was updated.[14][16] The Seastar design used in Jaguar was upgraded to the Gemini interconnect used in Titan which connects the nodes into a direct 3D torus interconnect network.[17] Gemini uses wormhole flow control internally.[17] The system memory was doubled to 584 TiB.[15] 960 of the XK7 nodes (10 cabinets) were fitted with a Fermi based GPU as Kepler GPUs were not then available; these 960 nodes were referred to as TitanDev and used to test code.[14][15] This first phase of the upgrade increased the peak performance of Jaguar to 3.3 petaFLOPS.[15] Beginning on September 13, 2012, Nvidia K20X GPUs were fitted to all of Jaguar's XK7 compute blades, including the 960 TitanDev nodes.[14][18][19] In October, the task was completed and the computer was finally renamed Titan.[14]

In March 2013, Nvidia launched the GTX Titan, a consumer graphics card that uses the same GPU die as the K20X GPUs in Titan.[20] Titan underwent acceptance testing in early 2013 but only completed 92% of the tests, short of the required 95%.[14][21] The problem was discovered to be excess gold in the female edge connectors of the motherboards' PCIe slots causing cracks in the motherboards' solder.[22] The cost of repair was borne by Cray and between 12 and 16 cabinets were repaired each week.[22] Throughout the repairs users were given access to the available CPUs.[22] On March 11, they gained access to 8,972 GPUs.[23] ORNL announced on April 8 that the repairs were complete[24] and acceptance test completion was announced on June 11, 2013.[25]

Titan's hardware has a theoretical peak performance of 27 petaFLOPS with "perfect" software.[26] On November 12, 2012, the TOP500 organization that ranks the world's supercomputers by LINPACK performance, ranked Titan first at 17.59 petaFLOPS, displacing IBM Sequoia.[27][28] Titan also ranked third on the Green500, the same 500 supercomputers ranked in terms of energy efficiency.[29] In the June 2013 TOP500 ranking, Titan fell to second place behind Tianhe-2 and to twenty-ninth on the Green500 list.[30][31] Titan did not re-test for the June 2013 ranking,[30] because it would still have ranked second, at 27 petaFLOPS.[32]

Titan uses Jaguar's 200 cabinets, covering 404 square meters (4,352 ft²), with replaced internals and upgraded networking.[33][34] Reusing Jaguar's power and cooling systems saved approximately $20 million.[35] Power is provided to each cabinet at three-phase 480 V. This requires thinner cables than the US standard 208 V, saving $1 million in copper.[36] At its peak, Titan draws 8.2 MW,[37] 1.2 MW more than Jaguar, but runs almost ten times as fast in terms of floating point calculations.[33][36] In the event of a power failure, carbon fiber flywheel power storage can keep the networking and storage infrastructure running for up to 16 seconds.[38] After 2 seconds without power, diesel generators fire up, taking approximately 7 seconds to reach full power. They can provide power indefinitely.[38] The generators are designed only to keep the networking and storage components powered so that a reboot is much quicker; the generators are not capable of powering the processing infrastructure.[38]

Titan has 18,688 nodes (4 nodes per blade, 24 blades per cabinet),[39] each containing a 16-core AMD Opteron 6274 CPU with 32 GB of DDR3 ECC memory and an Nvidia Tesla K20X GPU with 6 GB GDDR5 ECC memory.[40] There are a total of 299,008 processor cores, and a total of 693.6 TiB of CPU and GPU RAM.[36]

Initially, Titan used Jaguar's 10 PB of Lustre storage with a transfer speed of 240 GB/s,[36][41] but in April 2013, the storage was upgraded to 40 PB with a transfer rate of 1.4 TB/s.[42] GPUs were selected for their vastly higher parallel processing efficiency over CPUs.[40] Although the GPUs have a slower clock speed than the CPUs, each GPU contains 2,688 CUDA cores at 732 MHz,[43] resulting in a faster overall system.[34][44] Consequently, the CPUs' cores are used to allocate tasks to the GPUs rather than directly processing the data as in conventional supercomputers.[40]

Titan runs the Cray Linux Environment, a full version of Linux on the login nodes that users directly access, but a smaller, more efficient version on the compute nodes.[45]

Titan's components are air-cooled by heat sinks, but the air is chilled before being pumped through the cabinets.[46] Fan noise is so loud that hearing protection is required for people spending more than 15 minutes in the machine room.[47] The system has a cooling capacity of 23.2 MW (6600 tons) and works by chilling water to 5.5 °C (42 °F), which in turn cools recirculated air.[46]

Researchers also have access to EVEREST (Exploratory Visualization Environment for Research and Technology) to better understand the data that Titan outputs. EVEREST is a visualization room with a 10 by 3 meter (33 by 10 ft) screen and a smaller, secondary screen. The screens are 37 and 33 megapixels respectively with stereoscopic 3D capability.[48]

In 2009, the Oak Ridge Leadership Computing Facility that manages Titan narrowed the fifty applications for first use of the supercomputer down to six "vanguard" codes chosen for the importance of the research and for their ability to fully utilize the system.[34][49] The six vanguard projects to use Titan were:

• S3D, a project that models the molecular physics of combustion, aims to improve the efficiency of diesel and biofuel engines. In 2009, using Jaguar, it produced the first fully resolved simulation of autoigniting hydrocarbon flames relevant to the efficiency of direct injection diesel engines.[49]
• WL-LSMS simulates the interactions between electrons and atoms in magnetic materials at temperatures other than absolute zero. An earlier version of the code was the first to perform at greater than one petaFLOPS on Jaguar.[49]
• Denovo simulates nuclear reactions with the aim of improving the efficiency and reducing the waste of nuclear reactors.[34] The performance of Denovo on conventional CPU-based machines doubled after the tweaks for Titan and it performs 3.5 times faster on Titan than it did on Jaguar.[49][50]
• Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a molecular dynamics code that simulates particles across a range of scales, from quantum to relativistic, to improve materials science with potential applications in semiconductor, biomolecule and polymer development.[51]
• CAM-SE is a combination of two codes: Community Atmosphere Model, a global atmosphere model, and High Order Method Modeling Environment, a code that solves fluid and thermodynamic equations. CAM-SE will allow greater accuracy in climate simulations.[49]
• Non-Equilibrium Radiation Diffusion (NRDF) plots non-charged particles through supernovae with potential applications in laser fusion, fluid dynamics, medical imaging, nuclear reactors, energy storage and combustion.[49] Its Chimera code uses hundreds of partial differential equations to track the energy, angle, angle of scatter and type of each neutrino modeled in a star going supernova, resulting in millions of individual equations.[52] The code was named Chimera after the mythological creature because it has three "heads": the first simulates the hydrodynamics of stellar material, the second simulates radiation transport and the third simulates nuclear burning.[52]
• Bonsai is a gravitational tree code for n-body simulation. It has been used for the 2014 Gordon Bell prize nomination for simulating the Milky Way Galaxy on a star by star basis, with 200 billion stars. In this application the computer reached a sustained speed of 24.773 petaFlops.[53]
• VERA is a light-water reactor simulation written at the Consortium for Advanced Simulation of Light Water Reactors (CASL) on Jaguar. VERA allows engineers to monitor the performance and status of any part of a reactor core throughout the lifetime of the reactor to identify points of interest.[54] Although not one of the first six projects, VERA was planned to run on Titan after optimization with assistance from CAAR and testing on TitanDev. Computer scientist Tom Evans found that the adaption to Titan's hybrid architecture was more difficult than to previous CPU-based supercomputers. He aimed to simulate an entire reactor fuel cycle, an eighteen to thirty-six month-long process, in one week on Titan.[54]

In 2013 thirty-one codes were planned to run on Titan, typically four or five at any one time.[47][55]

The code of many projects has to be modified to suit the GPU processing of Titan, but each code is required to be executable on CPU-based systems so that projects do not become solely dependent on Titan.[49] OLCF formed the Center for Accelerated Application Readiness (CAAR) to aid with the adaptation process. It holds developer workshops at Nvidia headquarters to educate users about the architecture, compilers and applications on Titan.[56][57] CAAR has been working on compilers with Nvidia and code vendors to integrate directives for GPUs into their programming languages.[56] Researchers can thus express parallelism in their code with their existing programming language, typically Fortran, C or C++, and the compiler can express it to the GPUs.[56] Dr. Bronson Messer, a computational astrophysicist, said of the task: "an application using Titan to the utmost must also find a way to keep the GPU busy, remembering all the while that the GPU is fast, but less flexible than the CPU."[56] Moab Cluster Suite is used to prioritize jobs to nodes to keep utilization high; it improved efficiency from 70% to approximately 95% in the tested software.[58][59] Some projects found that the changes increased efficiency of their code on non-GPU machines; the performance of Denovo doubled on CPU-based machines.[49]

The amount of code alteration required to run on the GPUs varies by project. According to Dr. Messer of NRDF, only a small percentage of his code runs on GPUs because the calculations are relatively simple but processed repeatedly and in parallel.[60] NRDF is written in CUDA Fortran, a version of Fortran with CUDA extensions for the GPUs.[60] Chimera's third "head" was the first to run on the GPUs as the nuclear burning could most easily be simulated by GPU architecture. Other aspects of the code were planned to be modified in time.[52] On Jaguar, the project modeled 14 or 15 nuclear species but Messer anticipated simulating up to 200 species, allowing far greater precision when comparing the simulation to empirical observation.[52]

• Jaguar (supercomputer) – OLCF-2
• Summit (supercomputer) – OLCF-4
• Oak Ridge Leadership Computing Facility

• Media related to Titan (supercomputer) at Wikimedia Commons
• Official website
• "GeForce GTX Titan". Nvidia. Archived from the original on March 26, 2013. Retrieved March 26, 2013.