We would like to demonstrate that Pi can be efficiently calculated to 62.8 trillion decimal places with limited hardware, personnel and budgetary resources. The current world record for the most accurate number of Pi will serve as a reference for our performance.    

Calculating the most accurate number of Pi is an unofficial benchmark in the area of high-performance computing. Although there is no practical application in knowing trillions of decimal places of a transcendental number, this is however, a useful test of our abilities to run extremely resource-demanding computations. Such an endeavour not only requires the selection of the appropriate hardware but also needs expertise in assembling, configuring and tuning hardware and software for maximum performance.  Furthermore, we would like to test our ability to reliably manage a several-months-long running computation. The challenge for this record attempt is taken up by project leader Thomas Keller and by the head of DAViS, Prof. Dr Heiko Rölke.

DAViS as part of the University of Applied Sciences of the Grisons is consolidating expertise in the area of high-performance computing (HPC) and HPC infrastructure for a mandate of the canton of Grisons. This also includes implementing a new degree programme in the field of 'Computational and Data Science'. Prospective students will benefit from access to powerful hardware and will be able to practice newly acquired theoretical knowledge using state-of-the-art hardware and software.

News

Statistic

Pi computation progress: 100%
Pi verification progress: 0%
Downtime (backup or maintenance): 18.57hrs (backup) / 27.55hrs (backup) / 41.64hrs (backup) / 25.51hrs (backup) / 26.47hrs (backup) - 139.74hrs (Total downtime)

Calculating Pi to the trillion digit range demands extreme amounts of RAM and swap space as well as high memory bandwidth. For this type of computation processor performance is of secondary importance.

Processors
2 AMD EPYC 7542 CPUs, 32 cores each

Memory (RAM)
1 TB

Disk space total
510 TB

HDDs
38 hard disk drives running at 7,200 rpm and with 16 TB each. 34 of these disks are used for swap space and 4 disks will hold the final Pi digits

SSDs
2 SSDs in a RAID configuration for storing the operating system

Although our installed RAM is very large compared to a regular desktop PC, it would only be enough to calculate Pi to a few 100 billion decimal places. To work around this limitation, the Pi calculation program stores a large portion of its memory on hard disk drives. This technique is called ‘swapping’ and is a method also employed by regular PCs, for example with Windows or Mac. Performance-wise, SSDs (very fast hard disks based on semiconductor technologies) would be of great advantage for this type of calculation. However, for our Pi computation and also future computations, we expect a large amount of disk write access, which would result in high wear and early failure of an SSD. Since SSD storage prices are 5 to 10 times higher compared to conventional HDDs, we opted for HDDs in the design process. In our case the risk of early failure applies primarily to swap storage space. We use SSDs for storing the operating system because we anticipate fewer write cycles. Additionally, storing the operating system requires only a few GBs of space.

To somewhat compensate for the speed disadvantage using HDDs compared to SSDs, we configured y-cruncher for highly parallel access to the swap area. This configuration allows us an aggregated read/write speed to the swap memory area of about 8.5 GB/s using 34 HDDs. The same result could be achieved by using only three SSDs. However, with just three SSDs, we would have missed the Pi storage requirement of about 310 TB by several factors.

The final Pi digit will need 63 terabytes of storage space. An additional 91 TB of hard disk space is assigned for the storage of intermediate calculation, called checkpoints. As mentioned before, up to 310 TB of storage space is used for the swap area, and for the occasional backup we have earmarked another 180 TB of storage space.

The system used in this calculation is made up of two units:

The system is hosted in a standard server rack and receives cooling and emergency power. Cooling is imperative to keep all components below the maximum allowed temperature recommended by the manufacturer.

The computing unit with CPUs, memory and SSDs consumes approx. 300 watts of power. The JBOD with 38 HDDs requires approx. 430 watts.

Since the computer and the disks need to be continuously cooled during operation (the CPU cores can heat up to 80° C  and the disks up to 40° C), this heat has to be dissipated with air cooling. According to American Power Conversion, we need an additional 950 watts for cooling. We estimate our power demand to total at around 1,700 watts.

In comparison:

• a conventional hair dryer draws about 2,000 watts of power
• 1.75 GFlops/Watt would put us in 153rd place on the Green500 list of supercomputers

For calculating Pi we use the software y-cruncher. This software has been employed by previous world record runs of Pi and is characterised by the fact that it takes advantage of modern processor features. y-cruncher has been developed by the American programmer Alexander Yee and can be run on any standard PC with Linux or Windows to calculate mathematical constants such as Pi.

We use the operating system Ubuntu 20.04 on the computation unit. This system allowed us to tune various parameters to decrease calculation time. Among other things, we configured the following settings to improve execution time:

• disabled various security features
• activated an IO scheduler optimised for the computation
• disabled power-saving settings
• changed to a faster file system
• made BIOS optimisations

Some of these optimisations have serious disadvantages. For example less reliability or higher power consumption. Others only improve execution time for a specific software or calculation type.

The number Pi is an irrational and transcendental number. This entails that the number Pi can neither be represented as a fraction (irrational) nor as polynomial (transcendental).

In 1988, the Chudnovsky brothers developed the following algorithm for calculating the number Pi:

A noteworthy property of the algorithm is that the sum (sigma symbol) has a limit of infinity (lying eight). The reason is that Pi is an irrational number, which converges but can never be represented with final precision. Having only limited computing resources at hand, we will never be able to calculate Pi precisely. The algorithm belongs to the family of hypergeometric series and relies for its computation mainly on integer numbers. Thanks to these properties we can calculate Pi without rounding or approximations. This avoids rounding errors, which can lead to precision errors. Rounding errors have led to spectacular failures in the past.

Currently, the Chudnovsky formula is still the most efficient method of calculating Pi to an arbitrary number of digits. Using this method, the number Pi is calculated to an average precision of 14.81 digits per iteration. All recent world records have been calculated using this method. This algorithm was implemented in y-cruncher using C/C++ and and is currently used by DAViS and previously by Timothy Mullican and Google in their respective world record calculations.

There are two possibilities of checking the newly calculated digits for their correctness: one could recalculate the number of digits with another algorithm (for example with the Gauss-Legendre algorithm). However, this would again take at least as much time (and probably a lot more time) than the original calculation. Fortunately, the Bailey-Borwein-Plouffe formula was discovered in 1995 by Simon Plouffe. With this formula an arbitrary digit of the number Pi can be calculated without needing to calculate previous digits! For example, one could calculate the digit 10,581 without having to calculate all the preceding digits. However, calculating a single digit of Pi is very computationally complex (exponential), so only some of the last digits of Pi are verified for a given calculation.

Two years later, the French mathematician Fabrice Bellard found another and faster way to calculate random digits of Pi. This method is used by y-cruncher to verify the calculated number.

High Performance Computing (HPC) is by no means restricted to the area of calculating mathematical constants. HPC is in fact used in many areas of applied research and development. As ever increasing amounts of data require ever more powerful hardware and software, HPC is for example used in the process of DNA sequence analysis or machine learning.

Among other research projects, DAViS provides computing power for the Translaturia project (machine learning) or supports the allergy and asthma research of the SIAF in Davos with DNA sequence analysis.

DAViS is a regional competence centre for research, industry and education in the areas of data analysis and scientific computing. We collaborate with the national high-performance computing centre (CSCS) in Lugano operated by ETH Zurich and cooperate with the Swiss Institute for Allergy and Asthma Research (SIAF) at the University of Zurich.

As DAViS is part of the University of Applied Sciences of the Grisons, we can draw on a broad range of domain-specific expertise and run an advanced IT infrastructure to support our research. In the case of the world record attempt, we are working closely with the IT services department of the University of Applied Sciences during the hardware setup and the commissioning of the new server.

For inputs and questions regarding the Pi world record attempt, please contact Thomas Keller or Heiko Rölke.

Further team member
Swiss Institute of Allergy and Astma Research (SIAF) 
PD, Ph.D. Katja Bärenfaller 
+41 81 410 08 49 
katja.baerenfaller@siaf.uzh.ch