Primary references

• TOP500 list pages (June 1993, June 1997, June 2000, June 2010, June 2022, June 2023, November 2023, June 2024, November 2024, June 2026)

  Cores, Rmax, Rpeak and power of the systems used in Table 1.1 and the programs; Frontier June 2024: 8,699,904 cores, 1,206 PFlop/s, 1,714.81 PFlop/s, 22,786 kW; rank-500 Rmax values of 1993 (0.42 GFlop/s), 2010 (24.67 TFlop/s), 2024 (2.13 PFlop/s), 2026 (2.66 PFlop/s). June 2023 and November 2023: Frontier 8,699,904 cores, Rmax 1,194 PFlop/s, Rpeak 1,679.82 PFlop/s, 22,703 kW (the slide matches these lists and June 2024).

• TOP500, The Linpack Benchmark

  The benchmark solves a dense linear system; LU with partial pivoting with 2/3 n^3 + O(n^2) operations; definitions of Rmax, Nmax, N1/2, Rpeak; the performance does not reflect overall performance.

• HPL 2.3 source (testing/ptest/HPL_pdtest.c, HPL.dat)

  Rate computed from (2/3)N^3 + (3/2)N^2 (lines 231-233); scaled residual with infinity norms and the pass test (lines 393-406); threshold 16.0 (HPL.dat line 13).

• TOP500, HPCG list June 2024

  HPCG described as a complement to the Linpack measurement; Frontier 14,054 TFlop/s and Fugaku 16,004.5 TFlop/s on HPCG.

• OLCF Frontier User Guide

  One 64-core EPYC and four MI250X (eight GPUs) per node; 512 GB DDR4; 64 GB HBM2E per GPU; 110 compute units per GPU; 47.8 TFLOPS vector double-precision per MI250X; 9,856 compute nodes at the date of reading.

• Demmel, CS267 notes for lectures 15 and 16 (1996), file lecture24.html: Solving the discrete Poisson equation

  Table of serial time and storage of dense LU, band LU, Jacobi, sparse LU, CG, SOR, FFT, multigrid; Jacobi spectral radius cos(pi/(n+1)); no fill-in outside the band; FFT and multigrid only for Poisson-like equations.

• Shewchuk, An introduction to the conjugate gradient method without the agonizing pain (1994)

  p. 33: CG minimizes the energy norm of the error; neglect of CG in the 1960s and revival in the 1970s as an iterative method. p. 37 (section 10): iteration bound ceil(0.5 sqrt(kappa) ln(2/eps)). p. 38: O(m sqrt(kappa)); O(n^(3/2)) in 2-D and O(n^(4/3)) in 3-D.

• Hestenes and Stiefel, Methods of conjugate gradients for solving linear systems (1952)

  Abstract: an iterative algorithm giving the solution in n steps, a special case of a method that includes Gaussian elimination; introduction: Lanczos developed a closely related method.

• Turing, On computable numbers (1936)

  Received 28 May 1936, read 12 November 1936; the machine with a tape of squares bearing symbols; arguments that computable numbers include all numbers naturally regarded as computable.

• von Neumann, First Draft of a Report on the EDVAC (1945), Godfrey reprint

  Title page: dated 30 June 1945; contract between the Army Ordnance Department and the University of Pennsylvania.

• MacTutor: Aleksei Krylov

  Father an artillery officer; fifty years of teaching at the naval academy; General and Chief Naval Ship Construction Inspector in January 1908; medal of the Royal Institution of Naval Architects. Captain of the Admiralty and acting superintendent of the Towing Tank on 3 January 1900.

• Wikipedia: Aleksey Krylov (secondary)

  Imperial Russian Navy; the 1931 paper on the characteristic polynomial (Izv. Akad. Nauk SSSR 7(4) 491-539).

• MacTutor: Boris Galerkin

  Soviet mathematician and engineer (1871-1945); locomotive and boiler plant engineer; Petersburg Technological Institute; method dated 1915; no rank recorded.

• Science Museum Group: Charles Babbage's Difference Engines

  Difference Engine No. 1 calculated and printed series of values by finite differences using addition and subtraction; one seventh completed 1832; work halted 1833; 17,500 pounds spent.

• Weik, The ENIAC Story (1961)

  Name; dedication 15 February 1946; 19,000 vacuum tubes and almost 200 kW; 200 microsecond addition time, 2.6 ms multiplication and 25 ms average division for 10-digit numbers; firing and bombing tables at the Ballistic Research Laboratory; differential analyzer with ten integrating units and its trajectory runs.

• Moore, Cramming more components onto integrated circuits (1965), reprint Proc. IEEE 86(1) 1998

  Complexity at minimum component cost increased by roughly a factor of two per year for at least ten years; 65,000 components per circuit by 1975.

• Dennard et al., Design of ion-implanted MOSFET's with very small physical dimensions (1974), National Academies reprint

  Scaling factors of Table 1: dimensions, voltage, current, capacitance and delay by 1/s (kappa in the paper), power per circuit by 1/s^2, power density constant; the subthreshold characteristic does not scale.

• EIA, How much electricity does an American home use?

  Average annual electricity purchase of a U.S. residential customer in 2022: 10,791 kWh.

• TOP500, Frontier system page

  Nmax = 24,837,120 of the Linpack run for the current entry (Rmax 1,353 PFlop/s, Rpeak 2,055.72 PFlop/s); HPCG 14,054.0 TFlop/s.

• Wikipedia: Boris Galerkin (secondary)

  In 1939 Galerkin became head of the structural mechanics department of the Navy's engineering school (VITU) and a lieutenant general, without earlier army service.

• D. M. Ritchie and K. Thompson, The UNIX Time-Sharing System (BSTJ 57(6), 1978 revision of CACM 17(7), 1974)

  Section I: earliest version circa 1969-70 on PDP-7/-9, second version on the unprotected PDP-11/20, third with multiprogramming; section 3.3 special files read and written like ordinary files; section V image and process definitions; 5.1 fork, 5.2 pipes, 5.3 execute, 5.5 exit/wait; section VI shell as command-line interpreter.

• L. Torvalds, Linux's History (31 July 1992, mirror at CMU)

  Post of 25 August 1991 ('just a hobby'); version 0.01 'sometimes around this time'; GNU programs running under the kernel (5 October 1991); early licence stricter than the GNU copyleft, changed with 0.12, out 5 January 1992.

• kernel.org, Linux kernel v1.0 archive

  linux-1.0.tar.gz listed with date 13 March 1994.

• Tanenbaum-Torvalds debate, Open Sources appendix A

  Tanenbaum 29 Jan 1992: monolithic kernel is a single a.out running in kernel mode; microkernel runs most of the OS as separate processes with message passing; performance argument; Torvalds: 'True, linux is monolithic, and I agree that microkernels are nicer'.

• Linux kernel documentation: Kernel Entry (x86 entry code)

  Entry points: syscall instruction, int 0x80, interrupts, architecturally defined exceptions.

• Linux kernel documentation: Page Table Isolation

  Section 23.3 Overhead: CR3 moves at interrupt, syscall and exception entry and exit, on the order of a hundred cycles each.

• Linux kernel documentation: x86-64 memory map

  48- and 57-bit virtual addresses; 4-level and 5-level page tables.

• Linux kernel documentation: Concepts overview (memory management)

  Virtual memory primer (demand paging, protection, hierarchical page tables, top-level register, offset bits); TLB as a cache of translations; huge pages reduce TLB pressure.

• Linux kernel documentation: Overview of the Linux Virtual File System

  VFS is the software layer that provides the filesystem interface to user space and lets different filesystem implementations coexist; system calls open/stat/read/write are called from process context.

• Linux kernel documentation: The /proc filesystem

  VmPTE is the size of page table entries; Pss definition: pages divided by the number of processes sharing them.

• Linux kernel documentation: CFS Scheduler

  Ideal precise multi-tasking CPU; virtual runtime per task.

• Linux kernel documentation: EEVDF Scheduler

  Transition from CFS to EEVDF began in version 6.6; EEVDF aims to distribute CPU time equally among runnable tasks of the same priority.

• Linux kernel documentation: Scheduler Nice Design

  Multiplicative nice levels: nice 10 vs 11 split like -5 vs -4, one gets 55 % and the other 45 %; nice +19 gets 1.5 %.

• man-pages: syscall(2)

  x86-64: instruction syscall, number in rax, arguments rdi, rsi, rdx, r10, r8, r9, result in rax; errors stored in errno by the library.

• man-pages: fork(2)

  Separate memory spaces with equal content at fork; child has own PID and parent PID equal to the caller; copy-on-write implementation with the page-table duplication penalty.

• man-pages: execve(2)

  Replaces the program with newly initialized stack, heap and data segments; PT_INTERP interpreter loads needed shared objects.

• man-pages: sched(7)

  SCHED_OTHER standard time-sharing; SCHED_FIFO and SCHED_RR with static priorities 1-99 preempting SCHED_OTHER; all scheduling is preemptive; the scheduler schedules threads.

• man-pages: vdso(7)

  Purpose of the vDSO (avoid the user/kernel switch overhead for frequently used calls); __vdso_clock_gettime exported.

• POSIX.1-2024 (IEEE Std 1003.1-2024), fork

  The child shall be an exact copy of the parent except for listed attributes including a unique process ID; created with a single thread.

• KernelNewbies: Linux 6.12

  Release 17 November 2024; PREEMPT_RT merged after 20 years; EEVDF transition completed (secondary source).

• POSIX.1-2024 (IEEE Std 1003.1-2024), Shell and Utilities, chapter 2

  Section 2.9.1.4 Command Search and Execution: if the command name does not contain any slash, the search sequence applies (special built-ins, functions, utilities, PATH).

• Linux kernel source, kernel/sched/core.c, tag v6.6.87 (stable tree)

  Comment 'a multiplier of 1.25'; table sched_prio_to_weight: nice 0 = 1024, nice 5 = 335, nice 10 = 110, nice 19 = 15 (lines 11550-11571).

• Linux kernel source, mm/memory.c, tag v6.6.87 (stable tree)

  copy_present_pte calls page_try_dup_anon_rmap for anonymous pages (candidate per-page work of fork, Exercise 6).

• KernelNewbies: Linux 6.6

  Release 29 October 2023; the CFS process scheduler is replaced by EEVDF (secondary source).

• The UNIX Time-Sharing System (Ritchie and Thompson, CACM 17(7), 1974)

  A file does not exist within a directory; each directory has at least two entries; no links to directories and none between file system hierarchies; seven protection bits and set-user-ID; i-node fields; link creation and removal by changing the link count

• POSIX.1-2024 Base Definitions, chapter 4 General Concepts

  4.5 sticky-bit deletion conditions; 4.7 class-based access rule, privileged processes, additional and alternate mechanisms; 4.8 directed graph

• POSIX.1-2024 chmod

  owner or privileged may change the mode; symbolic-mode semantics; bits of the creation mask are excluded when who is omitted

• POSIX.1-2024 umask

  the file mode creation mask utility

• POSIX.1-2024 sys/stat.h

  names of the mode bits

• inode(7)

  inode fields; mode bit values; the owner of a new file is the effective user ID; meaning of the sticky bit on a directory

• path_resolution(7)

  resolution steps and errors; the three-class permission rule (owner, then group or supplementary group, then other)

• credentials(7)

  effective and supplementary IDs are used for file permission checks

• open(2)

  O_TRUNC, O_APPEND; mode and ~umask; the group of a new file depends on set-group-ID of the parent directory; EACCES for creation

• mkdir(2)

  mode & ~umask & 0777; owner and group of a new directory; set-group-ID inheritance; EEXIST, EACCES

• rmdir(2), link(2), unlink(2), rename(2)

  ENOTEMPTY; EPERM for directories; EXDEV across file systems; unlink deletes a name, and the file when the last link and last descriptor are gone; sticky-bit conditions; EACCES without write permission on the containing directory

• symlink(7)

  permissions of a symbolic link are not used and are always 0777 on Linux

• chmod(2) and chown(2)

  only a privileged process may change the owner; the owner may change the group to one of the owner's groups; set-user-ID and set-group-ID cleared on a change by an unprivileged user; chmod needs the owner or a privileged process

• free(1) (procps-ng)

  used is calculated as total - available

• login.defs(5) and /etc/login.defs of the verification machine

  UMASK is the initial file mode creation mask used by pam_umask (default 022); the comment in /etc/login.defs states that USERGROUPS_ENAB yes turns 022 into 002 for private user groups

• pam_umask(8)

  option usergroups: if the user is not root and the username equals the primary group name, the umask group bits are set equal to the owner bits (022 becomes 002, 077 becomes 007)

• Filesystem Hierarchy Standard 3.0

  shareable/static distinction; /home is site-specific; /tmp is not preserved; /var/tmp is preserved

• GNU Coreutils 9.4 manual (installed copy, info coreutils)

  ls -l modification time, file types, s S t T, the '.' security-context and '+' ACL markers; mv copy-and-remove across file systems; rm prompting and recovery warning; the umask exception of chmod; setgid directories and the digit rule of chmod

• Lecture W1L5 recording

  Process management, text processing commands.

• Lecture W1L6 recording

  Memory and networking commands.

• POSIX.1-2017 Base Definitions, chapter 9 Regular Expressions and section 3.206

  9.1 leftmost-longest matching rule; 9.3.5 range expressions specified only in the POSIX locale; BRE back-references; ERE grammar without back-references; 3.206 definition of a line (V1_chap03).

• POSIX.1-2017 Shell Command Language, 2.8.2, 2.9.2, 2.9.3

  Pipeline: stdout to stdin as if by creating a pipe; exit status of last command; shell may wait for all commands; asynchronous lists, stdin from /dev/null when job control is disabled, $!; exit status greater than 128 for a signal-terminated command.

• POSIX.1-2017 utilities awk, sort, uniq, grep

  awk arithmetic follows ISO C floating point; sort -n initial numeric string.

• GNU grep manual

  2.1.6 -r searches the working directory without a file operand; 2.3 exit status 0, 1, 2; 3.5 back-references; 3.6 BRE against ERE differences.

• GNU awk manual

  Chapter 1 pattern-action rules; 4.5.1 default field splitting; 7.1.4 BEGIN and END; uninitialized variables are zero.

• GNU coreutils manual: sort, uniq, nohup, df

  sort -n does not recognize exponential notation, -g converts a prefix to floating point; uniq detects only adjacent repeats; nohup ignores hangup and writes nohup.out; df reports file-system space.

• Owens, Reppy and Turon, Regular-expression derivatives re-examined, J. Funct. Program. 19(2) 2009, 173-190

  Definitions 2.1 and 2.2 (syntax, language); section 3.1 derivative rules due to Brzozowski; section 3.2 matching by derivatives.

• Cox, Regular Expression Matching Can Be Simple And Fast, 2007

  Backtracking against Thompson NFA on a?^n a^n; Thompson NFA O(mn); GNU grep and GNU awk build DFAs; back-references are not regular and need exponential search in the worst case; Thompson 1968 described.

• Linux man-pages: ps(1), top(1), kill(1), signal(7), wait(2), pipe(7), sched_setaffinity(2), proc_pid_stat(5), proc_pid_status(5), proc_meminfo(5), free(1), vmstat(8), swapon(8), ip(8), ping(8), ssh(1), scp(1)

  Process state codes; default ps selection; %CPU and %MEM; kill default SIGTERM; SIGKILL and SIGSTOP cannot be caught and signal numbers by architecture; zombie definition and NOTES; pipe capacity 65,536 bytes and SIGPIPE; affinity mask; stat state field; VmSize and VmRSS; MemAvailable since 3.14; free columns; vmstat, swapon, ip descriptions; ping default payload; ssh -G; scp uses SFTP.

• POSIX.1-2017 Base Definitions 11.1.10 (Modem Disconnect)

  SIGHUP is sent to the controlling process of the terminal.

• GNU Bash manual

  Signals: an interactive shell resends SIGHUP to all jobs, running or stopped. Exit Status: a command killed by a fatal signal N has status 128+N.

• POSIX.1-2024 Shell Command Language 2.9.2

  The pipefail option and the exit-status rule of a pipeline: zero if all commands returned 0, otherwise the status of the rightmost command that returned non-zero.

• proc_pid_statm(5) and clone(2)

  statm values are inaccurate; /proc/pid/smaps and smaps_rollup give accurate values; CLONE_THREAD makes the new task a thread of the caller process.

• Linux kernel documentation, The /proc filesystem

  MemAvailable estimate and its inputs.

• RFC 1122, Requirements for Internet Hosts

  3.2.1.3 (p. 31): {127, any} internal host loopback address must not appear outside a host.

• RFC 792, Internet Control Message Protocol

  Echo type 8, echo reply type 0 (p. 14); data of the echo must be returned in the reply (p. 15).

• RFC 4251, The Secure Shell (SSH) Protocol Architecture

  Section 1: transport layer, user authentication and connection protocols.

• OpenSSH 9.0 release notes

  scp switches from the legacy scp/rcp protocol to SFTP by default; the legacy protocol expanded remote wildcards through the remote shell.

• Demmel, CS267 notes for lectures 15 and 16 (1996), Solving the Discrete Poisson Equation using Jacobi, SOR, Conjugate Gradients, and the FFT

  Jacobi iteration; rho_Jacobi(n) = cos(pi/(n+1)); approximation 1 - 0.5 (pi/(n+1))^2; Gauss-Seidel rate rho squared.

• Lecture W2L1 recording

  Shell script definition, structure, command substitution, arithmetic, variables, env, export, unset, PATH, read.

• Lecture W2L2 recording

  if/elif/else, for, while, functions, factorial, disk monitoring script.

• POSIX.1-2017 Shell Command Language (Issue 7, 2018 edition)

  2.1 '#!' first line unspecified; 2.6.3 command substitution in a subshell environment, trailing newlines removed; 2.6.4 only signed long arithmetic required; 2.8.2 status 126/127, full eight bits, above 128 for signals; 2.9.1 command search, ENOEXEC fallback, assignments before a command exported for it only; 2.9.4 if/for/while over command lists; 2.9.5 functions; 2.12 execution environments, export attribute passed to utilities, subshell changes do not affect the shell, each pipeline command in a subshell environment; 2.14 export, return (value above 255 unspecified).

• POSIX.1-2017 test utility

  Operands: integer operators, = and != only for strings, evaluation by argument count, -f and -e; exit 0/1/>1; rationale that Korn-shell string comparisons were not carried into test.

• POSIX.1-2024 test utility (Issue 8)

  s1 > s2 and s1 < s2 (collation order of the current locale) are operators of test; rationale: they were later added once support became widespread.

• execvp(3) (exec(3) page), Linux man-pages

  If the attempted execve failed with ENOEXEC, these functions execute /bin/sh with the path of the file as its first argument.

• POSIX.1-2017 XBD chapter 8 Environment Variables

  PATH: searched beginning to end, first executable wins, zero-length prefix means the current directory.

• GNU Bash Reference Manual, version 5.3 (18 May 2025)

  3.2.3 pipelines in subshells and lastpipe; 3.5.5 and 6.5 arithmetic (fixed width, no overflow check, octal, variable evaluated as expression); 3.7.2 command search, hash table, script fallback; 3.7.3 subshell; 3.7.4 environment, automatic export of inherited variables, prefix assignments; 3.7.5 exit status, 128+N; 4.2 local return status, read options; 7.2 wait returns the status of the last id; 6.4 string comparison in [[.

• execve(2), Linux man-pages 6.19 (2026-02-08)

  Program replaced; envp is the environment of the new program; interpreter scripts need execute permission and a '#!interpreter [optional-arg]' first line and receive 'interpreter [optional-arg] path arg...'; on Linux the whole optional text is one argument; limit 255 characters since Linux 5.1, 127 before.

• ld.so(8), Linux man-pages 6.19 (2026-08-03)

  LD_LIBRARY_PATH: directories searched for ELF libraries at execution time; a zero-length entry means the current directory.

• GCC manual, Environment Variables

  CPATH, C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, OBJC_INCLUDE_PATH for header search; LIBRARY_PATH for link-time library search.

• Slurm Workload Manager, sbatch (version 26.05)

  #SBATCH directives are read by Slurm, literal text to the shell, end at the first non-comment line; --export default ALL loads the user's environment; NIL and NONE pass only SLURM_* and SPANK variables.

• Intel ARK, Core i5-12450H

  4 P-cores x 2 threads, 4 E-cores x 1 thread, 12 threads (as stated by the authoring brief).

• TOP500, Frontier system page

  June 2024 entry: Rmax 1,206.00 Pflop/s, Rpeak 1,714.81 Pflop/s, 8,699,904 cores, 22,786 kW; June 2026 entry: 9,066,176 cores, Rmax 1,353.00, Rpeak 2,055.72 Pflop/s, Nmax 24,837,120.

• TOP500 list June 2008

  Rank 1 Roadrunner 1,026 Tflop/s, rank 2 478.2 Tflop/s.

• TOP500 list June 2008, ranks 401 to 500

  Rank 500 Rmax 9.00 Tflop/s.

• TOP500 list June 2019, ranks 401 to 500

  Rank 500 Rmax 1.02 Pflop/s.

• TOP500 list June 2026

  Rank 1 LineShine 2,198.40 Pflop/s; Frontier rank 3 with 1,353.00 Pflop/s.

• TOP500 list June 2026, ranks 401 to 500

  Rank 500 Rmax 2.66 Pflop/s.

• TOP500 Linpack project page

  Rmax, Nmax, N1/2, Rpeak; LU with partial pivoting; operation count 2/3 n^3 + O(n^2); exclusion of Strassen-type and lower-precision methods.

• HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers, version 2.3

  64-bit dense solver; two-dimensional block-cyclic distribution; needs MPI and BLAS.

• US Army Ordnance, Historical Monograph: Electronic Computers within the Ordnance Corps (1961), chapter II The ENIAC

  Final assembly fall 1945; dedication 15 February 1946; acceptance July 1946; Aberdeen January 1947; operational August 1947; Ordnance Department as sponsor.

• Slurm Workload Manager, Overview

  Three key functions: allocation, framework for starting and monitoring work, arbitration of contention by a queue.

• OpenHPC

  Community effort aggregating provisioning, resource management, I/O clients, development tools and scientific libraries for HPC Linux clusters.

• Lustre Software Release 2.x Operations Manual

  Section 1.2.2: MDS, MDT, OSS, OST, clients, several clients may write different parts of a file; 1.3: aggregate bandwidth is the minimum of disk and network bandwidth; 1.3.1: striping as RAID 0, default stripe_count 1 and stripe_size 1 MB.

• M. J. Flynn, Very high-speed computing systems, Proc. IEEE 54(12), 1901-1909 (1966)

  Four classes SISD, SIMD, MISD, MIMD; definition of a stream.

• M. J. Flynn, Some computer organizations and their effectiveness, IEEE Trans. Comput. C-21(9), 948-960 (1972)

  SISD limited by data dependencies, Minsky conjecture for SIMD, saturation of MIMD at lockout fraction 1/n.

• Intel 64 and IA-32 Architectures Optimization Reference Manual, Documentation Changes, 355308-051 (October 2026)

  E-cores support AVX2 and AVX-VNNI, not AVX-512; symmetric ISA; P-cores can have Hyper-Threading; Golden Cove 48 KB data cache; multiply/FMA 2x256-bit; FMA latency 4 (ports 0,1).

• Notebookcheck, Intel Core i5-12450H

  4 P-cores and 4 E-cores, 12 threads; P-core clock 2.0 to 4.4 GHz, E-core 1.5 to 3.3 GHz.

• uops.info instruction table (instructions.xml)

  Measured latency, reciprocal throughput and micro-operation counts of VFMADD231PD (YMM) on Alder Lake P-cores (latency 4, 0.50, ports 0 and 1) and E-cores (latency 6, 1.00, 2 micro-operations); VADDPD and VMULPD forms.

• Intel M80C286 High Performance CHMOS Microprocessor with Memory Management and Protection (data sheet)

  Designed for multi-tasking; four-level protection for task isolation; hardware task switching; one instruction switches tasks.

• D. T. Marr et al., Hyper-Threading Technology Architecture and Microarchitecture, Intel Technology Journal 6(1) (2002)

  Two logical processors, shared execution resources, duplicated architecture state, simultaneous execution; die area under 5 percent; gains up to 30 percent on common server benchmarks.

• Linux man-pages: pthreads(7)

  Threads share global memory (data and heap), each has its own stack.

• J. D. C. Little, A proof for the queuing formula L = lambda W, Operations Research 9(3), 383-387 (1961)

  Under stationarity, mean number in system equals arrival rate times mean time in system.

• R. P. Brent, The parallel evaluation of general arithmetic expressions, J. ACM 21(2), 201-206 (1974)

  Expression evaluation in time 4 log2 n + 10(n-1)/p on p processors.

• Slurm Workload Manager, Multi-core/Multi-thread Support

  Job option --hint=[no]multithread ('[don't] use extra threads with in-core multi-threading'); in-core multi-threading 'may also improve performance' for communication-intensive applications.

• U. Drepper, What every programmer should know about memory, version 1.0 (2007)

  SRAM and DRAM cell structure and cost (pp. 5-6); 64-byte lines, address split, dirty line, coherence rules, Pentium M cycle counts (pp. 15-16); associativity and Table 3.1 (pp. 17-19); write policies (p. 25); MESI and shared-bus slowdowns (pp. 25-27); NUMA factor, Altix, clusters are not NUMA, 2007 default striping (pp. 42-44)

• P. J. Denning, The Locality Principle, Communications of the ACM 48(7), 2005, 19-24

  Locality defined since 1980 by a distance D(x,t) <= T with temporal and spatial distances; working set W(t,T)

• Y. H. Kim, M. D. Hill, D. A. Wood, Implementing stack simulation for highly-associative memories (University of Wisconsin-Madison, 1991)

  Mattson et al.'s stack simulation for LRU memories; the first k stack entries give the blocks of a memory of k blocks; definition of stack distance

• R. L. Mattson, J. Gecsei, D. R. Slutz, I. L. Traiger, Evaluation techniques for storage hierarchies, IBM Systems Journal 9(2), 1970, 78-117

  Origin of the stack property used in Theorem 1 (cited through Kim, Hill and Wood)

• P. H. J. Kelly, Advanced Computer Architecture, chapter 4 part 1, Imperial College London (lecture notes dated October 2026)

  AMAT = HitTime + MissRate x MissPenalty (citing Hennessy and Patterson 6th ed. appendix B.3, p. B22); compulsory, capacity and conflict misses

• W. A. Wulf, S. A. McKee, Hitting the memory wall: implications of the obvious, ACM SIGARCH Computer Architecture News 23(1), 1995, 20-24

  Title cited as the origin of the term memory wall; bibliographic data checked in Crossref

• A. Ertl, The Memory Wall Fallacy

  Description of the Wulf and McKee argument as assuming a fixed miss probability, and its criticism

• The Linux kernel documentation, NUMA memory policy

  System default policy once running is local allocation (allocation on the node of the CPU where it takes place); interleave during boot

• M. Al-Fares, A. Loukissas, A. Vahdat, A scalable, commodity data center network architecture, ACM SIGCOMM 2008, 63-74

  Oversubscription defined against bisection bandwidth (section 2.1, p. 64); a k-port fat tree supports k^3/4 hosts (section 3, p. 66); 48-port switches, 27,648 hosts at full bisection bandwidth (abstract)

• SGI, Altix ICE 8400 press text, reprinted by Scientific Computing World (undated page)

  Scale-out blade platform; quad data rate InfiniBand; up to 128 processors (1,536 cores) per cabinet; up to 65,536 compute nodes

• R. Wolff (SGI), SGI Altix Ultraviolet, slides (undated)

  UV 1000 with single system image of up to 256 sockets and 16 TB shared memory over NUMAlink 5; up to 32768 sockets in the hybrid configuration; hub offload of MPI and UPC

• Intel Core i5-12450H processor specifications (ARK)

  8 cores (4 performance, 4 efficient), 12 threads, 12 MB Intel Smart Cache, at most 2 memory channels, supported memory types, launch Q1 2022

• ServeTheHome, Intel 12th Gen Core Alder Lake details at Intel Innovation 2021

  Each performance core has 1.25 MB of L2 cache; each cluster of four efficient cores shares 2 MB of L2 (used only to list core-type placement as a candidate cause)

• ISO/IEC 9899:2011 committee draft N1570

  5.1.2.4 p4 p.17 conflict; p7 p.18 modification order; p11 p.18 release/acquire (synchronizes with); p16 and p18 p.19 happens before; p13 NOTE 6 p.18 mutex order; p25 p.20 data race; 7.17.3 p.275; 7.17.7.4 pp.283-284; 7.17.7.5 pp.284-285; 7.17.8.1 p.285

• POSIX Base Definitions Issue 7 (2018), 4.12

  list of memory-synchronizing functions; simultaneous reads allowed

• POSIX pthread_mutex_lock

  ownership; relock and unlock table; EDEADLK for error-checking relock

• POSIX pthread_barrier_wait

  blocks until the required threads arrive; PTHREAD_BARRIER_SERIAL_THREAD; reset

• OpenMP API 5.2 (November 2021)

  critical pp.299-300; barrier pp.301-302; atomic pp.311-312; weak clause note 15.8.3 p.311; implicit barrier of a parallel region p.224

• Lamport, How to make a multiprocessor computer that correctly executes multiprocess programs (1979)

  definition of sequential consistency; two-process mutual exclusion protocol failing under reordering

• Dijkstra, Cooperating sequential processes, EWD123

  mutual-exclusion problem and conditions; rejected trials; Dekker solution

• Peterson, Myths about the mutual exclusion problem (1981)

  bibliographic record only

• Sewell et al., x86-TSO (CACM 53(7), 2010)

  SB test; store buffers, MFENCE, locked instructions, XCHG implicitly locked

• Drepper, Futexes are tricky (v1.6, 2011)

  three-state futex mutex with cmpxchg, xchg and futex calls

• Mellor-Crummey and Scott, Algorithms for scalable synchronization (TOCS 9(1), 1991)

  abstract; 2.4 p.29 the MCS lock guarantees FIFO with compare-and-swap; Fig. 5 and p.30 swap-only variant very nearly FIFO in practice; centralized and sense-reversing barrier 3.1, Fig. 8

• Chandy and Lamport, Distributed snapshots (TOCS 3(1), 1985)

  global state during a computation, checkpointing use; the algorithm does not alter the computation

• Tian and Shih, Software techniques for shared-cache multi-core systems (Intel, 2012)

  hold approach with OpenMP reduction; contention-only threads; false sharing

• Coffman, Elphick, Shoshani, System deadlocks (1971)

  four deadlock conditions (attribution only)

• Intel Core i5-12450H specifications (ARK)

  4 P-cores x 2 threads, 4 E-cores x 1 thread, 12 threads

• Lecture W3L1 recording

  Moore's law, Dennard scaling, plateau, parallel computing slide, bag of integers, Amdahl's law.

• Lecture W3L2 recording

  Numerical modelling and errors, plate problem, discretization, system layers, ISA and ABI.

• Moore, Cramming more components onto integrated circuits, Electronics 38(8), 1965

  Complexity for minimum component cost rose by roughly a factor of two per year; expected to continue for at least 10 years; 65,000 components in 1975; author at Fairchild Semiconductor.

• Moore, Progress in digital integrated electronics, IEDM 1975

  Complexity approximately doubled every year since introduction; a slope that might approximate a doubling every two years by the end of the decade.

• Dennard et al., Design of ion-implanted MOSFET's with very small physical dimensions, IEEE JSSC SC-9(5), 1974

  Delay per circuit falls by K, power per circuit by K^2, power density constant (text beside Table I).

• Bohr, A 30 year retrospective on Dennard's MOSFET scaling paper, IEEE SSCS Newsletter, Winter 2007

  0.7x feature size per generation, 2x density; 2x per 18 months in Intel transistor counts in the 1970s-1980s, every 2 years since the mid-1990s; sub-threshold leakage limits threshold and supply voltage scaling.

• Rupp, microprocessor-trend-data (50yrs/transistors.dat, frequency.dat)

  99 transistor counts and 101 clock frequencies, 1971-2021; fits of doubling times.

• Amdahl, Validity of the single processor approach..., AFIPS 1967 (2007 reprint)

  Housekeeping 40 percent; overhead limits throughput to five to seven times; no equations.

• Gustafson, Reevaluating Amdahl's law, CACM 31(5), 1988

  Scaled speedup N + (1-N)s' (credited to E. Barsis); speedups 1021, 1020, 1016 on 1024 processors, serial fractions 0.4-0.8 percent; criticism of reading s as the fixed-size fraction.

• Goldberg, What every computer scientist should know about floating-point arithmetic, ACM Computing Surveys 23(1), 1991

  Machine epsilon (beta/2) beta^-p bounds the relative rounding error.

• Saad, Iterative Methods for Sparse Linear Systems, 2nd ed., SIAM 2003

  Theorem 4.1 (p. 115): convergence iff spectral radius below 1; Theorem 4.16 (p. 126): SOR-Jacobi eigenvalue relation; sec. 13.2 pp. 424-427: eq. 13.5 and 13.14, eigenvalues of the 1-D and 2-D model problems.

• Demmel, CS 267 Lecture 24, 1995

  Table of serial costs of Poisson solvers; rho_Jacobi = cos(pi/(n+1)) in the 2-norm; Gauss-Seidel in row order not straightforwardly parallelizable.

• NIST CODATA, Boltzmann constant

  k_B = 1.380649e-23 J/K, exact.

• GCC manual, Optimize Options and x86 Options (online, 2026-10-26)

  -ftree-parallelize-loops possible only for independent iterations, profitable for CPU-bound loops; -march code may not run on other processors (x86 Options page).

• System V ABI, AMD64 Architecture Processor Supplement, draft 0.99.6

  Scope (sec. 1, p. 9); argument passing and return of values (sec. 3.2.3); user-level integer register sequence (A.2.1).

• Intel Core i5-12450H specifications (ARK) and TechPowerUp

  4 P-cores x 2 threads + 4 E-cores x 1 thread; turbo 4.40 / 3.30 GHz.

• TechPowerUp, Intel Core i5-12450H

  4 performance cores and 4 efficient cores, 12 threads, turbo up to 4.4 GHz (performance) and 3.3 GHz (efficient).

• GCC manual, x86 Options (online, 2026-10-26)

  -march=cpu-type: code 'may not run at all on processors other than the one indicated'; -march=native: 'the result might not run on different machines'.

• Lecture W3L3 recording

  Memory hierarchy, paths, virtual memory, paging, terminology.

• Lecture W3L4 recording

  Shared and distributed memory, CPU vs GPU, Frontier, Spacetime2.

• Lecture W3L5 recording

  Grand challenges, Moore, energy, Grace, Hopper, superchip, ABINIT.

• Wulf and McKee, Hitting the memory wall: implications of the obvious (1995)

  Name and statement of the memory wall (as reported by Machanick).

• Machanick, Approaches to addressing the memory wall (2002)

  Definition of the memory wall; CPU 50 % per year against DRAM 7 % per year; gap 1.4 per year, doubling in 2.1 years.

• Ertl, The Memory Wall Fallacy

  Fixed miss rate assumption; longer lines; concurrent misses.

• McCalpin, STREAM reference information

  Array size rule, best of trials, byte counting conventions, write allocate, 16/16/24/24 bytes and 0/1/1/2 flop.

• Williams, Waterman, Patterson, Roofline (UCB/EECS-2008-134; CACM 52(4) 2009)

  Operational intensity; Min(peak, bandwidth x intensity); ridge point; ridge points 6.7 and 4.4; 7-point stencil 8 flop per 24 bytes, 0.33.

• Horowitz, Computing's energy problem (ISSCC 2014)

  DRAM access 1-2 nJ against about 10 pJ per cache access or operation; DRAM I/O above 20 pJ per bit; 10 pJ per bit = 0.6 nJ per 8 B.

• Kim, Dally, Scott, Abts, Technology-driven, highly-scalable dragonfly topology (ISCA 2008)

  p, a, h; radix; virtual router; g = ah+1, N = ap(ah+1); balance a = 2p = 2h; one global hop; cost 20 % and 52 % lower.

• NVIDIA GH200 Grace Hopper Superchip datasheet (March 2025)

  72 Neoverse V2 cores; caches; LPDDR5X sizes and bandwidths; FP64 34 TFLOP/s, FP64 tensor 67; HBM3 4 TB/s; NVLink-C2C 900 GB/s; NVL2 column.

• NVIDIA GH200 Grace Hopper Superchip Architecture whitepaper v1.21

  72 cores, 500 GB/s, 3.2 TB/s bisection (p. 6); Table 1 (p. 7); 1.3 pJ per bit (p. 13); coherency, NUMA (p. 18); ABINIT Figure 19 (p. 30).

• NVIDIA GH200 product page

  900 GB/s coherent interface; one shared page table per process.

• OLCF Frontier User Guide

  77 cabinets, 9 856 nodes; about 2 EFLOP/s; node design; 23.9 TFLOP/s per GCD; 36+36 GB/s; 4 NICs of 200 Gbit/s; 679 PB Orion.

• Glenski (HPE), Frontier System Architecture, OLCF, 15 February 2023

  74 cabinets, 9 408 nodes; 205 GB/s DDR4; 26.5 TFLOP/s per GCD; switch links 16/31/16; 74 groups; caption of 4 links per group pair.

• TOP500, Frontier system page

  Rmax, Rpeak, cores, power June 2022 to June 2026; HPCG 14 054.0 TFLOP/s; ranks.

• Lecture W3L6 recording

  OpenMP overview, syntax, hello world, concurrency definitions, fork-join, thread count and thread number.

• Lecture W4L1 recording

  Parallel regions, pi by the midpoint rule, SPMD with cyclic and block distribution, timing table, speedup and Amdahl's law, overhead.

• OpenMP Application Programming Interface, Version 5.2 (November 2021)

  p. 23-24: fork-join, implicit barrier, nested region with one thread when nesting is disabled, different numbers of threads may give different numeric results; p. 27: update granularity, definition of data race, result unspecified; p. 69: _OPENMP is yyyymm; p. 76: structured blocks; p. 97: automatic variables declared inside the construct are private; p. 99: variables referenced in a parallel construct without default clause are shared; p. 223: team creation, thread numbers consecutive from zero, team size constant; p. 226-227: Algorithm 2.1 for the number of threads; p. 348-349: omp_set_num_threads, omp_get_num_threads; p. 413-414: omp_get_wtime per-thread time; p. 600: OMP_NUM_THREADS sets nthreads-var.

• Amdahl, Validity of the single processor approach to achieving large scale computing capabilities (AFIPS 1967; 2007 IEEE SSCS News reprint)

  Housekeeping is 40 percent of executed instructions, might be halved but not reduced by a factor of three; overhead alone limits throughput to five to seven times; the reprint's editors state the paper has no equations and one figure.

• Dagum and Menon, OpenMP: an industry standard API for shared-memory programming, IEEE Computational Science & Engineering 5(1), 1998, pp. 46-55

  Abstract: OpenMP is a set of compiler directives and callable runtime library routines extending Fortran (and separately C and C++) to express shared-memory parallelism; it builds on X3H5 concepts and extends them to coarse-grain parallelism; the standard includes a runtime library with environment variables.

• Menon, OpenMP Application Program Interface, slides, Cray User Group 1998

  Slide 5: Fortran v1.0 specification October 1997, C/C++ v1.0 specification August 1998.

• OpenStax, Calculus Volume 2, section 3.6 Numerical Integration

  Theorem 3.5, eq. 3.12: for f continuous with a second derivative bounded by M on [a,b], the midpoint-rule error is at most M(b-a)^3/(24 n^2).

• Drepper, What every programmer should know about memory (2007)

  p. 15 (section 3.2): cache lines of 64 bytes are the norm; pp. 65-67 (section 6.4.1): writes to a shared line need exclusive state and many request-for-ownership messages; false sharing measured at 390%, 734% and 1,147% (one line against one line per thread, 2-4 threads) on four Pentium 4 processors; no growing penalty on a single-processor quad-core Core 2.

• Intel, Intel Core i5-12450H processor specifications (ARK)

  Total cores 8: 4 performance cores (2 threads each) and 4 efficient cores (1 thread each); 12 threads; maximum turbo frequency 4.40 GHz (P-core) and 3.30 GHz (E-core); launched Q1 2022.

• GNU libgomp manual, GOMP_SPINCOUNT

  A waiting thread spins 300,000 times when OMP_WAIT_POLICY is undefined (30 billion for ACTIVE, 0 for PASSIVE); if there are more OpenMP threads than available CPUs, 1000 spins (ACTIVE) and 100 spins (undefined).