Chapter 1: Memory and caches

A modern core can add two numbers in a fraction of a nanosecond. Fetching a number it does not already hold from main memory takes around a hundred nanoseconds: long enough for several hundred additions. Almost everything about how fast a program runs today follows from that gap, and from the machinery built to hide it.

That machinery is a hierarchy of caches: small, fast memories close to the core that keep copies of what the program has used recently, so that most loads never travel to main memory at all. Caches work because programs are predictable in one particular way: they tend to use again what they just used, and to use next what sits beside it. When a program keeps that promise, it runs at the speed of the cache. When it breaks it, it runs at the speed of memory, which can be a hundred times slower, and nothing in the source code says which is happening.

This chapter is about seeing which is happening.