Average Memory Access Time (AMAT)

Every memory reference pays the hit time. The fraction that miss pay the penalty on top. That gives $\mathit{AMAT} = t_h + m \cdot t_p$, and the standard illustration is a 1-cycle cache with a 2% miss rate and a 100-cycle penalty: $1 + 0.02 \ imes 100 = 3$ cycles. The cache hits 98% of the time and still triples the average access. Two percent of accesses are producing two thirds of the cost.

Once you see that, the design conclusions invert. Halving the hit time from 1 cycle to 0.5 takes AMAT from 3 to 2.5. Halving the miss rate from 2% to 1% takes it from 3 to 2. The rare expensive event dominates, which is why processors grew second and third-level caches instead of simply making the first one faster: an L2 replaces a 100-cycle trip to DRAM with a 12-cycle trip, and the formula nests, with the L2's own AMAT becoming the L1's miss penalty.

The number to distrust is the miss rate, because there are two of them. Local miss rate is misses divided by accesses to that cache; global miss rate is misses divided by all CPU references. An L2 with a 40% local miss rate sounds terrible and may be excellent, since it is only seeing the 2% of traffic L1 already rejected. Quoting one where the other belongs is the standard way to make a cache hierarchy look better or worse than it is. It is also why this page works in clock cycles rather than nanoseconds: the cycle count is the property of the design, and it survives the next clock bump.