Modern hard drives contain read and write buffers to smooth the flow
of data to and from the disk surfaces. While these are beneficial, the
write buffer can also introduce unexpected delays. This directly affects
the 99th-percentile latency of latency-sensitive
reads when reads and writes are intermixed. We discuss simple ways to
measure from a C program what is happening at the heads of a disk, and a
non-obvious technique for speeding up reads without slowing down writes.
The same simple programs can measure SSD behavior. The same insights
apply to understanding possible excessive web-access latency in your
cable modem.

The subtext of the talk is that there is no need to accept unknown
hardware or software as "black boxes" that cannot be further understood.
Well-designed simple experiments can lead to substantial insight.
Some thought required.