Von Klitzing was born at Poznan but when the town was restored to Poland after 1945 his family moved to West Germany. He was educated at the universities of Brunswick and Würzburg, where he received his PhD in 1972 and where he remained as a teaching fellow until 1980. After Klitzing served as professor of physics at the Technical University, Munich, from 1980 to 1984, he was appointed director of solid-state research at the Max Planck Institute, Stuttgart.

In 1980 von Klitzing began work on the Hall effect, first described by Edwin Hall in 1879. Hall noted that when a current flows in a conductor placed in a magnetic field perpendicular to the sample's surface, a potential difference (the Hall voltage) is produced acting at right angles to both the current and field directions. It is possible to measure a Hall resistance, defined in the normal way by dividing the Hall voltage by the current it produces.

Von Klitzing set out to make extremely precise measurements of the Hall resistance working with a two-dimensional electron gas. This can be formed by using a special kind of transistor in which electrons can be drawn into a layer between an insulator and a semiconductor. When the layer is thin enough, of the order of 1 nanometer (10^–9 meter), and the temperature is as low as 1.5 K, the electrons are forced into a two-dimensional plane parallel to the surface of the semiconductor.

Under normal conditions the Hall resistance increases directly with the strength of the magnetic field. In contrast von Klitzing found that under his experimental conditions the resistance became quantized, varying in a series of steps as the magnetic field was changed. Von Klitzing went on to establish that the Hall resistance at each step was a function of Planck's constant h, the fundamental constant of quantum theory, and could be used to measure h very accurately. For his discovery of the quantized Hall effect von Klitzing was awarded the 1985 Nobel Prize for physics.