Born in Tuscaloosa, Alabama, Van de Graaff studied engineering at the University of Alabama, gaining his BS in 1922 and his MS in 1923. He enrolled in 1924 at the Sorbonne in Paris where he was inspired by the lectures of Marie Curie to study physics. In 1928 he obtained a PhD from Oxford University for research into the motion of ions in gases. It was during these studies that he conceived of an electrostatic generator that could radically improve on existing types, such as the Wimshurst machine, by building up electric charge on a hollow insulated metal sphere. A year later he returned to America and started working as a research fellow at Princeton. In 1931 he moved to the Massachusetts Institute of Technology as a research associate, serving as associate professor of physics from 1934 until he resigned in 1960.

While at Princeton he constructed, in 1931, the first model of his generator, now known as the Van de Graaff generator. The charge was carried to the hollow sphere by means of an insulated fabric belt and once transferred could accumulate on the outer surface of the sphere, leading ultimately to potentials of 80,000 volts. This was eventually increased to over a million volts.

At MIT Van de Graaff developed the generator for use as a particle accelerator. This Van de Graaff accelerator used the generator as a source of high voltage that could accelerate charged particles, such as electrons, to high velocities and hence high energies. It was thus to be a major tool in the developing fields of atomic and nuclear physics. One of Van de Graaff's aims was to explore the possibility of uranium fission and to try to create elements with larger atoms than uranium.

In collaboration with John Trump, an electrical engineer, he adapted the generator to produce high-energy x-rays, which could be used in the treatment of cancer. The first x-ray generator began operation in a Boston hospital in 1937. During World War II Van de Graaff was director of the radiographic project of the Office of Scientific Research and Development in which the generator was developed for another use: the examination of the interior structure of heavy ordnance by means of x-rays.

In 1946 Trump and Van de Graaff formed the High Voltage Engineering Corporation to market Van de Graaff accelerators and x-ray generators to hospitals, industry, and scientific research establishments. Van de Graaff was director and chief physicist and in 1960 left MIT to work there full time as chief scientist.