Rabi and his parents emigrated to America from Rymanow in Poland, where he was born, while he was still young. He subsequently grew up in a Yiddish-speaking community in New York, where his father ran a grocery store. He was educated at Cornell, graduating in 1919, and Columbia, where he obtained his PhD in 1927. After two years in Europe he returned to Columbia where he spent his whole career until his retirement in 1967, being appointed professor of physics in 1937 and the first University Professor (a position with no departmental duties) there in 1964.

While in Germany (1927) Rabi had worked under Otto Stern and was impressed with the experiment Stern had performed with Walter Gerlach in which the use of molecular beams led to the discovery of space quantization (1922). Consequently Rabi began a research program at Columbia where he invented the atomic- and molecular-beam magnetic-resonance method of observing atomic spectra, a precise means of determining the magnetic moments of fundamental particles. Using his techniques after World War II, experimentalists were able to measure the magnetic moment of the electron to nine significant figures, thus providing a powerful tool for the testing of theories in quantum electrodynamics. The method had wide applications to the atomic clock, to nuclear magnetic resonance, and to the maser and laser. For this work Rabi was awarded the Nobel Prize for physics in 1944.

During the war Rabi worked on the development of microwave radar. In the postwar years he was a member of the General Advisory Committee of the Atomic Energy Commission, serving as its chairman (1952–56) following the resignation of J. Robert Oppenheimer. As a member of the American delegation to UNESCO he originated the movement that led to the foundation of the international laboratory for high-energy physics in Geneva known as CERN.