Mottelson, who was born in Chicago, Illinois, graduated from Purdue University in 1947 and gained his PhD in theoretical physics at Harvard University in 1950.

From Harvard, Mottelson gained a traveling fellowship to the Institute of Theoretical Physics in Copenhagen (now the Neils Bohr Institute). There he worked with Neils Bohr's son, Aage Bohr, on problems of the atomic nucleus. In particular, they considered models of the nucleus and combined the two principal theories current at the time – one based on independent particles regarded as arranged in shells and the other treating the nucleus as a collective entity exhibiting liquid-drop- like behavior – and advanced a unified theory. They worked out the consequences of the interplay between the individual particles and the collective motions, specified the structure of the rotational and vibrational excitations and the coupling between them, and showed how the collective concepts could be applied to the nuclei of various elements. For their work on nuclear structure Mottelson, Bohr, and James Rainwater (Bohr's earlier collaborator at Columbia University) shared the 1975 Nobel Prize for physics.

Mottelson held a research position in CERN (the European Center for Nuclear Research) from 1953 until 1957, then returned to Copenhagen to take up a professorship at the Nordic Institute for Theoretical Atomic Physics (NORDITA) adjacent to the Neils Bohr Institute. He took Danish nationality in 1973.

Together with Aage Bohr, he has published Nuclear Structure (2 vols. 1969–75).