Cleve, who was born in the Swedish capital Stockholm, became assistant professor of chemistry at the University of Uppsala in 1868 and was later made professor of general and agricultural chemistry there. He is mainly remembered for his work on the rare earth elements.

In 1874 Cleve concluded that didymium was in fact two elements; this was proved in 1885 and the two elements named neodymium and praseodymium. In 1879 he showed that the element scandium, newly discovered by the Swedish chemist Lars Nilson (1840–1899), was in fact the eka-boron predicted by Dmitri Mendeleev in his periodic table. In the same year, working with a sample of erbia from which he had removed all traces of scandia and ytterbia, Cleve found two new earths, which he named holmium, after Stockholm, and thulium, after the old name for Scandinavia. Holmium in fact turned out to be a mixture for, in 1886, Lecoq de Boisbaudran discovered that it also contained the new element dysprosium.

Cleve is also remembered as the teacher of Svante Arrhenius.