Croll, the son of a stonemason and crofter from Cargill in Scotland, started work as a millwright. He became caretaker at Anderson's College, Glasgow, in 1859, and was later made resident geologist in the Edinburgh office of the Geological Survey, where he remained until his retirement in 1880.

In 1864 he studied the work of A. J. Adhemar and began research into the idea of an astronomical causation of ice ages. He developed the theory that the answer lay in the orbital history of the Earth. Using work done by Urbain Leverrier in 1843, he found that the degree of eccentricity had been subjected to substantial change – 100,000 years ago it was highly eccentric while 10,000 years ago its eccentricity was quite small. He concluded that if winter occurred when the Earth was furthest away from the Sun in its precessional cycle and if the orbit of the Earth was at its most eccentric, then the two factors would produce an ice age. He followed Adhemar in seeing this as alternating between the two hemispheres and having a period of about 26,000 years.

This view was generally accepted by other geologists, notably by James Geikie in his pioneering work The Great Ice Age (1874–84), but tests made on the theory were too rudimentary to be conclusive.

Croll's work was published in his Climate and Time (1875) and Climate and Cosmology (1885).