Haldane, the son of a lawyer, was educated at the university in his native city of Edinburgh, where he obtained his MD in 1884. He worked first at the University of Dundee but moved to Oxford to assist his uncle, John Burdon-Sanderson, the professor of physiology in 1887. Haldane was made reader in physiology in 1907 but resigned in 1913 to become director of the Mining Research Laboratory, initially sited in Doncaster but transferred to Birmingham in 1921.

From the beginning of his career Haldane sought to apply the results of physiological research to the solution of practical social and industrial problems. He was much concerned with problems of ventilation in mines and in 1896 published an important report, Causes of Death in Colliery Explosions. He was struck by the fact that in a serious explosion in 1896 only 4 out of 57 miners died from the blast and its effects, the vast majority succumbing to carbon monoxide poisoning. Haldane recommended the simple and effective safety procedure of taking mice down the pit: with their higher metabolic rate they would show the effects of carbon monoxide poisoning long before it reached levels dangerous to man.

He also worked for the admiralty on the problems faced by their divers at high pressures. It had been known for some time that rapid decompression produced the liberation of nitrogen bubbles into the bloodstream, with crippling and often lethal effects. Haldane showed how such effects could be minimized by pointing out that however long a diver had been exposed to compressed air it was always safe to halve the pressure; that is, it is just as safe to ascend from six to three atmospheres as from two to one. Using his technique over £5,000,000 of gold was recovered from the wreck of the Lusitania between 1917 and 1924.

Haldane also investigated the response of the human body to high temperatures. Haldane's main work as a pure physiologist, however, was on the mechanism of respiration. In 1906 he published his most significant paper, in collaboration with John Priestley, which demonstrated the key role of carbon dioxide in the regulation of breathing. They showed that it was not a deficiency in oxygen that triggers the respiratory reflex but an excess of carbon dioxide in the arterial blood, acting on the respiratory center in the midbrain. Their work was published in full in Respiration (1935; 2nd edition).

In much of his work Haldane used for an experimental subject his precocious son J. B. S. Haldane, later to become one of the leading biologists of the 20th century.