Rutherford, the son of an Edinburgh physician, studied under William Cullen and Joseph Black at Edinburgh University and became a doctor of medicine in 1777. In 1786 he was made professor of botany and keeper of the Royal Botanic Garden at Edinburgh.

Rutherford was the first to distinguish between carbon dioxide and nitrogen. A thesis he wrote in 1772, De aere fixo dicto aut mephitico (On Air said to be Fixed or Mephitic), contains some of Joseph Priestley's later discoveries. In his experiment mice were allowed to breathe in a closed container. The fixed air (carbon dioxide) was absorbed by caustic potash. The remaining air, Rutherford pointed out, was not fixed but would not support life or combustion and he called it ‘mephitic air’. He had in fact isolated nitrogen about the same time as Karl Scheele.