Langley, the son of a schoolmaster, was born in Newbury and educated at Exeter Grammar School and Cambridge University. Although he originally intended to join the Indian Civil Service, Langley fell under the influence of Michael Foster, professor of physiology at Cambridge, and turned to physiology instead. He was appointed lecturer in histology in 1884 and in 1903 he succeeded Foster as professor of physiology, a post he retained until his death.

Langley began his research career in 1875, when Foster asked him to observe the physiological effects of jaborandi juice, an alkaloid derived from an American shrub. Foster was particularly interested in its effects on the heart but Langley was more impressed by its power to evoke copious secretion, a process he spent the next 15 years studying.

His most significant work, however, was done on the autonomic nervous system, a term he himself coined in 1898. In collaboration with William Dickinson, Langley began (in 1889) to explore the sympathetic nervous System in some detail. This was made possible by the discovery that nicotine would selectively block nerve impulses at the sympathetic ganglia, enabling Langley to distinguish those nerves that actually ended at the ganglia from those that merely passed through them. Langley was thus able to show that only one ganglion lay along the sympathetic nerve on its path from the spinal cord to its goal.

In 1901 Langley followed this by exploring the problem of how nerves communicate with the muscle cells with which they form junctions. Basically the sympathetic system increases heartbeat, raises blood pressure, and contracts smooth muscle, i.e., it prepares an animal for action. Langley noted that when an extract from the adrenal gland was injected into an animal, these responses were produced. It was left to the British physician Thomas Elliott, a pupil of Langley, to propose that it was the adrenal extract, later termed adrenaline (epinephrine), that was released from sympathetic nerves to stimulate muscle.

Langley later published the details of his work in his Autonomic Nervous System (1921).