Little is known about Nicol's early life except that he was born in Scotland. Primarily a geologist, he lectured in natural philosophy at the University of Edinburgh where James Clerk Maxwell was probably one of his pupils. His first publication came when he was nearly 60.

Nicol is best remembered for his invention, announced in 1828, of the Nicol prism. This device, constructed from a crystal of Iceland spar (a natural form of calcium carbonate), made use of the phenomenon of double refraction discovered by Erasmus Bartholin. The crystal was split along its shorter diagonal and the halves cemented together in their original position by a transparent layer of Canada balsam. The ordinary ray was totally reflected at the layer of Canada balsam while the extraordinary ray, striking the cement at a slightly different angle, was transmitted. Nicol prisms made it easy to produce polarized light. For a long time they became the standard instrument in the study of polarization and played a part in the formation of theories of molecular structure.

Nicol also developed new techniques of preparing thin slices of minerals and fossil wood for microscopic examination. These techniques allowed the samples to be viewed through the microscope by transmitted light rather than by reflected light, which only revealed surface features. His lack of publications resulted in some 40 years elapsing before these techniques were incorporated into studies in petrology.