Johannsen was born in the Danish capital Copenhagen. On leaving school in 1872 he became apprenticed to a pharmacist as his father could not afford university fees. From his work in Danish and German pharmacies, Johannsen taught himself chemistry and developed an interest in botany. In 1881 he began work under Johan Kjeldahl in the chemistry department of the Carlsberg laboratories, investigating dormancy in seeds, tubers, and buds.

In 1892 Johannsen became lecturer at the Copenhagen Agricultural College. On reading Francis Galton's Theory of Heredity he was impressed by experiments demonstrating that selection is ineffective if applied to the progeny of self-fertilizing plants. Johannsen repeated this work using the Princess bean, but found that selection did work on the offspring of a mixed population of self-fertilizing beans. It was only when plants were derived from a single parent that selection had no effect. He called the descendants of a single parent a ‘pure line’ and argued that individuals in a pure line are genetically identical: any variation among them is due to environmental effects, which are not heritable. In 1905 he coined the terms genotype to describe the genetic constitution of an individual, and phenotype, to describe the visible result of the interaction between genotype and environment.

Johannsen explained his ideas in On Heredity and Variation (1896), which he revised and lengthened with the rediscovery of Gregor Mendel's laws and reissued as Elements of Heredity in 1905. The enlarged German edition of this work became available in 1909 and proved the most influential book on genetics in Europe. In the same year Johannsen proposed the term genes to describe Mendel's factors of inheritance. Johannsen's research, with its emphasis on the quantitative variation of characters in populations and the application of statistical methods, played a major role in the development of modern genetics from 19th-century ideas.

In 1905 Johannsen became professor of plant physiology at Copenhagen University and was made rector of the University in 1917. He spent his later years writing on the history of science.