Fleming, who was born at Lancaster, studied for a short time at University College, London, but left before graduating. However, he continued his work for a science degree in his leisure hours while employed first in a shipwright's drawing office and later as a stockbroker's clerk. Between 1871 and 1880 he had alternate periods of school science teaching and further study, including working under James Clerk Maxwell from 1877 at the new Cavendish Laboratory in Cambridge. In 1881 he was appointed professor of mathematics and physics at University College, Nottingham. From 1882 to 1885 he worked as consultant to the Edison Electric Light Company in London. He was then appointed professor of electrical technology at University College, London, a post he held for 41 years.

At University College Fleming gave special courses and experimented on wireless telegraphy, cooperating a great deal with Guglielmo Marconi. One of Fleming's outstandingly important inventions was the thermionic vacuum tube, a rectifying device based on an effect discovered by Thomas Edison. Fleming's diode consisted of a glass bulb containing two electrodes. One, a metal filament, was heated to incandescence by an electric current, so that it emitted electrons by thermionic emission. The second electrode (the anode) could collect electrons if held at a positive potential with respect to the filament (the cathode) and a current would flow. Current could not flow in the opposite direction – hence the name ‘valve’ for such devices. Lee de Forest developed the device into the triode for amplifying current.

Other scientific contributions by Fleming included investigations into the property of materials, transformer design, electrical measurements, and photometry. He was an outstanding teacher and highly successful as a popular lecturer. Fleming's left hand rule and right hand rule are mnemonics for relating the direction of motion, magnetic field, and electric current in electric motors and generators respectively.