Weber was the son of a professor of divinity and brother of the noted scientists Ernst Heinrich Weber and Eduard Friedrich Weber, both of whom worked in anatomy and physiology. He was born in Wittenberg in Germany and studied physics at Halle, where his early research concerned acoustics. He obtained his PhD in 1826 for a thesis on reed organ pipes. He remained teaching at Halle until 1831 when he was made professor of physics at Göttingen on the recommendation of the mathematician Karl Friedrich Gauss.

Some of Weber's research was done in collaboration with his brothers. Thus in 1824 he published work on wave motion with Ernst, and in 1833 he and Eduard investigated the mechanism of walking. However, most of his academic life was spent working with Gauss. In 1833 they built the first practical telegraph between their laboratories to coordinate their experiments on geomagnetism. In 1837 Weber lost his post for opposing the new king of Hannover's interference with the State constitution. Nevertheless, he stayed in Göttingen for a further six years until he was appointed professor at Leipzig. Here, he improved the tangent galvanometer invented by Hermann von Helmholtz and built an electrodynamometer suitable for studying the force produced by one electric current on another.

His main work was the development of a system of units that expressed electrical concepts in terms of mass, length, and time. Gauss had previously done this for magnetism. Since force was expressed in these dimensions, he was then able to find his law of electric force. The principle was not very satisfactory because it did not conserve energy, but with it Weber publicized the view that matter was made up of charged particles held together by the force. This inspired the direction that physics took in the latter half of the century. The units of Gauss and Weber were adopted at an international conference in Paris in 1881. The unit of magnetic flux (the weber) is named in his honor.

In 1849 he returned to his post in Göttingen and collaborated with R. H. A. Kohlrausch in measuring the ratio between static and dynamic units of electric charge. This turned out to be the speed of light; this unexpected link between electricity and optics became central to James Clerk Maxwell's great development of electromagnetic field theory.