Born in Brooklyn, New York, Karle was educated at the City College there and at the University of Michigan, where he obtained his PhD in 1943. After working on the Manhattan Project in Chicago, Karle moved in 1946 to the Naval Research Laboratory, Washington D.C., becoming chief scientist in the lab for the structure of matter in 1968.

While in Washington Karle began an important collaboration with Herb Hauptman exploring new ways to determine the structure of crystals using x-ray diffraction techniques. Before their work, the structure of anything but the simplest molecule was usually worked out with the so-called ‘heavy-atom’ technique. This involved substituting a heavy atom, such as mercury, in a definite position in the structure. The changes produced in the intensities of the diffraction patterns allowed the phases to be inferred. The method, however, is limited and time consuming.

In 1953 Karle and Hauptman published a monograph, The Phases and Magnitudes of the Structure Factors, in which they demonstrated how phase structures could be inferred directly from diffraction patterns. For their work in this field, Hauptman and Karle shared the 1985 Nobel Prize for chemistry.

In 1942 Karle married Isabella Lugoski, also a crystallographer. She was one of the earliest workers to apply the new direct method to a number of important molecules.