Kemeny was born in Budapest, Hungary. In 1938 his father was so alarmed by the Nazi annexation of Austria that he moved to the United States. The family followed in 1940 and Kemeny entered Princeton in 1943 to study mathematics. A year later he was drafted onto the Manhattan Project and sent to Los Alamos where he operated an IBM calculator. He returned to Princeton in 1946, completed his PhD in 1949, and moved to Dartmouth in 1953, serving as professor of mathematics (1956–68), as president of the college (1970–81), and once more, from 1981 until his retirement in 1990, as professor of mathematics.

Between 1963 and 1964 Kemeny, working with a Dartmouth colleague, Thomas Kurtz, developed BASIC (Beginner's All Purpose Symbolic Instruction Code), probably the best known of all computer languages. Previously the large computers could only be approached through specialized computer programmers. BASIC was conceived initially as something for Dartmouth students to use on Dartmouth computers. With a few simple self-evident commands and an equally simple syntax and vocabulary, it proved remarkably easy to use.

Because it was meant to be freely available to students, the software was placed in the public domain. Subsequently it became the most widely used computer language of the 1970s and 1980s.

Kemeny himself became something of a public figure. It was during his presidency that Dartmouth became coeducational and he did much to open up the college to minorities. He also campaigned against the Vietnam War. Kemeny was one of the main campaigners for the not altogether successful ‘new math’ introduced into America in the 1970s. In 1979 he was invited by President Carter to chair the committee set up to investigate the nuclear accident at Three Mile Island.