Wolff was born in Nice, France, to the German literary publishers Helen and Kurt Wolff, who had published works by Franz Kafka, Robert Musil, and Walter Benjamin. After relocating to the U.S. in 1941, they helped to found Pantheon Books with other European intellectuals who had fled Europe during the rise of fascism. The Wolffs published a series of notable English translations of European literature, mostly, as well as an edition of the I Ching that came to greatly impress John Cage after Wolff had given him a copy.
Wolff became an American citizen in 1946. When he was sixteen (in 1950) his piano teacher Grete Sultan sent him for lessons in composition to the new music composer John Cage. Wolff soon became a close associate of Cage and his artistic circle which was part of the New York School and included the fellow composers Earle Brown and Morton Feldman, the pianist David Tudor, and the dancer and choreographer Merce Cunningham. Cage relates several anecdotes about Wolff in his one-minute Indeterminacy pieces.
Almost completely self-taught as composer, Wolff studied music under Sultan and Cage. Later Wolff studied classics at Harvard University (BA, PhD) and became an expert on Euripides. Wolff taught Classics at Harvard until 1970; thereafter he taught classics, comparative literature, and music at Dartmouth College. After nine years, he became Strauss Professor of Music there. He retired from teaching at Dartmouth in 1999. In 2004, he received an honorary degree from the California Institute of the Arts. He was also awarded the Foundation for Contemporary Arts John Cage Award (1996).