Born in Visegrád, Hungary, as a child Zoller was taught classical violin by his father, who was a professional violinist. In his teens, he switched to flugelhorn, then bass, and finally guitar. Zoller quit school during the Russian occupation of Hungary following World War II and began playing professionally in Budapest jazz clubs. He escaped Hungary in 1948 just before the permanent Soviet blockade of the country and began his serious music career after he moved to Vienna in 1948. He formed a jazz group with the accordionist and vibraphonist Vera Auer. Zoller left Austria for Germany in 1954, where he played with pianist Jutta Hipp, saxophonist Hans Koller and trombonist Albert Mangelsdorff. Visiting American musicians Oscar Pettiford and Lee Konitz found Zoller's work notable and they urged him to move to the US which he did in 1959, after winning a scholarship to the Lenox School of Jazz. There he studied with Jim Hall and roomed with Ornette Coleman and Don Cherry, whose influence sparked Zoller's interest in free jazz.

Zoller played in drummer Chico Hamilton's group in 1960, with Benny Goodman and flautist Herbie Mann from 1962-1965. In 1965, he began leading a free jazz-influenced group with the pianist Don Friedman, and in 1968 co-led a group with Konitz and Mangelsdorff.

Zoller played and recorded with, among others, Tony Scott, Jimi Hendrix, Stan Getz, Fred Nelson, Red Norvo, Jimmy Raney, Herbie Hancock, Ron Carter, Shirley Scott and Cal Tjader. In addition, his concert and touring activities took him regularly to the European festival circuit, to Japan, and to various US jazz clubs.

Zoller was the founding president of the Vermont Jazz Center (1985) where he also taught music until 1998. In 1995, he received the Lifetime Achievement Award from the New England Foundation for the Arts for his lifelong musical contribution to jazz.^[3] He was also a designer of musical instruments; he patented a bi-directional pickup for guitars in 1971 and helped design his own signature line of guitars with different companies. He died in Townshend, Vermont. A tribute album of his music was released in Summer 2015 with ENJA Records. The recording includes artists such as Ron Carter, John Abercrombie, Mike Stern, Peter Bernstein, Pat Metheny' Jim Hall, Gene Bertoncini. The album, Message to Attila was produced and arranged by Grammy nominated Guitarist, Producer David Becker.