Born in Pittsburgh, Pennsylvania, and raised in Texas, Fite enrolled at the Houston Law School at the age of twenty. Terminating his law studies, he began seminary studies at St. Stephens College in Annandale-on-Hudson, New York where he discovered a love of acting. Three years later he abandoned school to join an acting troupe.
Backstage one day, he picked up a discarded spool and began to whittle. Soon after he became a full time sculptor, and in 1933, his former alma mater, St. Stephens now known as Bard College, hired him to help develop their fledgling fine arts program. Fite settled across the river from the college in 1938, purchasing a twelve-acre property, containing an abandoned bluestone quarry in Saugerties, New York. That same year he was invited by the Carnegie Institute to help restore Mayan sculpture in Honduras. Inspired by the Mayan construction techniques he saw on his trip, in the spring of 1939 he began work building an outdoor gallery on his property to display his figurative sculptures. Working alone, and utilizing quarryman’s tools, he constructed the gallery from bluestone rubble using an adaptation of dry key stone masonry. In 1962 he realized that the gallery, a monumental group of pedestals, ramps, steps and terraces, was a work in its own right. Removing his sculptures from the installation, he continued to construct the piece, which he named Opus 40, until his accidental death in 1976 from a fall in the quarry.