Born in Petersham, Massachusetts, Willard received only an elementary school education as a child. He worked on a farm and assisted in his father’s carpentry and joinery shop. Willard left for Boston in 1804, his first job involving setting piles for a wharf. Three years later he worked as a carpenter on the Exchange Coffee House in Boston (no longer extant). While working, Willard attended a drawing academy for two terms. In 1809 he became a carver, setting up a shop in Somerset Place and hiring assistants. For the next three years he worked with architect Charles Bulfinch on carvings for several churches being constructed in Boston. Willard began to make models of buildings, and was commissioned by Bulfinch in 1818 to make a model of the U.S. Capitol building. In 1824 Willard was authorized to draw a plan for the Bunker Hill Monument and was formally appointed the architect and superintendent the following year. He moved to West Quincy, Massachusetts, in 1825 to take charge of the Bunker Hill Quarry and the furnishing of stone for the monument. Because of the unusually large size of the stones, Willard proposed and helped establish a railway to transport them. He also designed the layout of Hall Cemetery located near the quarry, erecting within it a rejected column from the New York Merchants’ Exchange.
In addition to his commissions, Willard maintained a studio in Boston where he taught classes in stone carving and architecture. In 1843 he published Plans and Sections of the Obelisk on Bunker's Hill with the Details of Experiments Made in Quarrying the Granite. Later in life he retired to his farm in West Quincy where he sold tools and farming implements. Willard died in Quincy at the age of 77 and was buried in Hall Cemetery.