The estate was first established in 1654 when Thomas Pell signed a treaty with Siwanoy Indians for 9,000 acres of the Bronx and Lower Westchester. Generations of Bartows and Pells lived on the site until 1888, when it was sold to New York City for inclusion in the future Pelham Bay Park. The current mansion was built between 1836 and 1842 in the Greek Revival style, and its environs were transformed from pastureland, orchards, and lawns sloping down to the bay to a formal, terraced garden from 1915 to 1918 according to a landscape design plan by architects Delano & Aldrich. Landscape architect Ellen Shipman was hired between 1928 and 1931 to design plantings for the garden, but it is unknown if her plan was actually implemented. The mansion was reopened to the public as the Bartow Mansion Museum and Gardens in 1946, renamed Bartow-Pell in 1958.
More recently, the 9.9-acre grounds have undergone rehabilitation by MKM Partners based largely on historic photographs. The work reestablished the terrace’s lawn, reintroduced historic planting beds and species, selectively pruned trees to restore historic viewsheds, replaced tombstones in the family plot, and restored the chestnut grove. The Terrace Garden is comprised of a series of descending terraces divided into four quadrants and enclosed by masonry walls and wrought iron fencing with a square pool and cupid fountain at the center. Bluestone walkways and steps run along the axis created by the foundation from the mansion, and around the water garden. The estate was listed on the National Register of Historic Places in 1974, and designated a National Historic Landmark in 1977, which was expanded in 1978 to include the carriage house, walled gardens, and family memorial plot.