Established in 1855 on an undulating plateau that gently slopes to the north, this 206-acre cemetery was loosely modelled on Mount Auburn Cemetery in Cambridge, Massachusetts. It is bounded by Elm Hill Pike to the south, Lebanon Pike on the north, and Greenwood Cemetery to the east, merging seamlessly with Calvary Cemetery to the west, whose boundary is demarcated only by a pronounced transition in the style of the stone wall that encloses the site along the north. The original parcel, designed by Clement Woodson Nance, spanned 125 acres, with meandering paths that followed the natural topography of the site. The initial fifteen sections (now in the northwest of the current property) included a chapel and a vault set amid Victorian funerary architecture and interspersed with oaks and other plantings. In 1869 the Confederate Circle was added, a monument with thirteen concentric rows of graves crossed by four radial paths, all set around a central obelisk.
Unusually, the cemetery was first segregated by class rather than race, with Section 1 (in the northwest) reserved for only the most prominent citizens. From the 1890s to 1955, African American burials were barred from the cemetery, which expanded during this period to included Sections 16 to 35, fanning out to the southwest of the original parcel. These newer sections have fewer serpentine walkways but maintain the park-like layout of the original design. Located near the eastern edge of the site, eight rectilinear sections were added after 1955 and were given biblical names rather than numbers. The southern and northeastern edges of the cemetery are thickly wooded, buffering it from the adjoining railway and commercial development. Mount Olivet Cemetery was listed in the National Register of Historic Places in 2005.