The Society of Four Arts, dedicated to art, drama, literature and music, was founded in 1936. Two years later the Society opened its first building, a library, designed by architect Maurice Fatio in concert with a series of demonstration gardens designed by the Palm Beach Garden Club. The original seven gardens, each designed by a club members, included a Tropical Fruit Garden ( Mrs. Alfred G. Kay), Chinese Garden (Mrs. Lorenzo E. Woodhouse), Fragrant Moonlight Garden (Mrs. Joseph F. Gunster), Spanish Patio Garden (Mrs. John S. Phipps), British Colonial Garden (Mrs. Clifford V. Brokow), Florida Jungle Garden (Dr. Edmund LeRoy Dow), and a Formal Rose Garden (Mr. Hugh Dillman).

Beginning in 1939, the gardens have seen additions and “restorations,” often of necessity after storm damage. Following a 1947 hurricane, Mrs. Lorenzo Woodhouse replanted the Chinese Garden at her own expense. In 1956, the Garden Club engaged the firm of Innocenti and Webel to do a master plan. In a 1959 New York Times interview, Richard Webel noted that the work resulted in “a more spacious open fountain area as a foil for the relatively smaller scale of the other garden units.” Contiguous with the garden is the Philip Hulitar Sculpture Garden, named in memory of its founder and chief benefactor.