This Greek Revival home was built in 1811 and purchased in 1830 by Dr. Elijah Evans Jones, a prominent physician in Madison. Dr. Jones was also the Chairman of the Board of Trustees of the Georgia Female College in Madison and a major shareholder in the Georgia Railroad.

Heritage Hall was a private residence until 1977. The last private owner was Mrs. Sue Reid Walton Manley, known as "Madison's First Lady" for her gracious financial support of churches, education, the city, and numerous weddings, receptions, and parties hosted at the home. Mrs. Manley's granddaughter donated Heritage Hall to the Morgan County Historical Society, Inc., in 1977.

The Ghost Room numerous people have walked in the room and described astrange feeling. Some visitors say they have actually seen Virginia’s ghost lying in the bed,” “However, the most important reason the name was developed is because there is an etching of a woman on the bottom of the fireplace. The figure is believed to be Virginia holding her baby. In the past there have been many efforts to sand the image away but each time it returns.