Annie Haywood Ruffin was born in Raleigh, Wake County, North Carolina on March 28, 1829. The daughter of a prominent lawyer and later U.S. Senator, she and her siblings were members of Raleigh's elite social scene. She married Samuel Ruffin, a Confederate veteran from Alabama and died in Raleigh on January 7, 1917.