Robert Baxter McRary was born on November 21, 1860 in Lexington, North Carolina. A Lincoln University graduate, McRary was an educator, lawyer, and real estate agent in Lexington. During World War I, he served as chairman of the segregated War Saving Stamp Committee. In 1920, McRary was unlawfully run out of Lexington under threat of death after white citizens accused him of having an affair with Florence C. Varner, the wife of Henry B. Varner. Varner was a prominent and politically powerful white citizen of Lexington, serving as the Chairman of the Board of Directors for the state prison system during the administration of Gov. Thomas W. Bickett at the time of the allegations. Varner sought to destroy his wife and McRary through his newspaper, the Lexington Dispatch, and through the court system, bringing suits against both parties. For their part, Mrs. Varner and McRary both flatly denied the allegations, stating that McRary had been in the habit of visiting the Varner home but only with the motive of seeing the Varner's domestic employee, Anna Miller. Despite the testimonies of Mrs. Varner, McRary, and Miller to the effect that there was no affair, juries sided with Mr. Varner. McRary never returned to Lexington, living out the remainder of his life in Baltimore, Maryland. He died there on April 23, 1946.