Charles W. Phifer was born in Tennessee in January 1833. He attended the University of Mississippi and the University of North Carolina, from which he graduated in 1854. He and other members of his graduating class were in the same circles Margaret Eliza Cotten, of another prominent family in the State. He fought in the Civil War and moved to Texas after, where he was a superintendent of the Wichita tribe's "Indian school." He died in Savannah, Georgia on December 25, 1896.