Born on Park Place in Birmingham, Agee was always a collector, starting with seashells, then stamps, and finally maps.
He was a principal in the investment banking firm of Sterne, Agee & Leach, specializing in state and municipal finance. In addition to his work with civic and charitable organizations as a financial adviser, Agee was active in the development of the Alabama Historical Association. He led the effort to create highway signage for historic sites.
From the 1930s to the 1960s, Agee collected maps, focussing on early maps of the Southeastern United States and Alabama. Encouraged by his close friendship with library director Fant Thornley, Agee donated his collection of over 600 maps and atlases to the Birmingham Public Library. With Agee's guidance, the library has carefully conserved and expanded the collection, which is located in the Linn-Henley Research Library.
- "Rucker Agee: The Man Behind the Maps." (November 2002) Birmingham Historical Society Newsletter.