Carmichael was the son of Dugald and Caroline Coleman Carmichael of Dallas County. He attended the local schools and went on the Alabama Agricultural & Mechanical College at Auburn. He began teaching in 1883. In 1884 he began to read law and took on editorial duties for the weekly Alabama Free Press. In 1885 he advanced in his studies in the office of William Samford, gaining admittance to the Alabama State Bar in 1886.
Carmichael began his law practice in Greenville, moving to Gadsden for part of 1888 before relocating that fall to Birmingham as partner to Robert Thach. He left the partnership when he was elected chancellor in 1898. That same year he married the former Jessie Turner.
Active in Democratic politics, Carmichael served as a frequent state convention delegate and was an elector at large in the 1892 National Democratic Convention.
- Dubose, Joel Campbell (1904) Notable men of Alabama: Personal and Genealogical Vol. 1. Atlanta, Georgia: Southern Historical Association