Bryan attended Lovejoy Academy in Raleigh, North Carolina and entered the University of North Carolina in 1880, majoring in law and theology alternately. After graduating in 1885 he returned to his family and taught at the Gastonia Female Institute. He left home again in 1886 to pursue further theology study at Princeton University.
Bryan came to Birmingham while he was still studying at Princeton to serve as part-time pastor of Third Presbyterian Church, then meeting in a tent at the corner of 23rd Street and Avenue G on the city's Southside. After graduating in 1889, he was invited back to lead the congregation, then meeting in a wooden structure on the same site. He was ordained and installed as the full-time minister on August 4. He was introduced to Leonora Clayton Howze of Marion in 1890 and they married the following January and eventually had seven children.
As a minister, Bryan conducted large evangelistic and prayer gatherings with various groups across the city and region. He was an outspoken supporter of prohibition, civil rights and racial reconciliation in Birmingham. He is best remembered, however, for his tireless efforts to help the poor and homeless. It is said he would often arrive home at night without his overcoat because he had given it away to a stranger during the day. Though he was an ordained Reverend and honorary Doctor, he earned the title "brother" by addressing anyone he met as brother or sister.
The "Brother Bryan Mission" was founded in 1940 to continue his work with the less fortunate. The City of Birmingham renamed Magnolia Park as "Brother Bryan Park" in 1989, honoring the centennial of Bryan's arrival in Birmingham. Bryan Memorial Presbyterian Church in suburban Birmingham is also named in his honor.
- Blakely, Hunter B. (1953) Religion in Shoes; Brother Bryan of Birmingham. Richmond, VA: John Knox.
- Bryan, Rev. James A. (n.d.) Sermons. Birmingham, AL: A. H. Cather.
- "Will Brother Bryan move again?" (March 4, 2004) Birmingham News.
- A Short History of Third Presbyterian Church 1884-2004; pamphlet arranged for Third's 120th Anniversary service