Phillips was the son of Evan and Elizabeth Phillips, English by birth, who emigrated to the United States, residing in Louisville, Kentucky before settling in Ohio. John Phillips was educated in the public schools there and took a teaching position in Charleston, West Virginia in 1875. After saving enough for tuition he entered Marietta College in Marietta, Ohio, graduated with a bachelor of arts in 1880.
His application to serve as principal of the high school at Gallipolis, Ohio was accepted upon his graduation and after two years he was given a increase in salary. He resigned in 1883 to come to Birmingham to help establish a public school system here as superintendent. Through astute management the system became one of the largest and most well-regarded in the South and Phillips was given great esteem for the accomplishment. He married the former Nellie T. Cobbs on December 27, 1886
Phillips was elected president of the Chautauqua class of 1889, delivering addresses across the United States. In 1895 he was made president of the Southern Educational Association and served as president of the National Council of Education in 1902. During his career he completed post-graduate studies at the University of Chicago and the University of Edinburgh, Scotland.
Besides his work with the city's schools, Phillips founded the Birmingham Humane Society for the Prevention of Cruelty to Children and Animals in 1883 and opened the city's first public library in 1886, starting with a spare room adjoining his office.
- Phillips, John Herbert (1893) History and Literature in Grammar Grades. Monographs on Education. Boston, Massachusetts: D. C. Heath & Co.
- Phillips, John Herbert (1905) Old Tales and Modern Ideals: A Series of Talks to High School Students. New York, New York: Silver, Burdett and Company
- Dubose, John Witherspoon (1887) Jefferson County and Birmingham, Alabama: Historical and Biographical Birmingham: Teeple & Smith, Publishers; Caldwell Printing Works.
- Dubose, Joel Campbell (1904) Notable men of Alabama: Personal and Genealogical Vol. 1. Atlanta, Georgia: Southern Historical Association