Nelson was the son of Ernest and Hilma Nelson and graduated from Norway High School. He attended the Art Institute of Chicago before being drafted for the Korean War. After his discharge he completed a bachelor of arts and master of arts at the University of Chicago, and then completed a certificate in arts education at Harvard University.
In 1957 Nelson married the former Andrea Canavera. They both took jobs at Huntingdon College in Montgomery. Nelson was made an administrator of the college's fine arts department, and then came to Birmingham as the first director of the Alabama School of Fine Arts (ASFA), serving in that position for 25 years. He led the school's construction of its present downtown campus, which was later named in his honor. After retiring from ASFA, Nelson contributed regular columns as an arts critic for the News.
Nelson also gave lectures at the Birmingham Museum of Art and UAB, and taught Sunday school at First Methodist Church. He was board president for the Birmingham Opera and also served on the Alabama State Arts Council. He was also a member of the Birmingham Rotary Club.
Nelson died in August 2020 of COVID-19. He was survived by his three children; Elisa, Eric, and Kirsten; and three grandchildren.
- "James R. Nelson" obituary (August 3, 2020) The Birmingham News