Gilliam's family moved to Birmingham when he was young and he attended W. C. Davis Elementary School and Ullman High School. While in school, he began playing sandlot baseball with the Birmingham Browns of the Birmingham Industrial League, idolizing Willie Mays and Roy Campanella.
Gilliam visited New York City in 1953 for his sister's wedding. While there, the 6-foot-7 Gilliam was drawn into a practice with the Central Park Pearls. He signed with the team on the spot and was scouted from there by the Brooklyn Dodgers, signing his contract in 1955. He played for a while in their system, but was released due to a sore arm and returned to Birmingham.
Gilliam resumed his baseball career in 1957 with the Birmingham Black Barons, playing as both a pitcher and as a pinch hitter. Halfway through the season he transferred to the Raleigh Tigers in Raleigh, North Carolina. He nearly completed a shut-out of his former team when the Black Barons visited Raleigh late in the season.
In 1958 Gilliam started the season with the New Orleans Bears, but soon returned to Birmingham, wearing number 17 for three more seasons with the Black Barons. In 1961 he returned to the Raleigh Tigers for his last season in baseball. He was named the starter, opposite Satchel Paige, for the East All-Star Team in that year's East-West All-Star Game at Yankee Stadium.
- Powell, Larry (2009) Black Barons of Birmingham: The South's Greatest Negro League Team and Its Players Jefferson, North Carolina: McFarland ISBN 0786454806, p. 81