Walker played baseball and football for the Birmingham-Southern Panthers and the Howard Bulldogs, graduating from Howard College in 1922. He played eleven seasons as an infielder in the Minor Leagues, batting .300 for his career with 30 home runs. While playing for the Wilson Bugs of the Virginia League in 1926, he was hired to coach football, basketball and baseball at Atlantic Christian College in Wilson, North Carolina. His "Little Christians" football team went 6-1-1, allowing only one score.
In 1927, Walker became head coach at Elon College near Burlington, North Carolina and remained for 10 seasons. His football teams went 45-40-4 with four North State Conference championships.
From 1937 to 1950 Walker coached at Wake Forest University, where he compiled a record of 77-51-6 and led the Demon Deacons to two bowl games: A win over South Carolina in the 1946 inaugural Gator Bowl and a 20-7 loss to Baylor in the 1949 Dixie Bowl at Legion Field. He is the school's winningest head coach.
After quiting Wake Forest, Walker sought a higher paying job and joined longtime friend and former assistant Herman Hickman at Yale. After one year at Yale, he replaced the retiring Lew Hayman as the second head coach of the Canadian Football League's Montreal Alouettes. There he had a 59-48-1 record in eight seasons and won four division titles before retiring after the 1959 season. After his retirement he became a scout for the New York Giants.
One of Walker's longtime friends was Wake Forest alumnus Arnold Palmer, whom Walker tried unsuccessfully to recruit to his football team.