Brown, son of a coal miner, began playing baseball as a youngster in Alabama Byproducts Corporation's Bradford mining community near Pinson. He was added to the roster of the company's team when he was 15. He enlisted in the U. S. Army in 1950 and played baseball for his post in Munich, Germany. He was discharged in 1953 and was recruited to the Louisville Clippers by former James Bolin. He starred as a left-handed fastball pitcher and slugger in 1955, but returned to Alabama to recuperate after a late-season arm injury. He played briefly for the Birmingham Black Barons, then managed by Frank Evans, but soon retired from the league.
He returned to baseball as a member of Rick Woodward's Woodward Iron Company team which played in the Walter Horn Industrial League. He played for them for twenty years. He left Woodward to drive a truck for the Ashland Oil Company in the 1970s and was hired by the US Postal Service in the same capacity in 1980. He continued to play semi-professional baseball until 1996, often leading the league in hitting even into his 50s.