Rumore grew up in Birmingham and on a farm in Huffman. He graduated in 1941 from Ramsay High School and got his first radio job with WJLD-AM in Bessemer. After a year he moved to WSFA in Montgomery, then returned to Birmingham to work for WAPI-AM beginning on April 18, 1943.
Using the name Hill Billy Rose he hosted the "Round-Up" program on Saturday evenings. He also hosted the "Yawn Patrol" morning show. Other programs he produced included the "Golden River Boys", "Mid-Morning Melodies", "Round-Up Time", "Songs by Crosby", "Accent on Rhythm", "Wise and Otherwise", "Joe Emerson's Hymns", "Barn News", and "WAPI Dancing Party". One of the station's most popular announcers he regularly received thousands of letters each week from fans.
In 1948 the newly-married Rumore moved to WVOK-AM, the state's first 50,000-watt broadcaster and broadened his following nearly statewide, especially as his shows were rebroadcast from WBAM in Montgomery. He recorded his three daily Birmingham programs live with a studio audience and recorded the Montgomery show on tapes which were delivered by bus. When he was on vacation his brother, Duke Rumore filled in, until he got offered his own show on WSGN-AM.
On April 24, 1954 Rumore opened a music store, Rumore's Record Rack, at 1802 2nd Avenue North (underneath the parking deck). In 1964 he built a home studio in his basement, equipped with a teletype and an NOAA weather wire so that he could broadcast shows live from his home. It was there that 6-year old Sam Rumore, Duke's son, made his radio debut singing "Ave Maria" on a Christmas program. The broadcast equipment, teletype and other items from Joe Rumore's home studio are on display at the Alabama Historical Radio Society facility in downtown Birmingham.
Rumore worked with many of the same sponsors throughout his 41-year radio career. Some notable products he pitched were Golden Eagle Syrup, White Lily flour and SanAnn gasoline.