The two-story brick building, the first of its kind in Tuscaloosa, was erected on the site of Lovel's Tavern by John Pass and William Toxey in 1831. Billy Dunton operated the "Golden Ball" inn and tavern on the upper floor, above Pass & Toxey's store. Over the ensuing years the original building was enlarged to the north and given the grander name of "Washington Hall". A life-size portrait of General George Washington holding the bridle of a gray horse was mounted to a brick pier outside and served as the inn's sign. A saloon and dining room shared ground floor space with the hotel's lobby. The second-floor auditorium was used for meetings, stage productions and balls. Postmaster William Gould operated the city's post office from the building, which also housed attorney's offices and a barber shop.
In 1835 Charles Patterson bought the property, and soon passed it to William Clare, and then, in 1839, to Matthew Duffee. Duffee prospered as a hotelkeeper and later purchased the Indian Queen Hotel. He sold his Tuscaloosa property in advance of the Civil War and moved to Blount Springs.
During the war, the hotel was used as a Confederate prison, then as a hospital, and finally as a Federal barracks before being burned during Wilson's Raid in 1865. The site was later used for the First National Bank building.