Construction began on the U.S. Custom House in 1853, but was interrupted in 1859 due to costs and the possibility of South Carolina's secession from the Union. After the Civil War, construction was restarted in 1870 and completed in 1879. The building was placed on the National Register of Historic Places on October 9, 1974.

Land was purchased at Fitzsimons' Wharf at East Bay and Market. Construction started in 1853. Since the location was marshy, 7,000 piles that were 40 ft long were driven into the sand and were cut off at grade. A grillage or network of timber was laid. and a thick bed of concrete was constructed for the foundation. The granite walls of the basement were finished by 1855. After the marble-faced walls were erected, the columns were begun in 1858.

In 1859, there was increasing concern in Congress over the possible secession of South Carolina and the costs of construction. Representative John Letcher from Virginia called for a cessation of construction. Representative William Porcher Miles defended the construction with little enthusiasm. No funds were appropriated to continue construction in 1859. With the possibility of war, Congress only appropriated funds for protecting the construction from rain.

During the war, the building was damaged by shelling. In 1870, construction resumed. The original marble came from Hastings, New York. Because that quarry was abandoned, new marble was obtained from Tuckahoe, New Jersey. Fluted Corinthian columns surround the iron second floor gallery. The gallery is ornamented with fluted pilasters.

The windows are rectangular with pediments. The portico entrance doors are also pedimented. The building was topped with an entablature with architrave and an unadorned frieze with a dentiled cornice. The building has a low roof with an open balustrade.

Construction was completed in 1879. The total cost of construction was about $2,806,000.

Photo captured October 9, 2019.