The Octagon House was put together by John Richards for his wife Eliza in 1854. Mr. Richards was one of the first and most successful settlers of Watertown. He was a lawyer and owned a few mills.

The house is considered as one of the major family estates in Wisconsin built during the pre-war era. It consists of 5 floors made of solid bricks and modern facilities such as running water, ventilating systems and central heating. The best feature of the house is the spiral staircase that was built at the center of the building and goes up from the ground to the upper tower.

It was the private residence of the Richards family only until 1937. In 1938, it was sold to the Watertown Historical Society who opened it to the public as a museum.