What type of seafood are you interested in? I mean, there is the exotic stuff - such as lobster or stone crab, etc. but there are always basics too, depending on the locality.
The local stuff - I am thinking steamed crabs, spot, rockfish, oysters, etc - that come out of the Bay are great and plentiful around here.
So if you want good basic seafood like when you go to a beach area nearby you need to think about the restaurants that have a tie to those areas.
Arnest is one of them, in King William off Mechanicsville Tpke. Also most local places - e.g. Shackelfords in the west end, Awful Arthurs and others have great basic seafood.
Hardshell is phenomenal, but sometimes you just want a lb of spiced steamed shrimp in old bay.