Hop City has been steadily expanding their homebrew selection. It still could be better, but they've got a lot of malts, yeasts, and a moderate selection of hops. They also carry the cheapest liquid malt extract I've ever seen.
Just OTP on Roswell Rd. you'll find
Winecraft, who have a pretty good selection.
I've never been there, but I've heard that
Just Brew It is pretty good if you're south of Atlanta.
...and for the cheapest giant pots you can find, you go to the
Buford Highway Farmer's Market. You can get a 52 quart aluminum pot there for about 30 bucks.