Hop City has been steadily expanding their homebrew selection. It still could be better, but they've got a lot of malts, yeasts, and a moderate selection of hops. They also carry the cheapest liquid malt extract I've ever seen.
Just OTP on Roswell Rd. you'll find Winecraft,
who have a pretty good selection.
I've never been there, but I've heard that Just Brew It
is pretty good if you're south of Atlanta.
...and for the cheapest giant pots you can find, you go to the Buford Highway Farmer's Market.
You can get a 52 quart aluminum pot there for about 30 bucks.