There are two, the old one down by the ocean, and the big new one on the ridge between Encino and the Westside. The old one used to have antiquities, not sure now that it has reopened (it was closed for a number of years). The big one has revolving exhibits, and is a must see. The gardens there are spectacular.
I went last year and besides the views, there is an extreme amount of walking with lots of stairs, just to look at pictures, wall art, and statues that all begin to look the same after seeing so many. I myself prefer the Huntington Library or Hearst Castle for some good antiquities, great paintings and rare books.
Yeah, I have been up there a few times and I feel that the outside areas are more interesting than the inside stuff. I could (and have) spent all day hanging around outside taking in the great views and sketching the architecture.
It is very cool. I've lived here for ten years and just went for the first time this summer. It's five dollars to park, but admission is free. Definitely worth spending an afternoon there.
I live out of the area, but have been several times. I highly recommend it. As others have said, give yourself time just to walk around the grounds to see the views, the architecture and the landscaping. Inside, I think their photography exhibits are especially good! The new neoclassic sculpture gallery looks awesome.