Thanks very much for the response! For some reason I don't think I saw your response until today :-(
Went ahead and upgraded the head unit to get bluetooth USB and AUX in. Is probably missing a few things, but got a Kenwood KDC-BT558U at Fry's for $139. I know there are better ones out there, but dang it was nice to just remove the OEM and plug this one in - no wiring whatsoever and a nice upgrade from CD AM/FM. Duh. Did not replace speakers, so still running OEM Kenwoods.
I got cold feet on the rest. I was headed towards upgrading the coaxes with Polk
http://marine-speakers-review.toptenreviews.com/, Fosgate or Kicker 6.5s. Note all have a solid cone since the tweeter is suspended on the grill. I was hoping to drive these with the Fosgate Punch 600X4 that would give six coaxes 100W RMS each (back two in parallel so amp sees 2 ohms). I thought this would be better than going straight to a sub, in fact maybe enough bass to not have to do a sub at all.
After this weekend though, I believe you're right that the sub should come first. The lower-end system was an all JL deck boat with six 6.5's and one down-firing 12 under the helm (really small box, btw) all fed by a 5-channel JL amp (super super tiny). Think he was in for about $3K. The second boat's system was higher-end, but I left feeling like that's just way too much sound for us. I didn't mind the bass, but two boats down on the beach it was still hard to carry on a conversation, ostly because of mids/highs. I think I could run the components (with lows filtered) off the head unit with a modest sub/amp and be satisfied.
Also...spent some time revisiting Pete's thoughts on battery isolators. Recall his research on the Hellroaring isolator. There's a new one from Sterling
http://sterling-power-usa.com/ProSplit-RZeroVoltDropMarineBatteryIsolator-13.aspx that actually reads cooler to me. Either way, about $150 adder that many will say is overkill. Then I talk to a guy that had his system professionally done (with some brand of isolator) and he's replacing Optimas like they're going out of style (two-ish per year for ten years.) I've heard Optimas aren't as good as they used to be, but I've got 11 years on mine. Figure he must be doing something wrong, and if a high-end isolator will keep the batteries safe, I'd definitely prefer to do that.