I visited a handful of years ago for a few days. I loved Portland....in the summer anyway.
Portland Head Light, which is just south, is a pretty iconic spot -- kids would enjoy that presumably. It's in a park area too, so there's a bit more to see. My own picture:
Freeport, 20 minutes north, is a walkable shopping area, if that's how the fam wants to spend an afternoon. It's the HQs of L.L. Bean, lot of outlet stores, quainter spots too. Inside Portland, we mostly walked around the Old Port, which is amazing. I had a really good meal at Street and Co. restaurant, in Old Port.