That is about right, 20 miles from 95. Yes ! I think that Swan Lake and the Iris gardens are worth the trip. Bring some bread and feed the beautiful swans and ducks ! Enjoy the Iris' and have a pleasant walk around the lake on a very nicely laid out walking path ! I think that a picnic in the park would be a wonderful suggestion. The park has many tables set out for such an event, some are even covered.
As for dinner (or eating at some other time), since you are heading up 95, you might want to consider RedBone Alley in Florence SC. An excellent locally owned and operated resturant. I am not sure of the exact address, they are on the web. They have a location in Sumter too, but I would recommend the Florence location if you can get there. It is the original location and I understand that the head chef (and resturant founder) works out of that location.
Good luck, enjoy your trip and enjoy Sumter !!!!