Turning down invitations

You know what sucks?  Turning down an invitation.  I love people.  I love parties.  My husband's best friend is married to a lovely girl.  He is always asking us to join them for dinner or whatever.  It's obvious he wants for this best friend relationship to expand to the married couple friends status.  He tries.  My husband is not as social as I am.  He is perfectly happy having a best friend that he hangs out with independently and never having me socialize with his wife...or rarely, but he understands his friend's desire.  The most recent attempt however was an impossible one.  Mike invited David to play golf on Saturday morning.  And following golf, Gloria's brother (Mike's wife) was having a party.  An annual thing.  They have a pond on their property.  They play in the water, swimming tubing,  bouncy things.  They have a cookout.  They have a few beers and a few laughs.  It sounds like a great time.  Granted we don't really know Gloria's brother or Gloria's brother's family, but Mike invited us and told us to bring the boys.  Here's the thing:  we can't bring the boys.  Charlie would have wonderful time.  Jimmie would need to be either held in our lap the entire time, or restrained in some other manner, to prevent his walking full on into the pond and drowning.  We would never be able to relax and enjoy the party.  And getting a babysitter on such short notice wouldn't have been impossible, but would have been expensive.   Now here's the thing: we could absolutely take Jimmie out to a restaurant or to park (provided it is enclosed), but there is no way we can take him to a stranger's unenclosed yard with a pond.  It would be miserable.  So we come off like people who never want to do anything.  We do.  We would love to spend more time with this couple.   But this was just an impossible situation.  So what do you do?