We discussed boundaries with the kids (no going through the gate or out the driveway, stay within the fence, stay out of the pond) and then let them play on their own. They were in and out of the house all week and, while with this many people around they were rarely alone, we also weren't ensuring they were accompanied.
I don't mean to come across as judgemental but I'm surprised that people let four-year-olds roam around outside without direct supervision. I wouldn't have been allowed to do that until I was about eight. (I do think that my parents were overprotective in many ways, but that wasn't one of the ways I thought they were overprotective.)
Also, I wonder when I learned to swim. It must have happened before my coherent episodic memories begin at six years old. Are four-year-olds physically capable of swimming?