I had the same thoughts, for what I wanted a $500+ camera just didn't make sense. I ended up with a Nabi Square HD camera that has been perfect for what we wanted, and at $50 you aren't out much. I have used it skiing, wake surfing, tubing, boating, swimming, etc. It looks like it has been discontinued since I bought it a few years ago. Looks like there are a lot of sub $100 cameras on Amazon now that I would pick up to replace the one I have if it broke, etc. Video quality is not the greatest, but it has been good enough to throw up on the TVs at home and have decent pictures. It's probably not as good as a 4K go-pro video, but whatever...
The things that have been the most valuable to us:
1) Recording time, make sure there isn't a limit to how long it would record for. I remember seeing that stopped after 15 mins and you had to re-start the recording.
2) The waterproof housing has been awesome to have. Just hand it off to the kids and they can swim and take pics/videos, pretty fun to watch those afterwards.
3) The remote gets used all the time. I usually mount it to the tower for surfing/tubing/boating, so having a remote in my pocket makes using it super easy.
4) Mounting options, make sure you have a flexible mounting adapters for all the places you thing you want to place it. Sticker mounts, rail mounts, tripods, suction cups, etc.