I have anxiety also, not to the panic degree, but I get stressed out at the ramp, etc. I also don't have an "easy" place to store it. As a result, I keep my boat in a "rack-and-launch" program where they just put it in the water for me with a forklift, it makes that whole process easier. I had a similar "buyer's remorse" experience after my first outing, which included a bulged trailer tire and two dead batteries (stuck on the beach). However, let me tell you that being out on the water away "from it all" is a big anxiety reliever. The experiences that the boat has given us (watching my 6yo daughter wakeboard for the first time, discovering new islands, navigating the lock system, etc, etc, etc) is priceless.
If your only "worry" is things that you can solve by practice and learning, then treat it as a personal challenge, and something that will build experience. That stuff is easily overcome with some practice and patience. My concerns include summer storage, winter storage, towing and getting ANYTHING for the boat up here in Canada (shipping from the US is a headache most of the time, not to mention our lousy dollar), not to mention that I am a perfectionist and need to ensure that everything is in superior condition at all times on my boat.
Give it some time and ENJOY the moments that you are out there. I bet you will be a pro at maneuvering and trailering in no time. At least you have a good tow rig and looks like a good place to keep it!