I feel in a similar boat to the OP. I decided to go thru with seeing some SPs. I don't completely regret it, as I feel I chose MOSTLY wisely (saving $$ for a better experience more often than not, avoiding BP/CL altogether, seeing someone when I traveled, etc). But I do sometimes think: Damn, if I hadn't seen those ladies, I could have a new SUP board or new bike!
Why did I make the choice I made when the potential repercussions were so severe and life is, actually, quite good right now? I don't completely know myself, but part of it I think was the thrill in keeping it secretive, avoiding detection. We all take some risky decisions, whether going down a steep hill on bike/skis, taking a slightly larger wave, etc. The potential for life-altering injury exists all around. So I felt this was an extension of that - I'm never going to be the guy going in the backwoods or BASE jump, but seeing an SP provides its own thrill and sharpens the mind.
Other posters have provided pretty sound advice, I think - if I were to do it over, I would probably skip some of the SPs I saw and buy that toy I've wanted

And I would talk through more about how I was feeling on the midlife crisis front with my wife. I think the better idea is to talk through that feeling that something is missing, and see where that conversation goes.