There are two things I would say about this...
First, is that we can injure ourselves perfectly well without pole dancing. I've heard of people breaking their ankles simply by getting out of bed, or paralyzing themselves by going down the same set of stairs they'd been traversing for years. Humans are perfectly capable of hurting themselves- we're not exactly a perfectly-balanced bunch. You ever tried to breathe a muffin? Same sort of thing! We just have occasional problems. Our bodies are not machines, as much as we would like to think of them that way; machines are perfect, they do the same thing over and over without fail, and always the same. We're not like that.
Secondly, having dealt with an anxiety disorder for most of my life, I know a thing or two about fear. I HAVE actually fallen out of a Cross Ankle Release, and I was completely terrified of snapping my neck when attempting a somersault.
The thing I've learned most recently is this: The more you avoid the thing you are afraid of, the more afraid you will be of it.
I tried a somersault a few years ago. On my bed, with my friend giving me pointers, in the safest and least embarrassing situation I could possibly have created. I did it! A few times! And, I am no longer afraid of them. They aren't exactly part of my "regular movement," but I no longer feel fear.
I have been working on the Cross Ankle Release for some time- again, in the safest ways I can create. I do it lifting up from the ground, and simply holding the grip. I'm teaching myself to get used to the hold, and I'm teaching myself about all the nuances of the hold. I'll have rock-solid abs by the time I'm up to it, too ;)
It can be very difficult to face down fears, but it is best to do so right at the beginning, when you start to feel the fear in the first place. Tackling a nervous tension is far, far easier than taking down a full-blown phobia. I'm currently doing a very slow, controlled and quite terrifying practice of exposure training to get rid of my phobia of driving. It built up after years of avoiding driving at all- and, some bad experiences with my Ex.
Push through, and remember that you may be fallible... but you can do your best to be present, to practice safely, and to be respectful of the danger.