Hey there!
This is also a fear I struggle with, and I was working on, when I had been training before my shoulder injury.
I am totally scared to let go with my hands in this position, because I often feel that my leg position is precarious, and I apparently feel most comfortable gripping with my thighs, rather than the lower legs. I also think that, because all of the grip is so distributed along the legs, I feel the pull less, and wonder if my grip actually exists at all!
So, my best recommendation is to work slowly, but also to push yourself, just a little bit!
When working from the handstand, getting the legs up the pole requires a bit of a rocking back motion with the upper body, in order for the hands to come closer and closer to the pole. Once you are in position (and, I should add, anytime I'm in the inverted crucifix position, my legs are not exactly straight!) you can work on pushing your comfort zone by lifting one hand. Switch hands when you can, and remember to breathe.
From a basic invert, I found that positioning the legs was fairly easy (again, not straight! I worried about that a while, but I think looking at other people, and feeling it yourself, can be one of those things where you think other's legs are straight, but they really aren't), but getting my upper body around the pole and looking down was quite terrifying!
So, again, best way to do it is to grip hard with the legs, and carefully experiment. Don't bring your torso around, just take one hand off the pole. And then, switch hands. Eventually, you can work on bringing that torso around and wait, get yourself used to it. Then, start working on the hands again.
It's a form of desensitization that will really help you to get through this fear in general, but it is NOT a quick solution! These sorts of things can come up again and again, with each new move, with each day. It can be more difficult some days, and less on other days. You might feel like you make progress, and then lose it.
Just keep trying! Keep facing down that fear, keep pushing that comfort zone, a little bit.
Don't push too hard, or you might end up turning turtle and running right back to that comfort zone and not wanting to come out for a while.
Be gentle with yourself. This is something that your body and mind feel is terrifying, and probably is deep into your psyche. It's sort of like people who are afraid of spiders; it's not that they had a really bad experience with them, it's just something that happens, it's buried in there, deep and instinctual.
So, work slow, take baby steps. Breathe.