Oh gosh I thought it was just me who got freaked out by handstands! I also find a lot of moves are hit and miss for me too.
I have the Jamilla pole DVDs. On one of those, she teaches getting into a handstand from the floor, by putting your feet and legs on the pole, then "walking" your hand towards the pole, whiile gripping the pole with your legs. Start from standing with your back to the pole, then put your hands down on the floor in front of you, with your feet/legs on the opposite side of the pole, then hook one foot on, then the other one.
I also did a handstand/headstand workshop with Shimmy. She teaches you to practice handstands by going into a lunge, reaching as far forward as you can with your hand, place your hands on the ground, one leg straight, foot on the floor, and the other leg in the air, then do a kind of "hop" up with your foot that's on the floor, until you can get used to the feeling, before you progress to doing handstands against the wall.
When I learn a new move, I like to have an "exit strategy" in place, just in case, so that I can get out of the move safely.
Strangely, I seem to have no fear when doing "backwards" moves (spins, and CKR, CAR for instance - maybe it's something to do with being left handed!)