I've used all three. For reference, I'm about 5'2" and have hands that are just under 6" from wrist to tip of middle finger and about 2.5-3" wide in the palm. On a 50 mm, my fingers don't touch around the pole, on a 45mm they just barely touch thumb to middle finger at the closest I can bring them, and on a 40, there's overlap of the first knuckles of thumb and middle finger.
I find the 50mm to bring too big to comfortably do anything. The first pole I used was a 50mm chrome, then I switched to a 45mm at the studio (and what I have at home). (I did use a 50mm brass about a year later and found I was much better, probably due to improved/strengthened grip). I also have used 40mm chrome and either 40 or 38 mm brass (whatever is at Milan studio in Montreal). The smaller sizes felt unbelievably tiny, even though it was just a 5mm drop (feels much more obvious than a 50 to a 45, so you'll really notice a 50 go 40 mm drop).
However, I found the 40 mm the "easiest" to use, once you adjust to the size difference. It definitely requires less perceived strength/effort in hand gripped moves (e.g., extended butterfly, cupped grip shoulder mount), and although it does require you to grip tighten for leg grips, I didn't feel any less secure on it than on a 45mm, even in a CAR and bridged CAR. (I will say, some moves requiring a lot of tight/squeezed knee grips were more comfortable on it, like a Marley (holding both feet) or cupid/star (no hands or holding foot of bent leg)). But those are also moves I won't do on a 50mm, because I can just barely do them on a 45 without feeling like I'm crushing the back of my knee.
I'd say overall though, my favorite is the 45, since I find it easier to apply anything learned to both 50 (with some sketchy hand grip at times) and easily to the 40mm. If I were going get a 2nd pole though, it would probably be a 40 or 38mm. I rarely ever touch a 50mm now.
Is there anywhere you can go to test out one of the smaller poles first?