The answer to this totally depends on two factors: 1) what the definition of extreme is, and 2) whether we are talking about what most people do, or whether we are talking about what is possible at the highest levels of the sport.
As mikkixx said, if extreme means something that gives "you a massive adrenaline rush from height/speed/risk of death" then for most practicitioners the answer is no, but for the possibliities at the highest level of the sport, the answer is yes. For example, there are drops that require you to roll over away from the pole or let go entirely, like a backflip down the pole, and drop a considerable distance. If, alternatively, extreme means "extremely difficult" then the answer may be no for some levels, but is definitely yes for the more advanced levels of the sport. (Assuming we have defined "extremely difficult" by some objective metric, such as a combination of time, effort, and talent needed to learn something) So, overall, yes, I would say that pole is an extreme sport.