Even with a PhD in Physics, I was always afraid of studying number theory and cryptography. They seemed far too difficult. And, in fact, they are. But all math is hard - until you get used to it. It doesn't matter whether it's continuous or discrete, analysis or finite fields. Math is hard, but it's not that much harder than any other kind of math. You just need time and consistency. That is what I am learning.