Why is the contact lens in one eye stubbornly blurry?!

For the last month, I've been trying contact lens for the first time. A little bit of history. I've had an astigmatism in both eyes for years, but I've been stubborn about it. Finally two years ago, I decided to start wearing glasses. It was life changing. Since I wore eyeglasses all day, everyday, my astigmatism relaxed and it became hard to see without them. In a nutshell, I became dependent on my glasses. This made playing soccer hard (which I do every Sunday). So I decided to give contacts lens a try to be able to play in them.

Okay, so right away when I put in my contact lens, my vision is great! But as the day goes on, the right eye becomes blurry and I have to blink and sometimes move my eyeball all around to restore vision. The left has a bigger astigmatism, but it's not stubborn. It's taken to my lens right away. BTW, I've tried a few different brands. I finally found one that doesn't seem to move in my eyes. The other brands (toric lens) seemed to shift all the time in both eyes. But the one I have now, seems to hold strong. However, my right eye becomes blurry a lot. And at night... forget about it! I'm blind as a bat with my contact lens in. My optometrist says that eye is needs to get used to relaxing and not straining to see. And I should give it time so it relaxes and I get used to the contacts. Has anyone had this problem? It's so discouraging and depressing. I can't deal anymore.