How long should swelling take to go down on a nose injury?!

I had a really nasty fall on Saturday night, so tomorrow night will be the week anniversary of my nose injury! I went to the hospital and had a series of x-rays and the doctor was convinced there wasn't a break. I had a very slight black eye on the right side of my nose and there was quite a bit of swelling under my eye and on my nose but it didn't seem to be anything TOO serious.

Now all that's left of the tiny black eye is a shade of yellow around my eye socket, but my nose is still a tiny bit numb and there's definitely a bit of change in shape to the right side of my nose. Nothing hurts to touch or press gently but the change in shape is worrying me. Given that it's still a little bit numb is it likely that there's just a bit of swelling going on under the bone or something?

Is it possible that it's broken even though they said it wasn't? I'm paranoid because I broke my leg before and they sent me home saying it wasn't broken before I demanded a second look and it turns out it was broken. When should I return to the doctor if I notice the change in shape hasn't returned to normal? I realise with these things the sooner you get it sorted the better.