i had a hematoma skull fracture 4 years ago there was no opeartion i had to lay !

i want to box just as in sparring not in the ring the fracture has fully healed but i am not sure if that makes it ok to do boxing the fracture was at the back of the head the only long term symptom i have is rarely a bit of unsteadiness i cant find a answer any where for this question i have seen a doctor and he said boxing is not good because u get punched and the brain shakes to much i was like wow i didnt no that lol but he didnt answer my actual question.any help please???