If possible make the hat tighter on your head, or try and clip it to your hair with hair clips of some sort.