If you do it properly it will not damage them. You should only place them in the freezer if you suspect a tobacco beetle infestation.
Place them in a zip lok bag. Place them in the freezer for at least 72 hours (that gets the eggs also). Remove from freezer and let them thaw for 24 hours in room temperature. That should resolve the problem. Good luck.