Can anemia be harmful to my health in any way other than just annoying me by contributing to low energy levels?

The lack of iron associated with anemia can cause many complications, including hypoxemia, brittle or rigid fingernails, cold intolerance, impaired immune function, and possible behavioral disturbances in children.

Hypoxemia resulting from anemia can worsen the cardio-pulmonary status of patients with pre-existing chronic pulmonary disease. Brittle or rigid fingernails may be a result of abnormal thinness of nails due to insufficient iron supply. Cold intolerance occurs in one in five patients with iron deficiency anemia, and becomes visible through numbness and tingling. Impaired immune functioning leading to increased likelihood of sickness is another possible complication.

http://en.wikipedia.org/wiki/Anemia#Possible_complications