This idea really stems from Freudian psychology, with the Oedipus complex and the Electra complex where the child feels strong feelings for the parent of the opposite sex. This apparently occurs when the child is age 5 and below, though a lot of Freud's theories have been disproved now. It doesn't depend on the sex of the adult; children prefer the adults that they form a bond with at this age, which is shown by the attachment theory.