Citrines answer is very good.
Your father is human. Humans are NOT perfect, not even your father or your mother. Don't try to hold them up to be or test them by a standard of PERFECTION.
You do not know all the facts and circumstances about their marriage and all of what was going on with your father ... and you never will.
You should talk to him and let him know it is very disturbing to you and then let it go after that. He is probably very regretful and very embarrassed. Don't let this mistake ruin things for you and your father.
Rob