First off you have nothing here at all except your imagination ..... have you been violated? I am so sorry to be so blunt, I know you are very very concerned and confused but I would explore this first.
Ask your son what are his favorite things to do with Daddy, then ask what are his favorite things to do with you, and then ask him if there is anything you or Daddy does that he does not like. Ask if if he wishes anything could change at home? ..while you are at it ask your older son also. Speak soft but matter of factly.
There have been times when I imaged my child was too quiet...I have slipped by their bedside and tried to hear well, but did not want them to awaken...I may have looked oddly positioned to my spouse too.
Question ...has your husband been sexually abused as a young child? That does not mean he would be abusive however it is something to take into account...my guess is that something quite violating happened to you ..I am sorry if I am wrong ...or if I am right : ] good luck to you all. God Bless.