Why is my husband jealous of my past?

We had been friends before getting together so he knows a lot of my past and occurrences that have happened. Now years later, I am punished for them. His jealous side is over the top for me. At least once a week, we have to end up arguing over someone that I have not seen or talked to you in 3 years. I don't understand. I love my husband, and am totally faithful to him. I do not talk, see, or face book my exes. I am here for my husband 100%, but am punished for his thoughts. It has gotten to the point where I feel divorce is the only answer because I cant change my past. I have two kids, and if they talk about their dad I cringe, because I know that everything they say is going to be brought up later, because I will be questioned. It gets old and I am starting to resent him.I have tried talking to him, but he only turns it back on me asking" Do you want your exes in your life"? Nothing is ever resolved!! I don't want this to happen but I don't know what else I can do. Please help!!