Is it possible for two people to be bad for each other?

I know someone who is a really good person, very caring and loving. There's no problem with him whatsoever, but when he and I interact, he's got these inherent traits that counteract with mine in a bad way, and it's the same with me. Is it possible that two people can be "okay" persons but when they're together, they clash, and what's the science behind that? I need to understand and learn, because we're hurting each other.