A lot of people, writers included, have difficulty with the proper usage of each of these words. I cringe when I see on TV or read in a book a character saying, “I don’t know what you’re trying to infer by that, but…”
Mignon Fogarty, better known as Grammar Girl, has the clearest explanation for how each of these words can be used correctly. As you can see from her article, and the graphic accompanying this post, it’s dependent on whether you’re the writer or reader, speaker or listener.