Be sure it's clear to the person to whom you are talking that the conversation is finished. Then replace the receiver gently. A receiver banged down may seem like slamming the door in someone's face.