Well, I am not a teacher yet but I can tell you what my own teachers have done and told us how to handle it when we're met with this issue. Since you're teaching at the college/university level, I would honestly first ask politely what's the problem, if they want to share their thoughts with class and so on. If they would continue to disrupt the lessons I would simply ask them to leave. That's what our University teachers always did. They told the student to leave and to not return to the next classes unless they'll be quiet. I can tell you that it often worked
During our pedagogy classes, we were told that these students/pupils only want attention. The most common method of dealing with talkative students is to use the ''raise your hand'' method - when you want to talk, you raise your hand and the teacher gives you the word. Another good way is to give them some special assignment that they must complete (such as answers) by the end of the class. So, you force them to listen to what you're saying or they can't give correct answers. If they fail there's some sort of punishment - perhaps minus points in the future exam?
Overall, you are right and the teacher does have to deal with all sorts of students and pupils but always remember that there are at least few of them that enjoy your lessons and that you inspire.