I tryed a few times to write an answer, but it didn't go through.
I was writing about Incident management and writing an e-mail from one Incident item. That what you describe is true for users ...that have rights to see the item, and that is not a problem.
But, when I want to write an e-mail to someone who doesn't have rights to see one particular incident and this someone have occasional or full licence in the Incident management the system "offers" this mail address as external addresses. And in fact, this mail address is not external.More