He is talking about your family. Obviously, the man can't see you, but
you still ask"What did they do?"
The man, although he didn't hear you, says, "They had
to....
A man walks into your house. He says, "It's so sad about what
happened to the Grants. The poor family. I wish that they hadn't done
what they did."