The nature of your actions against others can hurt her. If you harm someone here, who's to say no one will come after Kaori in revenge against you? They could easily consider her to be on your side and therefore an enemy, a danger to safety in the village as well.

I cannot risk that. I will not have people hunting her down because of you. And while I will not harm you without cause, if you harm Kaori by association or otherwise it will be different.