It can be a little annoying yes. This is because in your own head, you have got your psychology straight. You know precisely what your strategy is and as an experienced trader you know that it is necessary to accept drawdowns in order to achieve long term gain. An investor however, can't see into your head! Many investors find it difficult to accept a loss, they panic when you enter a drawdown and want out.
The advice is exactly the same as Muiris said - you are in no real position to sort out the pscychology that is going on inside people's heads. That's something that those individals need to sort out themselves - therefore you need to simply keep on doing what you are doing, and keep doing it well. Try not to let it annoy you - If you do let it annoy you, you will find that it starts to play with your own psychology and you might start to trade differently in an attempt to keep those investors on board - THIS WOULD BE A GREAT MISTAKE. Do not let investor behaviour affect how you trade. Continue to trade as you are, making decisions based on your own criteria.
If some investors fall away that's cool. Don't worry. The investors that understand the associated risks and accept them will stay with you. Carry on doing what you do and you will make a nice return irrespective.