Does anyone else ever confuse their follower for some other player trying to rush in and take a collect quest item from them?

It happens to me over and over and over, (god damn you Belath!) I would have thought I'd be used to it by now but I'm not....