So, there are basically two reasons, both closely related to his sons: His own efficiency, as they get him distracted, for he goes after a dangerous daemon (the yellow-eyed one), and thus worries about them getting killed. He is afraid to actually see them die, like he saw their mother die.