I’ll offer up an alternative viewpoint. Because Xavier has the ability to see minds, he grows up as the most empathetic person on the planet. He sees what everyone else is feeling, and can necessarily see everyone else’s point of view. When his parents are cross with him, he understands why. He could grow up as one of the kindest and most accommodating people, only using his powers when he deems it absolutely unavoidable.