There will always be factors and people that we cannot control; how we respond can determine the quality of our lives.
There are many stories of spiritual masters embracing the presence of an annoying student in their community. There is even one story that documents a teacher paying an irritating person to live among his students. From an everyday perspective, this is difficult to comprehend. We generally work hard to avoid people and things that we find annoying so they don't bother us.
From a deeper spiritual perspective, however, irritation can be an important teacher and indicator that we are making progress on our path. Being able to remain centered and awake even when we feel uncomfortable is much more impressive than doing so in an environment where everything is to our liking. No matter how good we are at controlling our circumstances, there will always be factors and people that we cannot control. How we respond to these experiences to a great degree determines the quality of our lives. The goal of spiritual development is not to learn to control our environment--which is more of an ego-driven desire. And while having some measure of control over our external reality is important, it is when we are confronted with a person or situation that irritates us and we can choose not to react that we know have made progress spiritually. It is when we have mastered our internal reality that we will have become the masters of our lives.
The more we try to eliminate annoyances, instead of learning to handle them gracefully, the further we get from developing the qualities that come with spiritual growth, such as patience, tolerance, and acceptance. It is often in the presence of people and experiences we find annoying that we have an opportunity to develop these qualities. Fortunately for most of us, our lives offer an abundance of opportunities to practice and cultivate these traits.