Is asbestos hazardous if it's outside of your home?

The shingles on our house (or the outside of the walls of our house) have asbestos. They are not falling apart or anything but companies won't come in to remove it since it does contain asbestos. Is it hazardous to my family even if it isn't falling apart yet? Does anyone know how it works?