Depends what kind of worms. There are many beneficial worms, peanut worms, bristle worms (when they don't grow to 5', like Steve Weasts), spaghetti worms to name a few. However there are some that are no good. For instance fireworms are a no-no. I don't know all of them, but it would help if you posted a pic or knew the ID of it so we can advise accordingly.
There is no blanket statement about all worms, like there is no blanket statement about all fish, coral, starfish or crab.