What happened when the "deputation to the church" attempts to "deal with Mr. Hooper about the mystery" of the veil?

In "The Minister's Black Veil" by Nathaniel Hawthorne, when the "deputation to the church" attempted to "deal with Mr. Hooper about the mystery of the veil" they did not take any action out of fear of him, as the minister states "Have men avoided me, and women shown no pity, and children screamed and fled, only for my black veil?" The deputation of the church did nothing and had claimed it was "pronouncing the matter too weighty to be handled".