The answer lies in the scene where one of the female elders reminds William Hurt quite strongly that he, as well as all the other elders, swore to their oath. For one of the elders to go, the oath would be broken. Seems perfectly in character for people like that who have high moral standards (however questionable some of those standards may be).