Nephi probably surmised that among this fairly large group of people who had assembled near his garden wall there were probably a few Gadianton officials. And so there were. They immediately made their presence known.
There is nothing that makes a corrupt politician more angry than to have someone expose his connection with vice and crime. That is why these Gadiantons angrily attack the words of Nephi.