The fears of the priests are not without merit. They had no doubt witnessed or been very aware of the death of Noah at the hands of those who had followed him and abandoned their wives. With that known wrath by those who should have been the most favorable to the priests, it is no wonder that they supposed that the rest of the people might be even less inclined to welcome them back with open arms.
Clearly, however, they also had fears of the Lamanites. They did not immediately take themselves to Shemlon to plead for residence among the Lamanites. They saw themselves as without nation, and required to make their way on their own. This would have been a difficult situation for men who had been used to high luxury. They would now be entirely on their own for everything. Understanding their isolation, they would want women to complete their community, as well as satisfy their carnality.
It is also important to note that they did not return to attempt a similar capture of women of their own culture. It is very likely that they understood very well the nature of this crime, and chose to perpetrate it upon an enemy rather than upon their own people. It may have seemed illegal if Zeniffite/Noahite women were taken, but more acceptable to raid Lamanites and take the women under the same justification as one takes spoils of war.