As has been previously noted, it is quite possible that there is a cultural context to the phrase “unholy temples” that had a stronger image for Alma’s audience than it does for us. In the context of Mesoamerica, there were multiple options for temples, many of them dedicated to foreign gods, which would clearly qualify as “unholy.”
Alma does note that the filthiness is removable – after all, he has called for repentance and baptism. Nevertheless, he does indicate that the ability to remove the filthiness is not permanent.