Alma is addressing the group that arrived with the spokesperson, but he does acknowledge the spokesperson, or “thy brother.” Alma focuses on the assumption that this group has that they are excluded from religion because they are excluded from the place of worship. What Alma will teach them is that religion is not tied to place, and that they may worship anywhere. This might seem somewhat unusual to a modern audience, as we presume what Alma is teaching. Nevertheless, we might better understand if we were to take an analogy.
The best modern corollary is to a temple. That is a particularly sacred place, and the things that go on in that place cannot occur outside of that sacred place. The temple is restrictive, and there are many who cannot “worship their God” in the temple. The difference between this modern example and the one in Antionum is that the entrance requirements for the temple are spiritual, and all are able to meet them. In Antionum the entrance requirement was economic and social. Entrance to the Zoramite sacred space required a change in economic and social status, something to which these people had no access.