The reference in Mosiah 18:17 to "the church of God" is the first time in the Book of Mormon that any mention is made of a formal church organization among the Nephites. Much later in his abridgment, Mormon makes an interesting statement: "And behold, I am called Mormon, being called after the land of Mormon, the land in which Alma did establish the church among the people, yea, the first church which was established among them after their transgression" (3 Nephi 5:12).
What people was Mormon referring to and what was their transgression?
Option #1:
According to Sidney Sperry, "I take it that the 'transgression' referred to by Mormon is the situation that caused the Lord to call the elder Mosiah and his followers from among the main body of the Nephites. At any rate, from the days of the elder Mosiah until Alma set it up, there had been no formal church organization among the Nephites." [Sidney B. Sperry, Book of Mormon Compendium, p. 312]
Option #2:
It is interesting that when all the groups of Nephites, the people of Mosiah2, the people of Limhi, and the people of Alma1 were all gathered in the land of Zarahemla (Mosiah 25:4-6); it was apparently only necessary for Alma to baptize the people of Limhi (Mosiah 25:17-18). Mormon makes the comment that while still in the land of Lehi-Nephi, "king Limhi and many of his people were desirous to be baptized; but there was none in the land that had authority from God" (Mosiah 21:33). If the need for baptism was a sign of the "transgression" mentioned in 3 Nephi 5:12, then the transgression probably happened among the descendants of Zeniff's group. But when? Perhaps it began with Zeniff himself. He certainly was a religious man because he mentions that, "in the strength of the Lord did we go forth to battle against the Lamanites" (Mosiah 9:17) and Limhi mentions that Zeniff "was made king by the voice of the people" (Mosiah 7:9). Nevertheless, he mentions that he was "over-zealous to inherit the land of our fathers" (Mosiah 9:3). Mention is also made that he "went in unto the [Lamanite] king, and he covenanted with me that I might possess the land of Lehi-Nephi, and the land of Shilom" (Mosiah 9:6). When Ammon came from the land of Zarahemla to rescue Limhi's people, Limhi gathered his people together and reiterated this information, "ye all are witnesses this day, that Zeniff, who was made king over this people, he being over-zealous to inherit the land of his fathers, therefore being deceived by the cunning and craftiness of king Laman, who having entered into a treaty with king Zeniff, and having yielded up into his hands the possessions of a part of the land . . . and all this he did, for the sole purpose of bringing this people into subjection or into bondage" (Mosiah 7:21-22). Was the covenant with king Laman the beginning of the "transgression" mentioned in 3 Nephi 5:12? It is interesting that Zeniff's people never bothered to make contact with the Nephites in the land of Zarahemla. King Mosiah1, King Benjamin, and King Mosiah2 all apparently had authority from God. Perhaps Zeniff began the "transgression" by placing his people away from that authority and under covenant not to make contact with that authority.
Option #3:
On the other hand, perhaps the "transgression" happened with King Noah. The record states that "he did not walk in the ways of his father. For behold, he did not keep the commandments of God, but he did walk after the desires of his own heart" (Mosiah 11:1-2). It also says that he "changed the affairs of the kingdom. For he put down all the priests that had been consecrated by his father, and consecrated new ones in their stead" (Mosiah 11:4-5). King Noah and his priests were also guilty of killing the Lord's prophet Abinadi (Mosiah 17:11:13). [Alan C. Miner, Personal Notes] [See the commentary on 1 Nephi 4:26; 2 Nephi 6:2; Mosiah 6:3; and Mosiah 25:19,23]