There are multiple reasons why the Lamanites should break off and return to Mulek. Mormon tells us only that they flee rapidly. They also flee because they know that they do not have the time for a long engagement with an enemy army. Even though they might have been larger and able to eventually conquer the Nephites, that would take time, and they would have known that Moroni could then move on Mulek. Of course they did not know that he already had.