We do not see the reaction of the servant to this lament (and perhaps this is the best indication that the servant has a literary function, and does not represent a real person). The Lord specifically notes that the destruction of the promising fruit was related to the decision not to destroy the wild fruit - which was the solution encouraged by the servant! The servant's actions have directly led to the destruction of some of the good fruit! (see verses 26 and 27).
It is perhaps instructive that this obvious mistake by the servant is not commented upon. The servant's suggestion was made in earnest, and accepted in hope. That it did not come to fruition does not become the basis for blame of the servant.