Translation: The term privily is used with some frequency in the King James Version of the Bible. However, the idiom put away privily is used only once:
Matt. 1:19
19 Then Joseph her husband, being a just man, and not willing to make her a publick example, was minded to put her away privily.
This particular verse, coming as part of the birth narration of the Savior, was probably fairly well known, and is a probable model for the particular idiom Joseph used here. This is not a copy, but a borrowing of a known idiom.