Can you tell how it's done specifically? Are you zfs send'ing to another directory and encrypt it entirely on the fly while transferring to Glacier?

Does it do incremental backup transfer to Glacier or does it have to transfer the entire encrypted blob every time?

No intermediate directory/file at all, all done on the fly.

    sudo zfs send -i <LAST_BACKUPED_SNAPNAME> <CUR_SNAPNAME> | gpg -e -r <YOURKEY> --compress-algo none -v | pv | aws s3 cp --storage-class DEEP_ARCHIVE - s3://<PATH_TO_DEST>.zfs.gpg

The very first time you do it, you will need to do a full backup (ie. without the `-i <...>` option). Afterwards, subsequent backups can be done with the -i, so only the incremental difference will be backed up.

I have a path/naming scheme for the .zfs.gpg files on s3 which include the snapshot from/to names. This allows to determine what the latest backed up snapshot name is (so the next one can be incremental against that). And also use when backing up, since the order or restore matters.

But how do you verify or test your backups in this scenario?

Pretty much the exact reverse of backing it up https://news.ycombinator.com/item?id=29541729

    aws s3 cp s3://... - | gpg -d ... | zfs recv ...

When restoring the order of restores matters, you first need to restore the full snapshot, and then the subsequent incremental ones in order.

I meant, how to test them without incurring a large cost.

Ah gotcha, I haven't done full restore of my main dataset.

I've only verified with a smaller test dataset to validate the workflow on s3 deep archive (retrieval is $0.02/GB). I've done full backup/restore with the zfs send/gpg/recv workflow successfully (to a non aws s3 destination), and used s3 for quite a long time for work and personal without issue, so personally I have high confidence in the entire workflow.