Trying to understand confusing 'zfs diff' output

[Juan Manuel Palacios]

Hi everyone,

I'm trying to understand a result I'm seeing when comparing the diff of two
local ZFS snapshots that were successfully replicated to a remote pool over
SSH against the diff of their remote counterparts. The latter shows three
files as having been deleted, while the local diff of the exact same two
snapshots doesn't.

Following are those diffs, with empty lines entered manually into the local
diff where the remote one shows the deleted files; moreover, lines right
above & below this confusing part of the diffs have been trimmed for
brevity's sake, since they were identical across both outputs.

1) Local diff (FreeBSD 10.3-RELEASE-p21 system):

-> zfs diff zroot/mysql at automated_2017-07-31_23:45:04-EDT
zroot/mysql at automated_2017-08-01_23:45:03-EDT | gawk '{ match($2, /\/(.*)/,
matches); printf "%s\t%s\n", $1, matches[1]; }'
(trimmed)
M       mysql/data/knet at 002dlrs/lrs_providers.ibd

M       mysql/data/snap/sessions.ibd
M       mysql/data/knet at 002dlrs/sessions.ibd
M       mysql/data/leads/demo_requests.ibd
M       mysql/tmp/nk-dev.sql.gz


M       mysql/data/knet/account_manager_memberships.ibd
(trimmed)


2) Remote diff (FreeBSD 10.3-RELEASE-p19 system):

-> zfs diff backup/mysql at automated_2017-07-31_23:45:04-EDT
backup/mysql at automated_2017-08-01_23:45:03-EDT | gawk '{ match($2,
/\/mnt\/backup\/(.*)/, matches); printf "%s\t%s\n", $1, matches[1]; }'
(trimmed)
M       mysql/data/knet at 002dlrs/lrs_providers.ibd
-       mysql/tmp/nk-dump--2017-07-19_01:32:16-EDT.sql
M       mysql/data/snap/sessions.ibd
M       mysql/data/knet at 002dlrs/sessions.ibd
M       mysql/data/leads/demo_requests.ibd
M       mysql/tmp/nk-dev.sql.gz
-       mysql/replication/mysql-bin.001241
-       mysql/replication/mysql-bin.001242
M       mysql/data/knet/account_manager_memberships.ibd
(trimmed)

So, as I was saying, the remote diff shows these three files as having been
deleted, because they were:

1) Locally:

-> ls -l /mysql/.zfs/snapshot/automated_2017-07-31_23\:45\:
04-EDT/tmp/nk-dump--2017-07-19_01\:32\:16-EDT.sql
-rw-r--r--  1 jmpp  jmpp    46M Jul 19 01:32 /mysql/.zfs/snapshot/
automated_2017-07-31_23:45:04-EDT/tmp/nk-dump--2017-07-19_01:32:16-EDT.sql

-> ls -l /mysql/.zfs/snapshot/automated_2017-08-01_23\:45\:
03-EDT/tmp/nk-dump--2017-07-19_01\:32\:16-EDT.sql
ls: /mysql/.zfs/snapshot/automated_2017-08-01_23:45:03-
EDT/tmp/nk-dump--2017-07-19_01:32:16-EDT.sql: No such file or directory


2) On the remote pool:

-> ls -l /mnt/backup/mysql/.zfs/snapshot/automated_2017-07-31_
23\:45\:04-EDT/tmp/nk-dump--2017-07-19_01:32:16-EDT.sql
-rw-r--r--  1 1001  1001  48288840 Jul 18 22:32 /mnt/backup/mysql/.zfs/
snapshot/automated_2017-07-31_23:45:04-EDT/tmp/nk-dump--
2017-07-19_01:32:16-EDT.sql

-> ls -l /mnt/backup/mysql/.zfs/snapshot/automated_2017-08-01_
23\:45\:03-EDT/tmp/nk-dump--2017-07-19_01:32:16-EDT.sql
ls: /mnt/backup/mysql/.zfs/snapshot/automated_2017-08-01_
23:45:03-EDT/tmp/nk-dump--2017-07-19_01:32:16-EDT.sql: No such file or
directory

And so for the other two files.

So, my question is, if the remote diff is correct in showing these files as
having been deleted between the two snapshots (it *is* correct in doing
that, right?), why does the local diff not show it also?

Thanks in advance for any help!

-- 
Juan Palacios
Senior Software Architect
<http://www.kognito.com>
135 West 26th St l 12th Floor l NY, NY 10001
212.675.9234 l 646.217.3677

Register for our upcoming webinar with The Healthy Minds Network and
AUCCCD:
Trends in Higher Education Mental Health: Research Highlights
<https://go.kognito.com/higheredtrendswebinar2017>

Connect with us!
<http://twitter.com/kognito>   <http://www.linkedin.com/company/kognito>
<http://www.facebook.com/KognitoConversations/>
<http://www.youtube.com/channel/UC1cyckrxa_lCegQMT2HPS-w>