zfs send/recv: STILL invalid Backup Stream
Larry Rosenman
ler at lerctr.org
Thu Jul 24 23:36:40 UTC 2014
On 2014-07-24 15:44, Larry Rosenman wrote:
> On 2014-07-24 15:26, Mark Martinec wrote:
>> 2014-07-24 21:57, Larry Rosenman wrote:
>>
>>>>>> Sending zroot/home at zxfer_26699_20140724135840 to
>>>>>> zroot/backups/TBH/zroot/home.
>>>>> Write failed: Cannot allocate memory
>>>> ====================================
>>>>> cannot receive new filesystem stream: invalid backup stream
>>>>> Error when zfs send/receiving.
>>>>> borg.lerctr.org /home/ler #
>>>>>
>>>>> well that's different.......
>>>>
>>>> Sounds familiar, check my posting of today and links therein:
>>>>
>>>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html
>>
>>> I'm not using netgraph to the best of my knowledge....
>>
>> Check to make sure:
>> vmstat -z | fgrep NetGraph
>>
>>> and the only fails on the SENDING host are:
>>> 8 Bucket: 64, 0, 41, 3555, 257774, 11,
>>> 0
>>> 12 Bucket: 96, 0, 96, 2569, 123653, 0,
>>> 0
>>> 16 Bucket: 128, 0, 17195, 506, 215573, 0,
>>> 0
>>> 32 Bucket: 256, 0, 340, 4670, 900638, 50,
>>> 0
>>> 64 Bucket: 512, 0, 10691, 365,
>>> 546888,185232,
>>> 0
>>> 128 Bucket: 1024, 0, 3563, 905, 348419, 0,
>>> 0
>>> 256 Bucket: 2048, 0, 2872, 162, 249995,59834,
>>> 0
>>> vmem btag: 56, 0, 192811, 51500, 502264,1723,
>>> 0
>>
>>
>> Adam Vande More gave other suggestions on that thread from 2011:
>>
>>
>> http://lists.freebsd.org/pipermail/freebsd-emulation/2011-July/008971.html
>>
>>
>> Mark
>
>
> thebighonker.lerctr.org /home/ler $ vmstat -z|fgrep NetGraph
> thebighonker.lerctr.org /home/ler $
>
> This on physical hardware, FWIW.
>
> thebighonker.lerctr.org /home/ler $ vmstat -z
> ITEM SIZE LIMIT USED FREE REQ FAIL
> SLEEP
>
> UMA Kegs: 384, 0, 202, 8, 202, 0,
> 0
> UMA Zones: 1664, 0, 202, 0, 202, 0,
> 0
> UMA Slabs: 80, 0, 361152, 38448, 1561588, 0,
> 0
> UMA RCntSlabs: 88, 0, 4360, 5, 4360, 0,
> 0
> UMA Hash: 256, 0, 8, 82, 82, 0,
> 0
> 4 Bucket: 32, 0, 4816, 33684, 2974043, 0,
> 0
> 6 Bucket: 48, 0, 6118, 2514, 624434, 0,
> 0
> 8 Bucket: 64, 0, 161, 3435, 265443, 11,
> 0
> 12 Bucket: 96, 0, 177, 2488, 130226, 0,
> 0
> 16 Bucket: 128, 0, 20301, 531, 224788, 0,
> 0
> 32 Bucket: 256, 0, 433, 4577, 945047, 50,
> 0
> 64 Bucket: 512, 0, 11137, 575, 579209,211292,
> 0
> 128 Bucket: 1024, 0, 3509, 959, 369839, 0,
> 0
> 256 Bucket: 2048, 0, 2847, 187, 269044,59834,
> 0
> vmem btag: 56, 0, 209393, 34918, 557576,1723,
> 0
> VM OBJECT: 256, 0, 91384, 791, 3480613, 0,
> 0
> RADIX NODE: 144, 0, 227133, 37683,16029821, 0,
> 0
> MAP: 240, 0, 3, 61, 3, 0,
> 0
> KMAP ENTRY: 128, 0, 10, 269, 10, 0,
> 0
> MAP ENTRY: 128, 0, 12666, 23604, 9357932, 0,
> 0
> VMSPACE: 448, 0, 108, 306, 75357, 0,
> 0
> fakepg: 104, 0, 0, 0, 0, 0,
> 0
> mt_zone: 4112, 0, 368, 0, 368, 0,
> 0
> 16: 16, 0, 267839, 2488,45160196, 0,
> 0
> 32: 32, 0, 148139, 1486,39174301, 0,
> 0
> 64: 64, 0, 291630, 504078,40379996, 0,
> 0
> 128: 128, 0, 182409, 35490,63455920, 0,
> 0
> 256: 256, 0, 89368, 230912,35841036, 0,
> 0
> 512: 512, 0, 741, 3987,24215854, 0,
> 0
> 1024: 1024, 0, 1371, 1385, 1726440, 0,
> 0
> 2048: 2048, 0, 277, 417,10240207, 0,
> 0
> 4096: 4096, 0, 38306, 17186, 913638, 0,
> 0
> SLEEPQUEUE: 80, 0, 613, 658, 613, 0,
> 0
> 64 pcpu: 8, 0, 1884, 1188, 1884, 0,
> 0
> Files: 80, 0, 829, 1571, 4279893, 0,
> 0
> rl_entry: 40, 0, 209, 1891, 209, 0,
> 0
> TURNSTILE: 136, 0, 613, 307, 613, 0,
> 0
> umtx pi: 96, 0, 0, 0, 0, 0,
> 0
> MAC labels: 40, 0, 0, 0, 0, 0,
> 0
> PROC: 1208, 0, 131, 121, 76219, 0,
> 0
> THREAD: 1168, 0, 519, 93, 2039, 0,
> 0
> cpuset: 72, 0, 270, 280, 426, 0,
> 0
> cyclic_id_cache: 64, 0, 0, 0, 0, 0,
> 0
> audit_record: 1248, 0, 0, 0, 0, 0,
> 0
> mbuf_packet: 256, 6519810, 4081, 3256, 4910227, 0,
> 0
> mbuf: 256, 6519810, 1052, 3251,93444045, 0,
> 0
> mbuf_cluster: 2048, 1018718, 7337, 13, 7337, 0,
> 0
> mbuf_jumbo_page: 4096, 509359, 12, 673,15015378, 0,
> 0
> mbuf_jumbo_9k: 9216, 150921, 0, 0, 0, 0,
> 0
> mbuf_jumbo_16k: 16384, 84893, 0, 0, 0, 0,
> 0
> mbuf_ext_refcnt: 4, 0, 0, 2510, 85094, 0,
> 0
> dtrace_state_cache: 4096, 0, 0, 0, 0, 0,
> 0
> g_bio: 248, 0, 0, 1600,29986631, 0,
> 0
> ttyinq: 160, 0, 180, 570, 2100, 0,
> 0
> DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0,
> 0
> ttyoutq: 256, 0, 95, 640, 1093, 0,
> 0
> ata_request: 336, 0, 0, 242, 33344, 0,
> 0
> vtnet_tx_hdr: 24, 0, 0, 0, 0, 0,
> 0
> cryptop: 88, 0, 0, 0, 0, 0,
> 0
> cryptodesc: 72, 0, 0, 0, 0, 0,
> 0
> FPU_save_area: 512, 0, 0, 0, 0, 0,
> 0
> taskq_zone: 48, 0, 1, 3070, 194184, 0,
> 0
> taskq_zone: 48, 0, 0, 0, 0, 0,
> 0
> VNODE: 472, 0, 109527, 66161, 2663226, 0,
> 0
> VNODEPOLL: 112, 0, 8, 762, 10, 0,
> 0
> BUF TRIE: 144, 0, 0, 105354, 0, 0,
> 0
> S VFS Cache: 108, 0, 114973, 60552, 2293717, 0,
> 0
> STS VFS Cache: 148, 0, 0, 0, 0, 0,
> 0
> L VFS Cache: 328, 0, 468, 145536, 800279, 0,
> 0
> LTS VFS Cache: 368, 0, 0, 0, 0, 0,
> 0
> NAMEI: 1024, 0, 0, 156,23968745, 0,
> 0
> DIRHASH: 1024, 0, 0, 0, 0, 0,
> 0
> NCLNODE: 528, 0, 0, 0, 0, 0,
> 0
> Mountpoints: 816, 0, 11, 19, 11, 0,
> 0
> range_seg_cache: 64, 0, 58377, 7157, 3500156, 0,
> 0
> zio_cache: 920, 0, 1, 42591,104595517, 0,
> 0
> zio_link_cache: 48, 0, 0, 44073,72482492, 0,
> 0
> zio_buf_512: 512, 0, 552060, 193836, 1557459, 0,
> 0
> zio_data_buf_512: 512, 0, 114904, 2056, 526943, 0,
> 0
> zio_buf_1024: 1024, 0, 4246, 58, 56140, 0,
> 0
> zio_data_buf_1024: 1024, 0, 51117, 1255, 211875, 0,
> 0
> zio_buf_1536: 1536, 0, 1957, 59, 72179, 0,
> 0
> zio_data_buf_1536: 1536, 0, 22199, 719, 127817, 0,
> 0
> zio_buf_2048: 2048, 0, 691, 43, 217643, 0,
> 0
> zio_data_buf_2048: 2048, 0, 13419, 1147, 87149, 0,
> 0
> zio_buf_2560: 2560, 0, 455, 20, 16265, 0,
> 0
> zio_data_buf_2560: 2560, 0, 9335, 493, 73967, 0,
> 0
> zio_buf_3072: 3072, 0, 245, 16, 4658, 0,
> 0
> zio_data_buf_3072: 3072, 0, 6940, 512, 53932, 0,
> 0
> zio_buf_3584: 3584, 0, 248, 10, 38558, 0,
> 0
> zio_data_buf_3584: 3584, 0, 5076, 352, 43162, 0,
> 0
> zio_buf_4096: 4096, 0, 464, 729,14217630, 0,
> 0
> zio_data_buf_4096: 4096, 0, 3857, 288, 41192, 0,
> 0
> zio_buf_5120: 5120, 0, 167, 9, 6442, 0,
> 0
> zio_data_buf_5120: 5120, 0, 6025, 461, 51297, 0,
> 0
> zio_buf_6144: 6144, 0, 115, 6, 3196, 0,
> 0
> zio_data_buf_6144: 6144, 0, 4453, 309, 38753, 0,
> 0
> zio_buf_7168: 7168, 0, 85, 12, 36945, 0,
> 0
> zio_data_buf_7168: 7168, 0, 3373, 273, 29007, 0,
> 0
> zio_buf_8192: 8192, 0, 44, 39, 1373975, 0,
> 0
> zio_data_buf_8192: 8192, 0, 2543, 205, 28736, 0,
> 0
> zio_buf_10240: 10240, 0, 66, 7, 2534, 0,
> 0
> zio_data_buf_10240: 10240, 0, 3986, 275, 39568, 0,
> 0
> zio_buf_12288: 12288, 0, 71, 15, 654156, 0,
> 0
> zio_data_buf_12288: 12288, 0, 2937, 209, 37077, 0,
> 0
> zio_buf_14336: 14336, 0, 32, 9, 3164, 0,
> 0
> zio_data_buf_14336: 14336, 0, 2236, 194, 23959, 0,
> 0
> zio_buf_16384: 16384, 0, 72627, 226, 2784131, 0,
> 0
> zio_data_buf_16384: 16384, 0, 1778, 155, 22372, 0,
> 0
> zio_buf_20480: 20480, 0, 210, 14, 598092, 0,
> 0
> zio_data_buf_20480: 20480, 0, 2724, 216, 29228, 0,
> 0
> zio_buf_24576: 24576, 0, 18, 14, 270172, 0,
> 0
> zio_data_buf_24576: 24576, 0, 2101, 173, 21241, 0,
> 0
> zio_buf_28672: 28672, 0, 10, 54, 503029, 0,
> 0
> zio_data_buf_28672: 28672, 0, 1406, 113, 18667, 0,
> 0
> zio_buf_32768: 32768, 0, 10, 11, 77998, 0,
> 0
> zio_data_buf_32768: 32768, 0, 1088, 96, 14116, 0,
> 0
> zio_buf_36864: 36864, 0, 6, 11, 79009, 0,
> 0
> zio_data_buf_36864: 36864, 0, 871, 87, 10066, 0,
> 0
> zio_buf_40960: 40960, 0, 8, 10, 80622, 0,
> 0
> zio_data_buf_40960: 40960, 0, 785, 57, 8149, 0,
> 0
> zio_buf_45056: 45056, 0, 50, 10, 70143, 0,
> 0
> zio_data_buf_45056: 45056, 0, 659, 61, 6889, 0,
> 0
> zio_buf_49152: 49152, 0, 190, 9, 73508, 0,
> 0
> zio_data_buf_49152: 49152, 0, 512, 42, 8420, 0,
> 0
> zio_buf_53248: 53248, 0, 25, 10, 99339, 0,
> 0
> zio_data_buf_53248: 53248, 0, 465, 30, 4838, 0,
> 0
> zio_buf_57344: 57344, 0, 2, 15, 95818, 0,
> 0
> zio_data_buf_57344: 57344, 0, 337, 33, 4753, 0,
> 0
> zio_buf_61440: 61440, 0, 1, 23, 98972, 0,
> 0
> zio_data_buf_61440: 61440, 0, 294, 33, 3792, 0,
> 0
> zio_buf_65536: 65536, 0, 1, 9, 114664, 0,
> 0
> zio_data_buf_65536: 65536, 0, 291, 29, 5152, 0,
> 0
> zio_buf_69632: 69632, 0, 2, 10, 90765, 0,
> 0
> zio_data_buf_69632: 69632, 0, 291, 22, 3555, 0,
> 0
> zio_buf_73728: 73728, 0, 1, 12, 87316, 0,
> 0
> zio_data_buf_73728: 73728, 0, 219, 23, 3361, 0,
> 0
> zio_buf_77824: 77824, 0, 2, 9, 101758, 0,
> 0
> zio_data_buf_77824: 77824, 0, 196, 25, 2907, 0,
> 0
> zio_buf_81920: 81920, 0, 0, 15, 129645, 0,
> 0
> zio_data_buf_81920: 81920, 0, 187, 18, 3578, 0,
> 0
> zio_buf_86016: 86016, 0, 1, 7, 99241, 0,
> 0
> zio_data_buf_86016: 86016, 0, 180, 22, 26051, 0,
> 0
> zio_buf_90112: 90112, 0, 1, 12, 51491, 0,
> 0
> zio_data_buf_90112: 90112, 0, 163, 18, 2438, 0,
> 0
> zio_buf_94208: 94208, 0, 0, 9, 51043, 0,
> 0
> zio_data_buf_94208: 94208, 0, 149, 12, 2288, 0,
> 0
> zio_buf_98304: 98304, 0, 2, 8, 50322, 0,
> 0
> zio_data_buf_98304: 98304, 0, 170, 25, 3002, 0,
> 0
> zio_buf_102400: 102400, 0, 2, 10, 44642, 0,
> 0
> zio_data_buf_102400: 102400, 0, 145, 19, 2017, 0,
> 0
> zio_buf_106496: 106496, 0, 0, 15, 161697, 0,
> 0
> zio_data_buf_106496: 106496, 0, 161, 16, 1944, 0,
> 0
> zio_buf_110592: 110592, 0, 0, 19, 146777, 0,
> 0
> zio_data_buf_110592: 110592, 0, 97, 15, 1636, 0,
> 0
> zio_buf_114688: 114688, 0, 2, 8, 51698, 0,
> 0
> zio_data_buf_114688: 114688, 0, 115, 20, 2691, 0,
> 0
> zio_buf_118784: 118784, 0, 0, 8, 62915, 0,
> 0
> zio_data_buf_118784: 118784, 0, 96, 11, 1440, 0,
> 0
> zio_buf_122880: 122880, 0, 0, 12, 85924, 0,
> 0
> zio_data_buf_122880: 122880, 0, 73, 17, 1510, 0,
> 0
> zio_buf_126976: 126976, 0, 1, 11, 86133, 0,
> 0
> zio_data_buf_126976: 126976, 0, 80, 9, 1351, 0,
> 0
> zio_buf_131072: 131072, 0, 43, 43, 548063, 0,
> 0
> zio_data_buf_131072: 131072, 0, 67545, 2, 646567, 0,
> 0
> lz4_ctx: 16384, 0, 0, 16, 2249605, 0,
> 0
> sa_cache: 80, 0, 109455, 66445, 2663034, 0,
> 0
> dnode_t: 744, 0, 501793, 259152, 873895, 0,
> 0
> dmu_buf_impl_t: 224, 0, 635273, 343604, 3042828, 0,
> 0
> arc_buf_hdr_t: 216, 0, 643049, 141229, 3158205, 0,
> 0
> arc_buf_t: 72, 0, 481443, 119872, 3705908, 0,
> 0
> zil_lwb_cache: 192, 0, 8, 5072, 66838, 0,
> 0
> zfs_znode_cache: 368, 0, 109455, 66105, 2663034, 0,
> 0
> pipe: 744, 0, 113, 177, 66135, 0,
> 0
> procdesc: 128, 0, 0, 0, 0, 0,
> 0
> ksiginfo: 112, 0, 237, 2528, 8311580, 0,
> 0
> itimer: 352, 0, 1, 32, 1, 0,
> 0
> KNOTE: 128, 0, 376, 1391,17307485, 0,
> 0
> socket: 696, 523340, 258, 302, 668458, 0,
> 0
> unpcb: 240, 523344, 151, 649, 130961, 0,
> 0
> ipq: 56, 31879, 0, 2059, 317, 0,
> 0
> udp_inpcb: 392, 523340, 19, 421, 469503, 0,
> 0
> udpcb: 16, 523586, 19, 2993, 469503, 0,
> 0
> tcp_inpcb: 392, 523340, 100, 370, 67983, 0,
> 0
> tcpcb: 1024, 523340, 78, 234, 67983, 0,
> 0
> tcptw: 88, 27810, 22, 1013, 22066, 0,
> 0
> syncache: 160, 15375, 0, 675, 45270, 0,
> 0
> hostcache: 136, 15370, 255, 499, 3892, 0,
> 0
> tcpreass: 40, 63700, 0, 2600, 7616, 0,
> 0
> sackhole: 32, 0, 0, 1875, 7184, 0,
> 0
> sctp_ep: 1408, 523340, 0, 0, 0, 0,
> 0
> sctp_asoc: 2352, 40000, 0, 0, 0, 0,
> 0
> sctp_laddr: 48, 80012, 0, 1328, 8, 0,
> 0
> sctp_raddr: 728, 80000, 0, 0, 0, 0,
> 0
> sctp_chunk: 136, 400026, 0, 0, 0, 0,
> 0
> sctp_readq: 104, 400026, 0, 0, 0, 0,
> 0
> sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0,
> 0
> sctp_asconf: 40, 400000, 0, 0, 0, 0,
> 0
> sctp_asconf_ack: 48, 400060, 0, 0, 0, 0,
> 0
> udplite_inpcb: 392, 523340, 0, 0, 0, 0,
> 0
> ripcb: 392, 523340, 2, 58, 3, 0,
> 0
> rtentry: 200, 0, 32, 408, 34, 0,
> 0
> selfd: 56, 0, 333, 2223,29658712, 0,
> 0
> SWAPMETA: 288, 2037438, 0, 0, 0, 0,
> 0
>
> thebighonker.lerctr.org /home/ler
and I ran the script again.... It got further:
18:20:52 25.1G zroot/home at zxfer_26699_20140724135840
18:20:53 25.1G zroot/home at zxfer_26699_20140724135840
18:20:54 25.1G zroot/home at zxfer_26699_20140724135840
18:20:55 25.1G zroot/home at zxfer_26699_20140724135840
18:20:56 25.1G zroot/home at zxfer_26699_20140724135840
18:20:57 25.1G zroot/home at zxfer_26699_20140724135840
18:20:58 25.1G zroot/home at zxfer_26699_20140724135840
Connection to home.lerctr.org closed by remote host.
warning: cannot send 'zroot/home at zxfer_26699_20140724135840': Broken
pipe
TIME SENT SNAPSHOT
warning: cannot send 'zroot/home at 2014-07-24': Broken pipe
cannot open 'zroot/backups/TBH2/home': dataset does not exist
borg.lerctr.org /home/ler/bin #
BUT we still fail.
#!/bin/sh
DATE=`date "+%Y-%m-%d"`
#DATE2=2013-03-24
#DATE2=`date -v "-1d" "+%Y-%m-%d"`
# snap the source
ssh root at tbh.lerctr.org zfs snapshot -r zroot@${DATE}
# zfs copy the source to here.
ssh root at tbh.lerctr.org "zfs send -v -R zroot@${DATE} | \
ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH2\""
# make sure we NEVER allow the backup stuff to automount.
/sbin/zfs list -H -t filesystem -r zroot/backups/TBH2| \
awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
borg.lerctr.org /home/ler/bin #
this was to a FRESH zroot/backups/TBH2
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: ler at lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
More information about the freebsd-current
mailing list