From nobody Thu Apr 21 21:58:34 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CB8EC198E2AC for ; Thu, 21 Apr 2022 21:58:42 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-ua1-x936.google.com (mail-ua1-x936.google.com [IPv6:2607:f8b0:4864:20::936]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kkryn6ZrGz3wLl; Thu, 21 Apr 2022 21:58:41 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: by mail-ua1-x936.google.com with SMTP id b11so2308650uaq.2; Thu, 21 Apr 2022 14:58:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lATVujw8CpuVgxuYeFdXPHsLZJA8F+0EsO7Ldqobcaw=; b=hBeKwEdwAZPJgjxjRXhxGkJn/J/KgSjF9Xafpq9zay+n93g8wYuJVbKo6kGXm0FkRL saeNAe+xFNNya2l/ioN0//m6kUdOjVOsyA8dJXxtRtjG6E2kQ7csjmSgu5FVOwitV+8g DKunaRXeXe5SJQFpZZFtHcHbhDCam0XluyJj4MxsNj1sW9f9MSI2FfuwuD2QRMzd6t7h 9Hry6PfEsk4VWJAvJw0kRcLqZfLRj5dms/VUD4HcY/RJxUPFJQf7aqkEH6h8tn8mG5I2 0dKrCxx8T0pGVHJN/LxoojVpbP23qfGNBv7+eiLoZLsu9SYATzfFazPyYnjL7jyBpjiv K0kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lATVujw8CpuVgxuYeFdXPHsLZJA8F+0EsO7Ldqobcaw=; b=FRi8vvoFVROg50eHASJnDhCxLZYhqQPbmiiLfOm6BNNGbTr3fO6tnhhC1UuWVd03fR ge6O/eej26Pea6oTCRfSOa//sJxXF7k4gX2HWpKeqp85nkt1Pju0Q+lgn56Yluq7QgUF BksHYsdIpWsZEyHB+ML07olk22XdQegbogSTIavEGP2Qsmp+kA60LyAvTswLMb1GVfAq UHnFbhWvMQtvG+THSqiMWFP7XLrEbeXcy3CZsRt1Mf9DaJ2fs+QUzfK5J+48bf6zrjis PJEOFcApuVN7YRZhQx3HTzEWjd+EA5XnyVR8thxmRYVpjHVcWRU8zqGpjbBgBiMuXWIh pGzQ== X-Gm-Message-State: AOAM530XGtM/6Yy4Xo6SJvpo4rt1HLnrBH/6Ae06WvEsZulk18zDCBUV 3iEtQGGCZio5WY6lgt06bc6ceftOaDq9UadzB5hVTVG3VXs= X-Google-Smtp-Source: ABdhPJzijzo6ioS00wuiKABO8yIqAdnY++PvFY2VVoyDibH6mUISnEn2S3M0UFH97YKrzta/9OtNm9nXyxcfDWYs2xQ= X-Received: by 2002:a9f:35f0:0:b0:34c:2205:fe50 with SMTP id u45-20020a9f35f0000000b0034c2205fe50mr635787uad.2.1650578321144; Thu, 21 Apr 2022 14:58:41 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <94c10113c3964b1bbd3f4ddf934c23cd@MAIL-HUB.pai.local> In-Reply-To: From: Rich Date: Thu, 21 Apr 2022 17:58:34 -0400 Message-ID: Subject: Re: ZFS reservations for type=volume To: Alan Somers Cc: Michael Jung , freebsd-fs Content-Type: multipart/alternative; boundary="00000000000081810c05dd313637" X-Rspamd-Queue-Id: 4Kkryn6ZrGz3wLl X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=hBeKwEdw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rincebrain@gmail.com designates 2607:f8b0:4864:20::936 as permitted sender) smtp.mailfrom=rincebrain@gmail.com X-Spamd-Result: default: False [-2.74 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(0.21)[0.211]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::936:from]; MLMMJ_DEST(0.00)[freebsd-fs]; NEURAL_HAM_SHORT(-0.95)[-0.953]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --00000000000081810c05dd313637 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable (My FBSD dev env isn't booted, but Linux and FBSD are the same in this regard...) # zpool list workspace NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED - # zfs create workspace/testme # zfs list workspace workspace/testme NAME USED AVAIL REFER MOUNTPOINT workspace 13.0T 21.5T 48.9G /workspace workspace/testme 96K 21.5T 96K /workspace/testme # df -h /workspace/testme /workspace/ Filesystem Size Used Avail Use% Mounted on workspace/testme 22T 128K 22T 1% /workspace/testme workspace 22T 49G 22T 1% /workspace # zfs set reservation=3D1T workspace/testme # zpool list workspace NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT workspace 35.4T 12.9T 22.5T - - 3% 36% 1.00x DEGRADED - # zfs list workspace NAME USED AVAIL REFER MOUNTPOINT workspace 14.0T 20.5T 48.9G /workspace workspace/testme 96K 21.5T 96K /workspace/testme # df -h /workspace/testme /workspace/ Filesystem Size Used Avail Use% Mounted on workspace/testme 22T 128K 22T 1% /workspace/testme workspace 21T 49G 21T 1% /workspace # That is, AIUI, reservation counts against the USED/AVAIL/FREE shown by "zfs list" or "df" of everything not the reservation-holder , not against the "ALLOC" shown by zpool list...unless it's actually allocated, of course. - Rich On Thu, Apr 21, 2022 at 5:46 PM Alan Somers wrote: > Isn't 1.52T the "AVAIL" value, and 24.6T is the "USED"? > > On Thu, Apr 21, 2022 at 2:56 PM Michael Jung > wrote: > > > > Ok that makes sense... so even though running ZFS since 9.x I'll ask th= e > newbie question.. > > > > If it contributes to the parents "used" value why is not reflected here > in "USED" or in "AVAIL"? > > > > I would expect USED to be my reservation amount of 15.6T + whatever > space was > > being used by other things on the pool. "USED" still sits at 1.52T. > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE > > > > Thanks again. > > > > -----Original Message----- > > From: Alan Somers [mailto:asomers@freebsd.org] > > Sent: Thursday, April 21, 2022 4:40 PM > > To: Michael Jung > > Cc: freebsd-fs > > Subject: Re: ZFS reservations for type=3Dvolume > > > > A dataset's reservation is local. It doesn't contribute to its parent's > reservation. Otherwise, you wouldn't be able to separately set a > reservation on the parent. But it _does_ contribute to the parent's "used= " > value. In that way, you're prevented from reserving too much data on the > parent's children. > > -Alan > > > > On Thu, Apr 21, 2022 at 2:28 PM Michael Jung < > mikej@paymentallianceintl.com> wrote: > > > > > > I have a zfs block dataset raid-5400-1/esxi-store1 that I share as an > > > iscsi target and that works great. I have set a reservation > > > > > > on that block device equal to its size so that is not sparse and thus > > > while I could over provision guests on the provided LUN, > > > > > > the storage presented as the LUN capacity will always be available. A= t > least this is what I want to achieve. > > > > > > > > > > > > What I find strange is that the reservation does not seem to be > > > applied to the ZFS pool =E2=80=98raid-5400-1=E2=80=99. Do you really = need > > > > > > to set your maximum reservation at the pool level, and then apply > > > reservations to all datasets on that volume? And if so > > > > > > I would assume you could never set reservations for datasets totaling > more than what was reserved for the pool =E2=80=98raid-5400-1=E2=80=99. > > > > > > > > > > > > I could build out a test environment and figure out constraints but > I=E2=80=99d really like to know the =E2=80=9Chow it is supposed to work= =E2=80=9D > > > > > > not the =E2=80=9Chow I find it to work=E2=80=9D. > > > > > > > > > > > > Thanks in advance. > > > > > > > > > > > > FreeBSD 14.0-CURRENT #4 main-n253875-8e72f458c6d: > > > > > > > > > > > > (this is a raidz2 pool =E2=80=93 not my draid pool) > > > > > > > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zfs list -o > > > name,type,used,avail,ratio,compression,reservation,refreservation,vol= s > > > ize raid-5400-1 > > > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > > > > > raid-5400-1 filesystem 18.0T 2.08T 1.36x on none none - <- no > reservation @pool > > > > > > root@draid:/usr/src/contrib/bearssl # > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zfs list -o > > > name,type,used,avail,ratio,compression,reservation,refreservation,vol= s > > > ize raid-5400-1/esxi-store1 > > > > > > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE > > > > > > raid-5400-1/esxi-store1 volume 16.9T 18.5T 1.78x zstd 15.6T 16.9T > 15.6T <- reservation @dataset > > > > > > > > > > > > root@draid:/usr/src/contrib/bearssl # zpool list > > > > > > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > > > > > > ccache 9.50G 9.10G 406M - - 88% 95% 1.00x ONLINE - > > > > > > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE - <- Free does > not reflect reservation @pool > > > > > > tank 18.5T 605G 17.9T - - 0% 3% 1.00x ONLINE - > > > > > > zfsroot 103G 33.3G 69.7G - - 31% 32% 1.00x ONLINE - > > > > > > root@draid:/usr/src/contrib/bearssl # > > > > > > > > > > > > > > > > > > CONFIDENTIALITY NOTE: This message is intended only for the use of th= e > > > individual or entity to whom it is addressed and may contain > > > information that is privileged, confidential, and exempt from > > > disclosure under applicable law. If the reader of this message is not > > > the intended recipient, you are hereby notified that any > > > dissemination, distribution or copying of this communication is > > > strictly prohibited. If you have received this transmission in error, > > > please notify us by telephone at (502) 212-4000 or notify us at: PAI, > > > Dept. 99, > > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245 > > > > > > > > > > > > > > > > > > Disclaimer > > > > > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and m= ay > be unlawful. > > > > > > This email has been scanned for viruses and malware, and may have bee= n > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. > > > > > > > > > > CONFIDENTIALITY NOTE: This message is intended only for the use > > of the individual or entity to whom it is addressed and may > > contain information that is privileged, confidential, and > > exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby > > notified that any dissemination, distribution or copying > > of this communication is strictly prohibited. If you have > > received this transmission in error, please notify us by > > telephone at (502) 212-4000 or notify us at PAI, Dept. 99, > > 2101 High Wickham Place, Suite 101, Louisville, KY 40245 > > > > > > Disclaimer > > > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and m= ay > be unlawful. > > > > This email has been scanned for viruses and malware, and may have been > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. > > --00000000000081810c05dd313637 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
(My FBSD dev env isn't booted, but Linux and FBSD= =C2=A0are the same in this regard...)

# zpool list work= space
NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE =C2=A0ALLOC =C2=A0 FREE =C2= =A0CKPOINT =C2=A0EXPANDSZ =C2=A0 FRAG =C2=A0 =C2=A0CAP =C2=A0DEDUP =C2=A0 = =C2=A0HEALTH =C2=A0ALTROOT
workspace =C2=A035.4T =C2=A012.9T =C2=A022.5T= =C2=A0 =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 =C2=A0 3= % =C2=A0 =C2=A036% =C2=A01.00x =C2=A0DEGRADED =C2=A0-
# zfs create works= pace/testme
# zfs list workspace workspace/testme
NAME =C2=A0 =C2=A0 = =C2=A0 =C2=A0USED =C2=A0AVAIL =C2=A0 =C2=A0 REFER =C2=A0MOUNTPOINT
works= pace =C2=A013.0T =C2=A021.5T =C2=A0 =C2=A0 48.9G =C2=A0/workspace
works= pace/testme =C2=A0 =C2=A096K =C2=A021.5T =C2=A0 =C2=A0 =C2=A0 96K =C2=A0/wo= rkspace/testme
# df -h /workspace/testme /workspace/
Filesystem =C2= =A0 =C2=A0 =C2=A0 =C2=A0Size =C2=A0Used Avail Use% Mounted on
workspace/= testme =C2=A0 22T =C2=A0128K =C2=A0 22T =C2=A0 1% /workspace/testme
work= space =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A022T =C2=A0 49G =C2=A0 22T =C2=A0 1%= /workspace
# zfs set reservation=3D1T workspace/testme
# zpool = list workspace
NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE =C2=A0ALLOC =C2=A0 F= REE =C2=A0CKPOINT =C2=A0EXPANDSZ =C2=A0 FRAG =C2=A0 =C2=A0CAP =C2=A0DEDUP = =C2=A0 =C2=A0HEALTH =C2=A0ALTROOT
workspace =C2=A035.4T =C2=A012.9T =C2= =A022.5T =C2=A0 =C2=A0 =C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0 = =C2=A0 3% =C2=A0 =C2=A036% =C2=A01.00x =C2=A0DEGRADED =C2=A0-
# zfs= list workspace
NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0USED =C2=A0AVAIL =C2=A0 = =C2=A0 REFER =C2=A0MOUNTPOINT
workspace =C2=A014.0T =C2=A020.5T =C2=A0 = =C2=A0 48.9G =C2=A0/workspace
workspace/testme =C2=A0 =C2=A096K = =C2=A021.5T =C2=A0 =C2=A0 =C2=A0 96K =C2=A0/workspace/testme
# df -h /wo= rkspace/testme /workspace/
Filesystem =C2=A0 =C2=A0 =C2=A0 =C2=A0Size = =C2=A0Used Avail Use% Mounted on
workspace/testme =C2=A0 22T =C2=A0128K = =C2=A0 22T =C2=A0 1% /workspace/testme
workspace =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A021T =C2=A0 49G =C2=A0 21T =C2=A0 1% /workspace
#
=
That is, AIUI, reservation counts against the USED/AVAIL/FRE= E shown by "zfs list" or "df" of everything not the res= ervation-holder , not against the "ALLOC" shown by zpool list...u= nless it's actually allocated, of course.

- Ri= ch

On Thu, Apr 21, 2022 at 5:46 PM Alan Somers <asomers@freebsd.org> wrote:
Isn't 1.52T the &quo= t;AVAIL" value, and 24.6T is the "USED"?

On Thu, Apr 21, 2022 at 2:56 PM Michael Jung
<mike= j@paymentallianceintl.com> wrote:
>
> Ok that makes sense... so even though running ZFS since 9.x I'll a= sk the newbie question..
>
> If it contributes to the parents "used" value why is not ref= lected here in "USED" or in "AVAIL"?
>
> I would expect USED to be my reservation amount of 15.6T + whatever sp= ace was
> being used by other things on the pool. "USED" still sits at= 1.52T.
>
> NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE
> raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE
>
> Thanks again.
>
> -----Original Message-----
> From: Alan Somers [mailto:asomers@freebsd.org]
> Sent: Thursday, April 21, 2022 4:40 PM
> To: Michael Jung <mikej@paymentallianceintl.com>
> Cc: freebsd-fs <freebsd-fs@freebsd.org>
> Subject: Re: ZFS reservations for type=3Dvolume
>
> A dataset's reservation is local. It doesn't contribute to its= parent's reservation. Otherwise, you wouldn't be able to separatel= y set a reservation on the parent. But it _does_ contribute to the parent&#= 39;s "used" value. In that way, you're prevented from reservi= ng too much data on the parent's children.
> -Alan
>
> On Thu, Apr 21, 2022 at 2:28 PM Michael Jung <mikej@paymentallianceintl.com= > wrote:
> >
> > I have a zfs block dataset raid-5400-1/esxi-store1 that I share a= s an
> > iscsi target and that works great. I have set a reservation
> >
> > on that block device equal to its size so that is not sparse and = thus
> > while I could over provision guests on the provided LUN,
> >
> > the storage presented as the LUN capacity will always be availabl= e. At least this is what I want to achieve.
> >
> >
> >
> > What I find strange is that the reservation does not seem to be > > applied to the ZFS pool =E2=80=98raid-5400-1=E2=80=99. Do you rea= lly need
> >
> > to set your maximum reservation at the pool level, and then apply=
> > reservations to all datasets on that volume? And if so
> >
> > I would assume you could never set reservations for datasets tota= ling more than what was reserved for the pool =E2=80=98raid-5400-1=E2=80=99= .
> >
> >
> >
> > I could build out a test environment and figure out constraints b= ut I=E2=80=99d really like to know the =E2=80=9Chow it is supposed to work= =E2=80=9D
> >
> > not the =E2=80=9Chow I find it to work=E2=80=9D.
> >
> >
> >
> > Thanks in advance.
> >
> >
> >
> > FreeBSD 14.0-CURRENT #4 main-n253875-8e72f458c6d:
> >
> >
> >
> > (this is a raidz2 pool =E2=80=93 not my draid pool)
> >
> >
> >
> >
> >
> > root@draid:/usr/src/contrib/bearssl # zfs list -o
> > name,type,used,avail,ratio,compression,reservation,refreservation= ,vols
> > ize raid-5400-1
> >
> > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE
> >
> > raid-5400-1 filesystem 18.0T 2.08T 1.36x on none none - <- no = reservation @pool
> >
> > root@draid:/usr/src/contrib/bearssl #
> >
> >
> >
> > root@draid:/usr/src/contrib/bearssl # zfs list -o
> > name,type,used,avail,ratio,compression,reservation,refreservation= ,vols
> > ize raid-5400-1/esxi-store1
> >
> > NAME TYPE USED AVAIL RATIO COMPRESS RESERV REFRESERV VOLSIZE
> >
> > raid-5400-1/esxi-store1 volume 16.9T 18.5T 1.78x zstd 15.6T 16.9T= 15.6T <- reservation @dataset
> >
> >
> >
> > root@draid:/usr/src/contrib/bearssl # zpool list
> >
> > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTRO= OT
> >
> > ccache 9.50G 9.10G 406M - - 88% 95% 1.00x ONLINE -
> >
> > raid-5400-1 24.6T 1.52T 23.1T - - 2% 6% 1.00x ONLINE - <- Free= does not reflect reservation @pool
> >
> > tank 18.5T 605G 17.9T - - 0% 3% 1.00x ONLINE -
> >
> > zfsroot 103G 33.3G 69.7G - - 31% 32% 1.00x ONLINE -
> >
> > root@draid:/usr/src/contrib/bearssl #
> >
> >
> >
> >
> >
> > CONFIDENTIALITY NOTE: This message is intended only for the use o= f the
> > individual or entity to whom it is addressed and may contain
> > information that is privileged, confidential, and exempt from
> > disclosure under applicable law. If the reader of this message is= not
> > the intended recipient, you are hereby notified that any
> > dissemination, distribution or copying of this communication is > > strictly prohibited. If you have received this transmission in er= ror,
> > please notify us by telephone at (502) 212-4000 or notify us at: = PAI,
> > Dept. 99,
> > 2101 High Wickham Place, Suite 101, Louisville, KY 40245
> >
> >
> >
> >
> >
> > Disclaimer
> >
> > The information contained in this communication from the sender i= s confidential. It is intended solely for use by the recipient and others a= uthorized to receive it. If you are not the recipient, you are hereby notif= ied that any disclosure, copying, distribution or taking action in relation= of the contents of this information is strictly prohibited and may be unla= wful.
> >
> > This email has been scanned for viruses and malware, and may have= been automatically archived by Mimecast, a leader in email security and cy= ber resilience. Mimecast integrates email defenses with brand protection, s= ecurity awareness training, web security, compliance and other essential ca= pabilities. Mimecast helps protect large and small organizations from malic= ious activity, human error and technology failure; and to lead the movement= toward building a more resilient world. To find out more, visit our websit= e.
>
>
>
>
> CONFIDENTIALITY NOTE: This message is intended only for the use
> of the individual or entity to whom it is addressed and may
> contain information that is privileged, confidential, and
> exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying
> of this communication is strictly prohibited. If you have
> received this transmission in error, please notify us by
> telephone at (502) 212-4000 or notify us at PAI, Dept. 99,
> 2101 High Wickham Place, Suite 101, Louisville, KY 40245
>
>
> Disclaimer
>
> The information contained in this communication from the sender is con= fidential. It is intended solely for use by the recipient and others author= ized to receive it. If you are not the recipient, you are hereby notified t= hat any disclosure, copying, distribution or taking action in relation of t= he contents of this information is strictly prohibited and may be unlawful.=
>
> This email has been scanned for viruses and malware, and may have been= automatically archived by Mimecast, a leader in email security and cyber r= esilience. Mimecast integrates email defenses with brand protection, securi= ty awareness training, web security, compliance and other essential capabil= ities. Mimecast helps protect large and small organizations from malicious = activity, human error and technology failure; and to lead the movement towa= rd building a more resilient world. To find out more, visit our website.
--00000000000081810c05dd313637--