From nobody Sat Jul 03 20:55:46 2021 X-Original-To: freebsd-toolchain@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F2CFE11E6DFE for ; Sat, 3 Jul 2021 20:55:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GHPP64Lv5z4jyq for ; Sat, 3 Jul 2021 20:55:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625345752; bh=mAyJgl+Nag4RaYB++I4f9H3ad6kkmQ0a1GqQk+em7PU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=qruGZIg21cZlIu3IEYJ7DVfAXxEF/CmGWaA+NQeDs6Km9Db4G+SLK97IpW7vf4OQW+7Dwyki3Rhah+Sy9AauPVzCFiO+xFUi4Xdi7+V7SioXFpQyv703LTjd9KGttOOPpdViL2FOpExF+nhNfoyZhTt5gVhLOEUkxMGl4USI594Pn6lB/LdK24LoM7Fz4MisPJB1cTQvTSlcePLyi1ikfvO/4cZsKRZt3KpO1GUIQAe6X3ptyRvicnly+ySzJplYKaCW1klclDkyp3rdxNnt9kFeZD3zLbSCiyF03db8cwB24ozE3eryLurgyuVXIEiw3JZ8wDzJfut0ZusPnK8ryA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625345752; bh=jklxwWaQSb0pv2Sh3TX/rof8LZZRLrq6IhHPaj/PnOT=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=JQaA1ITYIMUeaqTujSj4NrknjH2YnCajjNDzQq+gHbawqFnSQrp7PT5pJwa4XQ3lFQMdsL6rR8tcu4flBGqZcSApZN3u05LRN3tLWfv7VgWRNCpDV8XjiKZL6BLy2ywJqdqdHlX6x+mV7lqoSKlV/GwQAu1aDfucJkKYcO339bKx+uDs4ZrixfPcILMGH+BbGUJRQewT0r+HzFZEOlEuDKqhr1IItKxboM1CL34Q8WMUn0IcOB6wS4BD+pRUlmdKkCKr/ZFocsEwv1ZQRhaRG/x51IDG5z/AyKLRaVrDx2Uf4L97rRdtDvIG8V3+cTnCCgj/mfcj16VIfbio2/Mt1Q== X-YMail-OSG: ptWcCrQVM1mE.MZjGipTvNNUFOmdEQrEUT3En2zOEWGyo.0qcqEjNGqsKTo4e_Z .cZ5un.BMyrzZu00OHgbr6cMDs0v57WW5Ss3OVhvxgyE2bWbV1o3DYQ6YSn58mu8at4ym3KgnaRb bg_I96FbhQRwy8I9yWmiwCInJQ35bVy1hdJVh1lJjVFl2wVYXZC1p7aQzGZg4r44l1TlHag1oQsX 5gKtc4cb7_WGEgTw1NtFvwYv44OD0aBGw0tHmN7WM0SR04_LmTOzw6xldpIFdBriFaGdtLASREc6 k5s07oOAyqVg_in5CfU9e.cr6Lovfa2Npbb4A4ROFtf2IOiUN8pkPre3d49yp0zTxvd1iob2bsnh m.v9NN08H9JNTcfDTW_R40hSRhTeqwxRoonIHIxHk9cK6cRzQH_pIrvA2eovhzjNLmScDX1CVb6r Dn47A70X2Otpq.8CcAVTUDlQqlL1fNcr261wZkBF4ilfp2dofQOmlulXuDaUl5m6OfhErQCTeH4b PuckVuHNVdfV6LNxzBCtpLpyELzR56ba6f4K_dR0ZA1XVqy2Cv_mEp4FMQvHkL0gVp.krBJTnCPz 3oTZQu5pibfJWZ6pZUrJg5g1AcXcSgYzTGE4U5dEJI1pt7hU_03IHs28u.PZvidJh4Wdtdsn_.TU I6Chl2UIGC6RE8xC0QqZgWVpUsnMPksn4NfuBt_uxowPXHbN3wMP8LsM3EJUIiEcdQZTEOdZ4t2J yLiRjltEIhuDiWge1fPHqPNT_BlvJVTcZGnpu_eTZSEjKAdk_SA6yTqu1uN53VeGq_V1JEm9Fk_Q iisiA7PRjrF3Z5oGrMx1gT3L7w3vRkrKxfmBH2ilFM9qN0KFKoTXHPZfhwbVY1AHnuLfdHU__SwJ CME5or9dyolpbIwfMViZMW3HV5VbXrMscOKyrCg2xGSRGNgMvip0H3DzcqOcvXWALBnw7q0OgolM HopNuoAG76ih_FypqciFsylbk3DqScIUwkJF7Z6w1YL11wp2E6Ie2hVcjhqpE8YqMDuxdqDdN5p0 t6veEuBA8H6gMX.SStl1tC81P.HFp4mAbVWIJiFF5G8e5tzIlwglxrfh7q0CDpv3ojhJstevZoLV cdRedShmcJFywkRZDXhVzE9cqVgtgFrSRQ9buttB.678x_1hJdWRaGKlUq0UDrIN4cr4yUBY7b52 ySCOv64jjRH2.VDh4pfsPJWhzxY5rSrtD9lgKcNiNPXegdhIKrKezcRS_f1DdCa_XAj_b1UPOG7R 8cVlxWelpmIJTjIdmoB0DNBMbyI1pe3ov81WZsJ2Gb2z50dOWHzQx49P2Os0xyqRgs6dYlPXZ04H 0oY38QCAzySrhjw2tIGUyQJq7oylUzh6GjbYhsWpOtRYzr9H6yYITEcj348z5SsesjBPcYpwMQSh WzrxTXHG87u67G7gi5R_vmCBN_FI12jmtEAwU.AlWgIU4Pe9bOeN_rxrupklU0e_tPY9tx3HbLsa CgQ1.5HUXzDRi.ca1.KcRsHQ4torQUD9YbR.6ZrqIkhqqXYOENff4FiVSlZ.lCzMI3WaJpQSG_qp _hY01oYmnq5N33.v4qT91xgv6f.Jbo4CyMYZagWAoEQZL5heibG06kqZAEQpspn2QVVWSVCfKVmG CSsk8.lzp9uOolr5e4pRX4KWpM.aDO.3NRFmY8mxIckHbvRiB61FK6AX8bAg2eWzSdgtJtTbpNpc o2L8xOF4Fy5uaifJWPMC5Qua1g1B5431uYI5SfmZi58ls3h4v1H1zAN9xOoHbiWxKJfqSAJt6xMg 8n9puFpV1ok2925GqOkN3avheG751CVy53V9zpFhtFJBMMAy05pMQ7W3Flkya7DIuNh9LSqw3ykw u47SupdqlJ6EfjEe3ueEtnIFO1Hy9MayFnHJdrjQWa2CuTeurBX5C2x89BSTtwLtt0pRWtbgv8bl U521TlrUaF55seU4P4f1UlQ0rpAr_hCqiODAYTSW32p9ur1BhxBaaWZkWJiD1bZu4Bku75Wtsard 6OXPFn2u3Y0iSElqsi6ONGgp2GAySNjQ2W_CQRNdv4QqNRLvoCgUBysLW31aNCmr1hT3wtGeetxu A.BUjAH3NpC9pP4sdkgSeFD53NfBGARIUpNnnlgRKYJQ_p3xO.WVE6kX2NowtM66PmYNiHLKevga N0RZvtQWG2VQBAy_E38DoxYwPlqlQ0tHZ.xRQzHILkUqcaZZFmXXF9pXxvHtOlJwSEWeK6Ux4QNB GG4rCjn8g6Mp66yTPI1FIgWQ4r6a8buIqt6bp6D_7R4YTeGp_HkpIxR5hwCwrsTMR9wqhxqfD8TH AIkGDAEQgIuELkinZlolEcLK3Zju4M9LMHdqME.dEiUsG2iEloGErVY2EAcNdz1kfF5HjVnceVOO gbFFGil9BU3lTZ5JiBGdR3tQ3ruh0L61bMJQtcatS.QEQQc92J9kYRInMiE9PwQdAht7emwPSeT5 KH4OziNijMWytJ7tnV8O25HoAjxNubN5ujB0BZWwYv3DD6Hm3BflfJbHtieBCnQ_FCeUvl8jqhiA FYZT1oexTh7SUdj_4vZu5peO8WznkB2dn1krk4VsX4kUwNRiCkz_4D03jvBByX1y27FUOqbHW.ed I_O4K1CCHSc_equEieh8hhI7bQd9_UflkMKSWVMDcfeIzpyHLoNnhw.hKY2Lj5FieO6KyXz4WEO8 5Jzn_jGMAxHYxmTICQW7rdtItaAYDd5IU_2ry.4JuivO6D4IgdEH9xJpTDhzFgRBDEA76N3w- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Sat, 3 Jul 2021 20:55:52 +0000 Received: by kubenode544.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID f7224f8d049378c663635b529107a291; Sat, 03 Jul 2021 20:55:47 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Maintenance List-Archive: https://lists.freebsd.org/archives/freebsd-toolchain List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: llvm10 build failure on Rpi3 In-Reply-To: <380184FB-6BA1-4C2D-9C6B-E249C2CF1317@yahoo.com> Date: Sat, 3 Jul 2021 13:55:46 -0700 Cc: FreeBSD ports , freebsd-arm , FreeBSD Toolchain Content-Transfer-Encoding: quoted-printable Message-Id: <9CC00581-E8B4-45F0-A614-60A70E37B1B2@yahoo.com> References: <5A26965D-2DFD-4A46-A171-A382A61E3CFB@yahoo.com> <20210626035234.GA18893@www.zefox.net> <43513842-6FC0-4A89-8F0C-9EB2B328A5ED@yahoo.com> <9CFE71E2-23C3-4072-A8AD-74EDB339A146@yahoo.com> <60EEFD09-97DE-4B4F-BAFD-61B96EF60E27@yahoo.com> <77A35ACF-275F-44C8-AEEE-4EFE5B5CBEA4@yahoo.com> <20210703182546.GA17871@www.zefox.net> <380184FB-6BA1-4C2D-9C6B-E249C2CF1317@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GHPP64Lv5z4jyq X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=qruGZIg2; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.68.206:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.206:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.206:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-toolchain] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-3, at 13:15, Mark Millard wrote: > On 2021-Jul-3, at 11:25, bob prohaska wrote: >=20 >>>>> On 2021-Jul-2, at 19:23, Mark Millard = wrote: >>>>=20 >>>>>> Side note: >>>>>>=20 >>>>>> It llooks like = http://www.zefox.org/~bob/swaplogs/poudrierellvm10.log >>>>>> shows that you tried with: >>>>>>=20 >>>>>> Device 1K-blocks Used Avail Capacity >>>>>> /dev/da0s2b 1048576 25784 1022792 2% >>>>>> /dev/mmcsd0s2b 1048576 25124 1023452 2% >>>>>> Total 2097152 50908 2046244 2% >>>>>>=20 >> [hope the quotes are right!] >>=20 >> That's correct. The sequence of experiments ran something like this: >>=20 >> The Pi3 was configured with a a pair of ~3 GB swap partitions, one on >> microSD, the other on the 1 TB mechanical hard disk. Make was not = limited >> in the number of jobs it could parallel. OOMA was restrained by = putting >> vm.pageout_oom_seq=3D"4096" >> vm.pfault_oom_attempts=3D"20" >> in /boot/loader.conf The usual "excessive swap" warnings were = presented >> during boot and ignored by me.=20 >>=20 >> Worlds and kernels built wtihout trouble, so I tried building = www/chromium >> using poudriere. It stopped in /devel/llvm10 with the "expected = expression" >> error and continued to stop there despite updating /usr/ports several = times.=20 >> At no time were there any hints of swap problems. Resorting to a = GENERIC >> self-hosted kernel made no difference. /usr/src was not tampered = with.=20 >=20 > So you still have not tried an artifacts or snapshot kernel+world? >=20 >> Eventually I resorted to running make in devel/llvm10, to my surprise = it >> ran to completion. >=20 > Interesting. >=20 > Was this -j4? -j1? -j2? Any other interesting characteristics > for how it was run? >=20 > It would be interesting to see if building in a chroot > in that make style also worked (or a non-poudriere jail). >=20 >> It also ran make package successfully. Again I tried to >> build just devel/llvm10 using poudriere, again getting "expected = expression".=20 >>=20 >> At that point I resized the swap partitions to 1 GB each and tried = poudriere >> on devel/llvm10. That got rid of the excessive swap warnings, but = didn't help. >> Finally I placed=20 >> MAKE_JOBS_NUMBER=3D2=20 >> in /usr/local/etc/poudriere.d/make.conf and tried again. That still = failed, >> still with "expected expression".=20 >=20 > I'll note that the running build build shows Load Averages > of under 3. So the MAKE_JOBS_NUMBER=3D2 seems to be working. >=20 >> Since devel/llvm10 had created a package successfully, I tried = slipping a copy >> into poudriere's package directory, hoping it would find and use the = package >> to make further progress. Unfortunately, poudriere seems to remember = the failure >> and won't use the proffered package.=20 >=20 > After things build correctly, things tend to look something like > (using an example): >=20 > 2# ls -FTla /usr/local/poudriere/data/packages/main-CA53-default/ > total 12 > drwxr-xr-x 3 root wheel 512 Jul 3 07:19:32 2021 ./ > drwxr-xr-x 4 root wheel 512 Jul 1 19:25:44 2021 ../ > lrwxr-xr-x 1 root wheel 18 Jun 28 04:32:43 2021 .buildname@ -> = .latest/.buildname > lrwxr-xr-x 1 root wheel 20 Jun 28 04:32:43 2021 .jailversion@ -> = .latest/.jailversion > lrwxr-xr-x 1 root wheel 16 Jul 3 07:19:32 2021 .latest@ -> = .real_1625321972 > drwxr-xr-x 4 root wheel 512 Jul 3 07:19:32 2021 .real_1625321972/ > lrwxr-xr-x 1 root wheel 11 Jun 28 04:32:43 2021 All@ -> = .latest/All > lrwxr-xr-x 1 root wheel 14 Jun 28 04:32:43 2021 Latest@ -> = .latest/Latest > lrwxr-xr-x 1 root wheel 17 Jun 28 04:32:43 2021 meta.conf@ -> = .latest/meta.conf > lrwxr-xr-x 1 root wheel 16 Jun 28 04:32:43 2021 meta.txz@ -> = .latest/meta.txz > lrwxr-xr-x 1 root wheel 23 Jun 28 04:32:43 2021 packagesite.txz@ = -> .latest/packagesite.txz >=20 > But, if a bulk is in process or has finished after some package > had a build failure, there is also a: >=20 > .building/ >=20 > in there. That is what the message: >=20 > Using packages from previously failed build: ${PACKAGES}/.building >=20 > is about when starting poudriere bulk again. This is how > poudriere avoids rebuilding what successfully built --but > without adjusting the prior successful bulk build (if any). >=20 > So poudriere would have expected the file for devel/llvm10 's > build to be in that .building/ directory instead of down under > the .real_*/ directory. >=20 > (I've not checked if there is other record keeping in .building/ > about the materials as well.) >=20 > Going in a different direction, one way to force a build to > start over after a failure is to: rm -fr PATH/.building > before starting a new bulk build. This might be appropriate > if one suspects a problem of a kind that did not stop a > build but produced something for a build that fails to operate > correctly. >=20 >> It's still running, on lang/spidermoneky78. =20 >=20 > So lang/rust finished. That is interesting because it includes an > llvm build internally. >=20 > Also: had you updated to pick up the workaround for the rust > build failures on aarch64? I doubt it because they were > commited on 2021-July-02. See, >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256864#c18 >=20 > So that you did not get the process crash/core-dump during > lang/rust 's build is interesting. >=20 >> There were no reboots between experiments. >>=20 >> My first suspicion is that I've somehow screwed up the poudriere = setup, perhaps >> by a fumbled execution of poudriere jail -u, which I mistakenly = thought was >> needed after updating /usr/ports. >=20 > Again, poudriere does not control memory initialization in > the processes in the builders. >=20 >> The fact that the stoppage reported looks like >> a syntax error specific to devel/llmv10 which is unaffected by swap = pressure >> makes it seem unrelated to kernel or swap constraints.=20 >=20 > The files with the syntax errors are ones generated by llvm-tblgen > during the build and it is the output of llvm-tblgen that is corrupt, > showing evidence of having used memory not initialized like it should > have been. >=20 >> AIUI, the hardware of the Pi4 is considerably different from the Pi3 = in terms >> of memory management, noted from an interview with Eben Upton on = YouTube. >=20 > Why would Eben Upton be talking about FreeBSD's memory management? >=20 > I suspect that the talk is not about what you think it is about, > but some narrower aspects than the overall memory managment. >=20 >> He=20 >> didn't go into any detail. Whether that's relevant is unclear to me, = but it=20 >> does suggest the Pi4, even with restricted memory, won't behave like = a Pi3. >=20 > Various reserved memory areas and such will vary but FreeBSD > uses the same general memory management code, not completely > separate code. >=20 >> Is there any sort of sanity test for the poudriere system? If I = delete and >> re-create the existing jail can the existing package library be = preserved >> and re-used? If not, that's OK, I'd just like to know beforehand. >>=20 >=20 > # poudriere jail -jNAME -d > # poudriere jail -c -jNAME -m null -M /WORLDPATH -S /SRCPATH -v = 14.0-CURRENT >=20 > should work fine. But really all that you are > doing is (using an example from my environment) > is deleting and rewriting a few very small files > in a directory with the jail's name: >=20 > # ls -FTla /usr/local/etc/poudriere.d/jails/main-CA53/ > total 36 > drwxr-xr-x 2 root wheel 512 Jul 2 21:03:23 2021 ./ > drwxr-xr-x 3 root wheel 512 Jul 2 21:03:23 2021 ../ > -rw-r--r-- 1 root wheel 14 Jul 2 21:03:23 2021 arch > -rw-r--r-- 1 root wheel 5 Jul 2 21:03:23 2021 method > -rw-r--r-- 1 root wheel 33 Jul 2 21:03:23 2021 mnt > -rw-r--r-- 1 root wheel 2 Jul 2 21:03:23 2021 pkgbase > -rw-r--r-- 1 root wheel 14 Jul 2 21:03:23 2021 srcpath > -rw-r--r-- 1 root wheel 11 Jul 2 21:03:23 2021 timestamp > -rw-r--r-- 1 root wheel 13 Jul 2 21:03:23 2021 version >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/arch > arm64.aarch64 >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/method > null >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/mnt > /usr/obj/DESTDIRs/main-CA53-poud >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/pkgbase=20 > 0 >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/srcpath=20 > /usr/main-src >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/timestamp=20 > 1625285003 >=20 > # cat /usr/local/etc/poudriere.d/jails/main-CA53/version=20 > 14.0-CURRENT >=20 > The deletion/replacement of timestamp may have rebuild > consequences from appearing to have changed (or just > being missing). >=20 > Nothing about any of those is going to change how memory > initialization is working in llvm-tblgen's operation > for generating any *GenGlobalISel.inc files, other than > if the timestamp forces some sort of rebuild from scratch > of some build dependencies first. I'll note that the poudriere ports binding is similar for deletion and creation: just a few small files in a directory for the name used (default here): # ls -FTla /usr/local/etc/poudriere.d/ports/default/ total 24 drwxr-xr-x 2 root wheel 512 Apr 18 02:05:47 2021 ./ drwxr-xr-x 3 root wheel 512 Apr 18 02:05:47 2021 ../ -rw-r--r-- 1 root wheel 2 Apr 18 02:05:47 2021 created_fs -rw-r--r-- 1 root wheel 5 Apr 18 02:05:47 2021 method -rw-r--r-- 1 root wheel 11 Apr 18 02:05:47 2021 mnt -rw-r--r-- 1 root wheel 11 Apr 18 02:05:47 2021 timestamp # cat /usr/local/etc/poudriere.d/ports/default/created_fs=20 0 # cat /usr/local/etc/poudriere.d/ports/default/method=20 null # cat /usr/local/etc/poudriere.d/ports/default/mnt /usr/ports # cat /usr/local/etc/poudriere.d/ports/default/timestamp=20 1618736747 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)