From nobody Mon Apr 18 15:57:04 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A6C2511C9BAD; Mon, 18 Apr 2022 15:57:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Khs4x3DW2z4mdf; Mon, 18 Apr 2022 15:57:05 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1650297425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hRBhHESKC09QvrkYfIw2t2eZoyxZa0dBaTL5mNSi2Jw=; b=bS9NdH7Nvk3a3c/3eXrqnOpJTy/1MGuXA5rRhrCQxGHNsj0Fx3aoIg8W9tB4zkmZY4sDGR 29G1naY/DaEoQu6VRlhe1aA9WC+kqWDeyMpYDheeHYlDecscVgYthlMnjjMZmnyxeSM+PH qhvHMrl2gQE1/rYNNJVo432MB0vUrwn68zuKZdptQmM0cu/60yV6F1l74SDs/QGWfanf6s 0Dcj1eOeJMFZ7TDTkBYXXACLePJxf6UAlLuG2vLFc7Kc/jmk6+zBSpANcSIKB+oGkWyhKh dCqfMS+t/cZnwgb1rhvIgncPRMasijhheMAxfmq3l3KIeYDNjyHrfV6LP4J5DA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0A7D615746; Mon, 18 Apr 2022 15:57:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 23IFv4Sh031509; Mon, 18 Apr 2022 15:57:04 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 23IFv4fl031508; Mon, 18 Apr 2022 15:57:04 GMT (envelope-from git) Date: Mon, 18 Apr 2022 15:57:04 GMT Message-Id: <202204181557.23IFv4fl031508@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mitchell Horne Subject: git: 0a5c04a8926e - main - savecore: add an option to save a live minidump List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mhorne X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0a5c04a8926e0418c556dfb65efbb24b3bf41dd0 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1650297425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hRBhHESKC09QvrkYfIw2t2eZoyxZa0dBaTL5mNSi2Jw=; b=FQ8l1L5W9SkcgsCpoQqsj/iRrIkCHgaqYkyW87Tadp6CwRxLUVotZGfOTzDiodmHQO24Xu gPYPIrZqg6dr6iu2OoEywcFVyDxbSSLAA5Fz0TVmqgzLiayYq9SL9aOWm+G1pR9jX+cYzG 0zs0JLn7mENWHieXNN229Qu3j4epzc39VhkDnuK9KLiXuN6cIMpBsS1YrLDYZlCiIvohBs FJVbzYFTUbuI6t2Jf0tlM7boXoOXyhVQ2n4D/NTry3R9WbpHuVOsRDf1zQlIuMr1kAEDqo 4jLJjllvhf93HtNnQauciqbyVkksGuXj30d4da0FsHMYcMyzPUoAyf7vPRo4PA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1650297425; a=rsa-sha256; cv=none; b=ZGAJg20In6zHUO4HfeqIUNdOQLUmao0bbJ2fWgZ8X6NeV/COqcsJz3wrJagYTdVLWuzdam apdAiwaSgIMf2eTWGhBNrB8LSkdby4m8mP18bFc+a4XqOdFAS76Hcx4a21tBFLJcJSBCXm Ijqis6GXTeTFYvG+1OLbEGhpr0MG/CAwpLgR1z+9rvE1Pq/QYXVUrDvhIN+VcUEEboav/r FDJGWuMVDL6a0ADLyYmlh1lKxUldJOa4qKaettwqc6Vspea1/HYkApCxRbMMUBiLiT+rPA awx7zhwuSocFd8oPmRD32hwTJwnLkTWo+U6wrkgGen1niq5b7TWlElVslOtn8w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by mhorne: URL: https://cgit.FreeBSD.org/src/commit/?id=0a5c04a8926e0418c556dfb65efbb24b3bf41dd0 commit 0a5c04a8926e0418c556dfb65efbb24b3bf41dd0 Author: Mitchell Horne AuthorDate: 2022-04-18 15:22:07 +0000 Commit: Mitchell Horne CommitDate: 2022-04-18 15:56:16 +0000 savecore: add an option to save a live minidump The new '-L' flag will cause savecore to invoke the new mem(4) kernel dump ioctl, taking a dump of the running system and writing the result to a temporary file. Validation of the dump header is performed, similar to regular crash dumps, and the final result is written to livecore.X[.zst|.gz]. Also added is the '-Z' flag, which instructs the kernel to compress the livedump compressed with zstd, akin to the existing -z flag. This option has no effect in normal savecore(8) operation, but in theory could be extended to perform such compression while reading the dump from the dump device. Encryption is unsupported for live dumps. For example: 'savecore -Lz /var/crash' would create: /var/crash/livecore.0.gz Reviewed by: markj MFC after: 2 weeks Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D34347 --- sbin/savecore/savecore.8 | 26 +++++- sbin/savecore/savecore.c | 229 ++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 250 insertions(+), 5 deletions(-) diff --git a/sbin/savecore/savecore.8 b/sbin/savecore/savecore.8 index 512da5cc3dae..b1e0c38c6b44 100644 --- a/sbin/savecore/savecore.8 +++ b/sbin/savecore/savecore.8 @@ -28,7 +28,7 @@ .\" From: @(#)savecore.8 8.1 (Berkeley) 6/5/93 .\" $FreeBSD$ .\" -.Dd November 17, 2020 +.Dd April 4, 2022 .Dt SAVECORE 8 .Os .Sh NAME @@ -44,6 +44,11 @@ .Op Fl v .Op Ar device ... .Nm +.Fl L +.Op Fl fvZz +.Op Fl m Ar maxdumps +.Op Ar directory +.Nm .Op Fl -libxo .Op Fl fkuvz .Op Fl m Ar maxdumps @@ -86,6 +91,11 @@ Force a dump to be taken even if either the dump was cleared or if the dump header information is inconsistent. .It Fl k Do not clear the dump after saving it. +.It Fl L +Instruct +.Nm +to generate and save a kernel dump of the running system, rather than +copying one from a dump device. .It Fl m Ar maxdumps Maximum number of dumps to store. Once the number of stored dumps is equal to @@ -97,6 +107,14 @@ Uncompress the dump in case it was compressed by the kernel. .It Fl v Print out some additional debugging information. Specify twice for more information. +.It Fl Z +Compress the dump (see +.Xr zstd 1 ) . +This option is only supported in conjunction with the +.Fl L +option. +Regular dumps can be configured for compression with zstd using +.Xr dumpon 8 . .It Fl z Compress the dump (see .Xr gzip 1 ) . @@ -104,6 +122,10 @@ The dump may already be compressed if the kernel was configured to do so by .Xr dumpon 8 . In this case, the option has no effect. +.Pp +If used in conjunction with the +.Fl L +option, the requested live dump will be compressed with gzip. .El .Pp The @@ -171,9 +193,11 @@ is meant to be called near the end of the initialization file .Xr rc 8 ) . .Sh SEE ALSO .Xr gzip 1 , +.Xr zstd 1 , .Xr getbootfile 3 , .Xr libxo 3 , .Xr xo_parse_args 3 , +.Xr mem 4 , .Xr textdump 4 , .Xr tar 5 , .Xr crashinfo 8 , diff --git a/sbin/savecore/savecore.c b/sbin/savecore/savecore.c index 8b3a860f4086..9f1f04422a55 100644 --- a/sbin/savecore/savecore.c +++ b/sbin/savecore/savecore.c @@ -68,6 +68,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -106,9 +107,11 @@ __FBSDID("$FreeBSD$"); static cap_channel_t *capsyslog; static fileargs_t *capfa; static bool checkfor, compress, uncompress, clear, force, keep; /* flags */ +static bool livecore; /* flags cont. */ static int verbose; static int nfound, nsaved, nerr; /* statistics */ static int maxdumps; +static uint8_t comp_desired; extern FILE *zdopen(int, const char *); @@ -393,6 +396,12 @@ saved_dump_remove(int savedirfd, int bounds) (void)unlinkat(savedirfd, path, 0); (void)snprintf(path, sizeof(path), "textdump.tar.%d.gz", bounds); (void)unlinkat(savedirfd, path, 0); + (void)snprintf(path, sizeof(path), "livecore.%d", bounds); + (void)unlinkat(savedirfd, path, 0); + (void)snprintf(path, sizeof(path), "livecore.%d.gz", bounds); + (void)unlinkat(savedirfd, path, 0); + (void)snprintf(path, sizeof(path), "livecore.%d.zst", bounds); + (void)unlinkat(savedirfd, path, 0); } static void @@ -408,6 +417,9 @@ symlinks_remove(int savedirfd) (void)unlinkat(savedirfd, "vmcore_encrypted.last.gz", 0); (void)unlinkat(savedirfd, "textdump.tar.last", 0); (void)unlinkat(savedirfd, "textdump.tar.last.gz", 0); + (void)unlinkat(savedirfd, "livecore.last", 0); + (void)unlinkat(savedirfd, "livecore.last.gz", 0); + (void)unlinkat(savedirfd, "livecore.last.zst", 0); } /* @@ -731,6 +743,183 @@ DoTextdumpFile(int fd, off_t dumpsize, off_t lasthd, char *buf, return (0); } +static void +DoLiveFile(const char *savedir, int savedirfd, const char *device) +{ + char infoname[32], corename[32], linkname[32], tmpname[32]; + struct mem_livedump_arg marg; + struct kerneldumpheader kdhl; + xo_handle_t *xostdout; + off_t dumplength; + uint32_t version; + int fddev, fdcore; + int bounds; + int error, status; + + bounds = getbounds(savedirfd); + status = STATUS_UNKNOWN; + + xostdout = xo_create_to_file(stdout, XO_STYLE_TEXT, 0); + if (xostdout == NULL) { + logmsg(LOG_ERR, "xo_create_to_file() failed: %m"); + return; + } + + /* + * Create a temporary file. We will invoke the live dump and its + * contents will be written to this fd. After validating and removing + * the kernel dump header from the tail-end of this file, it will be + * renamed to its definitive filename (e.g. livecore.2.gz). + * + * If any errors are encountered before the rename, the temporary file + * is unlinked. + */ + strcpy(tmpname, "livecore.tmp.XXXXXX"); + fdcore = mkostempsat(savedirfd, tmpname, 0, 0); + if (fdcore < 0) { + logmsg(LOG_ERR, "error opening temp file: %m"); + return; + } + + fddev = fileargs_open(capfa, device); + if (fddev < 0) { + logmsg(LOG_ERR, "%s: %m", device); + goto unlinkexit; + } + + bzero(&marg, sizeof(marg)); + marg.fd = fdcore; + marg.compression = comp_desired; + if (ioctl(fddev, MEM_KERNELDUMP, &marg) == -1) { + logmsg(LOG_ERR, + "failed to invoke live-dump on system: %m"); + close(fddev); + goto unlinkexit; + } + + /* Close /dev/mem fd, we are finished with it. */ + close(fddev); + + /* Seek to the end of the file, minus the size of the header. */ + if (lseek(fdcore, -(off_t)sizeof(kdhl), SEEK_END) == -1) { + logmsg(LOG_ERR, "failed to lseek: %m"); + goto unlinkexit; + } + + if (read(fdcore, &kdhl, sizeof(kdhl)) != sizeof(kdhl)) { + logmsg(LOG_ERR, "failed to read kernel dump header: %m"); + goto unlinkexit; + } + /* Reset cursor */ + (void)lseek(fdcore, 0, SEEK_SET); + + /* Validate the dump header. */ + version = dtoh32(kdhl.version); + if (compare_magic(&kdhl, KERNELDUMPMAGIC)) { + if (version != KERNELDUMPVERSION) { + logmsg(LOG_ERR, + "unknown version (%d) in dump header on %s", + version, device); + goto unlinkexit; + } else if (kdhl.compression != comp_desired) { + /* This should be impossible. */ + logmsg(LOG_ERR, + "dump compression (%u) doesn't match request (%u)", + kdhl.compression, comp_desired); + if (!force) + goto unlinkexit; + } + } else { + logmsg(LOG_ERR, "magic mismatch on live dump header"); + goto unlinkexit; + } + if (kerneldump_parity(&kdhl)) { + logmsg(LOG_ERR, + "parity error on last dump header on %s", device); + nerr++; + status = STATUS_BAD; + if (!force) + goto unlinkexit; + } else { + status = STATUS_GOOD; + } + + nfound++; + dumplength = dtoh64(kdhl.dumplength); + if (dtoh32(kdhl.dumpkeysize) != 0) { + logmsg(LOG_ERR, + "dump header unexpectedly reported keysize > 0"); + goto unlinkexit; + } + + /* Remove the vestigial kernel dump header. */ + error = ftruncate(fdcore, dumplength); + if (error != 0) { + logmsg(LOG_ERR, "failed to truncate the core file: %m"); + goto unlinkexit; + } + + if (verbose >= 2) { + printf("\nDump header:\n"); + printheader(xostdout, &kdhl, device, bounds, -1); + printf("\n"); + } + logmsg(LOG_ALERT, "livedump"); + + writebounds(savedirfd, bounds + 1); + saved_dump_remove(savedirfd, bounds); + + snprintf(corename, sizeof(corename), "livecore.%d", bounds); + if (compress) + strcat(corename, kdhl.compression == KERNELDUMP_COMP_ZSTD ? + ".zst" : ".gz"); + + if (verbose) + printf("renaming %s to %s\n", tmpname, corename); + if (renameat(savedirfd, tmpname, savedirfd, corename) != 0) { + logmsg(LOG_ERR, "renameat failed: %m"); + goto unlinkexit; + } + + snprintf(infoname, sizeof(infoname), "info.%d", bounds); + if (write_header_info(xostdout, &kdhl, savedirfd, infoname, device, + bounds, status) != 0) { + nerr++; + return; + } + + logmsg(LOG_NOTICE, "writing %score to %s/%s", + compress ? "compressed " : "", savedir, corename); + + if (verbose) + printf("\n"); + + symlinks_remove(savedirfd); + if (symlinkat(infoname, savedirfd, "info.last") == -1) { + logmsg(LOG_WARNING, "unable to create symlink %s/%s: %m", + savedir, "info.last"); + } + + snprintf(linkname, sizeof(linkname), "livecore.last"); + if (compress) + strcat(linkname, kdhl.compression == KERNELDUMP_COMP_ZSTD ? + ".zst" : ".gz"); + if (symlinkat(corename, savedirfd, linkname) == -1) { + logmsg(LOG_WARNING, "unable to create symlink %s/%s: %m", + savedir, linkname); + } + + nsaved++; + if (verbose) + printf("dump saved\n"); + + close(fdcore); + return; +unlinkexit: + funlinkat(savedirfd, tmpname, fdcore, 0); + close(fdcore); +} + static void DoFile(const char *savedir, int savedirfd, const char *device) { @@ -748,6 +937,12 @@ DoFile(const char *savedir, int savedirfd, const char *device) uint32_t dumpkeysize; bool iscompressed, isencrypted, istextdump, ret; + /* Live kernel dumps are handled separately. */ + if (livecore) { + DoLiveFile(savedir, savedirfd, device); + return; + } + bounds = getbounds(savedirfd); dumpkey = NULL; mediasize = 0; @@ -1220,9 +1415,10 @@ init_caps(int argc, char **argv) static void usage(void) { - xo_error("%s\n%s\n%s\n", + xo_error("%s\n%s\n%s\n%s\n", "usage: savecore -c [-v] [device ...]", " savecore -C [-v] [device ...]", + " savecore -L [-fvZz] [-m maxdumps] [directory]", " savecore [-fkuvz] [-m maxdumps] [directory [device ...]]"); exit(1); } @@ -1235,10 +1431,11 @@ main(int argc, char **argv) char **devs; int i, ch, error, savedirfd; - checkfor = compress = clear = force = keep = false; + checkfor = compress = clear = force = keep = livecore = false; verbose = 0; nfound = nsaved = nerr = 0; savedir = "."; + comp_desired = KERNELDUMP_COMP_NONE; openlog("savecore", LOG_PERROR, LOG_DAEMON); signal(SIGINFO, infohandler); @@ -1247,7 +1444,7 @@ main(int argc, char **argv) if (argc < 0) exit(1); - while ((ch = getopt(argc, argv, "Ccfkm:uvz")) != -1) + while ((ch = getopt(argc, argv, "CcfkLm:uvZz")) != -1) switch(ch) { case 'C': checkfor = true; @@ -1261,6 +1458,9 @@ main(int argc, char **argv) case 'k': keep = true; break; + case 'L': + livecore = true; + break; case 'm': maxdumps = atoi(optarg); if (maxdumps <= 0) { @@ -1274,8 +1474,16 @@ main(int argc, char **argv) case 'v': verbose++; break; + case 'Z': + /* No on-the-fly compression with zstd at the moment. */ + if (!livecore) + usage(); + compress = true; + comp_desired = KERNELDUMP_COMP_ZSTD; + break; case 'z': compress = true; + comp_desired = KERNELDUMP_COMP_GZIP; break; case '?': default: @@ -1289,6 +1497,8 @@ main(int argc, char **argv) usage(); if (compress && uncompress) usage(); + if (livecore && (checkfor || clear || uncompress || keep)) + usage(); argc -= optind; argv += optind; if (argc >= 1 && !checkfor && !clear) { @@ -1301,7 +1511,15 @@ main(int argc, char **argv) argc--; argv++; } - if (argc == 0) + if (livecore) { + if (argc > 0) + usage(); + + /* Always need /dev/mem to invoke the dump */ + devs = malloc(sizeof(char *)); + devs[0] = strdup("/dev/mem"); + argc++; + } else if (argc == 0) devs = enum_dumpdevs(&argc); else devs = devify(argc, argv); @@ -1314,6 +1532,9 @@ main(int argc, char **argv) (void)cap_rights_init(&rights, CAP_CREATE, CAP_FCNTL, CAP_FSTATAT, CAP_FSTATFS, CAP_PREAD, CAP_SYMLINKAT, CAP_FTRUNCATE, CAP_UNLINKAT, CAP_WRITE); + if (livecore) + cap_rights_set(&rights, CAP_RENAMEAT_SOURCE, + CAP_RENAMEAT_TARGET); if (caph_rights_limit(savedirfd, &rights) < 0) { logmsg(LOG_ERR, "cap_rights_limit(): %m"); exit(1);