From nobody Mon Jun 27 19:35:36 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 64E3C86E488; Mon, 27 Jun 2022 19:35:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LWycn5Rjgz3lky; Mon, 27 Jun 2022 19:35:37 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656358538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Y49yigPYmo/IjWAkz+krQ6XZggKzvI406xq4rJersmk=; b=ec6ldtvd2SlbzPlghijzpal2HvzPLRqgXgTvblhGZIUb8V04qBGdbaglALLog1j0GfF5rY Ml2UO0PUybnF1r71siiOtYXTCcjCJIgiuAKE4a8bYshvolVam6Hfk/Fvk4xijKsPpVVVDE xoZNet8t1im0JpNSZA6/gKdsWLbOmGRWrHcd6hwjFPCvaYxwzMu8qV8fcIejMoUo/kdnTL 2HMy51STWLIHTCuWY4upM/vbVqyimGHMwL4B6Hl/ufQcNJl+WR51zHIgc4R86D9WOUf96B JhVzWDn+6CMt//foPyAlIEM2tBnfGBObv4dY9tfq2zrVzs7uBNJEKfMnKSzZLQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EFD906953; Mon, 27 Jun 2022 19:35:36 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 25RJZamp019841; Mon, 27 Jun 2022 19:35:36 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 25RJZaV3019840; Mon, 27 Jun 2022 19:35:36 GMT (envelope-from git) Date: Mon, 27 Jun 2022 19:35:36 GMT Message-Id: <202206271935.25RJZaV3019840@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mitchell Horne Subject: git: a8f5d758c41a - stable/13 - savecore: add an option to save a live minidump List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mhorne X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: a8f5d758c41a4acab53596aa01584cea826d9169 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656358538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Y49yigPYmo/IjWAkz+krQ6XZggKzvI406xq4rJersmk=; b=F47etHU/wjJvrPe46gSOHXsnGtDbaxnG7WExlt/6IOvv2Q5NgsTE4K6j11x8VM7x9u4Ouf CVcfHBIG73ynaQ3UiXKKxYZMQf2TeAlYSM9CyVA9QKR6SS+Jk4ioKfPa/9REP/q59nThd9 AXopgS8QOEgdYp9pH6omZkZK6yoOxGGY9CfXoeplykYxHRX3HfW7ugl5r46YM+56xKYewP VtIn7y6evgIt+I/Zbn6zbDx10E7/USUyEjsrIuWBWz0qYjU8C75dcCjVAYCVqzN30feml7 VuUNSbJqaSFLTcj5U+ud2XbJOnum9gF2A9/O8U86obx7z1Rocn+09zgLp/PTdg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1656358538; a=rsa-sha256; cv=none; b=r1//45rzz15T2e9mm8IuLKgNBaq/+QY+6PUYZJhEE4PUY5K7UqO8XyIajT6Pe+5upA4ruS 0kPF6GfqP9XZyisYZ/E6Zd0OyHg/EoBapIxvopgvIldTeiG5K750p1EJMa50uChwiXQMa8 vdohutFIxJRzu5Lz0wdYOafTf/rVKR71jmnkSX0iGUlhj3yIB6EaQG04jWuTOgFi2+lK17 4vg8kMTVgDuFc6tP/ZH2Cj8XD5Hrg/Z6BrjzP6MVDxkPFZdoOux5dzKuJsBzRhJqQMFY40 XxNG6oZIFbzqscpmeyHJl8bPhD7EUZKPgA2EooT09dorazrjD0v0yG6ZEjYJmg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by mhorne: URL: https://cgit.FreeBSD.org/src/commit/?id=a8f5d758c41a4acab53596aa01584cea826d9169 commit a8f5d758c41a4acab53596aa01584cea826d9169 Author: Mitchell Horne AuthorDate: 2022-04-18 15:22:07 +0000 Commit: Mitchell Horne CommitDate: 2022-06-27 19:32:06 +0000 savecore: add an option to save a live minidump The new '-L' flag will cause savecore to invoke the new mem(4) kernel dump ioctl, taking a dump of the running system and writing the result to a temporary file. Validation of the dump header is performed, similar to regular crash dumps, and the final result is written to livecore.X[.zst|.gz]. Also added is the '-Z' flag, which instructs the kernel to compress the livedump compressed with zstd, akin to the existing -z flag. This option has no effect in normal savecore(8) operation, but in theory could be extended to perform such compression while reading the dump from the dump device. Encryption is unsupported for live dumps. For example: 'savecore -Lz /var/crash' would create: /var/crash/livecore.0.gz Reviewed by: markj MFC after: 2 weeks Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D34347 (cherry picked from commit 0a5c04a8926e0418c556dfb65efbb24b3bf41dd0) --- sbin/savecore/savecore.8 | 26 +++++- sbin/savecore/savecore.c | 229 ++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 250 insertions(+), 5 deletions(-) diff --git a/sbin/savecore/savecore.8 b/sbin/savecore/savecore.8 index 512da5cc3dae..b1e0c38c6b44 100644 --- a/sbin/savecore/savecore.8 +++ b/sbin/savecore/savecore.8 @@ -28,7 +28,7 @@ .\" From: @(#)savecore.8 8.1 (Berkeley) 6/5/93 .\" $FreeBSD$ .\" -.Dd November 17, 2020 +.Dd April 4, 2022 .Dt SAVECORE 8 .Os .Sh NAME @@ -44,6 +44,11 @@ .Op Fl v .Op Ar device ... .Nm +.Fl L +.Op Fl fvZz +.Op Fl m Ar maxdumps +.Op Ar directory +.Nm .Op Fl -libxo .Op Fl fkuvz .Op Fl m Ar maxdumps @@ -86,6 +91,11 @@ Force a dump to be taken even if either the dump was cleared or if the dump header information is inconsistent. .It Fl k Do not clear the dump after saving it. +.It Fl L +Instruct +.Nm +to generate and save a kernel dump of the running system, rather than +copying one from a dump device. .It Fl m Ar maxdumps Maximum number of dumps to store. Once the number of stored dumps is equal to @@ -97,6 +107,14 @@ Uncompress the dump in case it was compressed by the kernel. .It Fl v Print out some additional debugging information. Specify twice for more information. +.It Fl Z +Compress the dump (see +.Xr zstd 1 ) . +This option is only supported in conjunction with the +.Fl L +option. +Regular dumps can be configured for compression with zstd using +.Xr dumpon 8 . .It Fl z Compress the dump (see .Xr gzip 1 ) . @@ -104,6 +122,10 @@ The dump may already be compressed if the kernel was configured to do so by .Xr dumpon 8 . In this case, the option has no effect. +.Pp +If used in conjunction with the +.Fl L +option, the requested live dump will be compressed with gzip. .El .Pp The @@ -171,9 +193,11 @@ is meant to be called near the end of the initialization file .Xr rc 8 ) . .Sh SEE ALSO .Xr gzip 1 , +.Xr zstd 1 , .Xr getbootfile 3 , .Xr libxo 3 , .Xr xo_parse_args 3 , +.Xr mem 4 , .Xr textdump 4 , .Xr tar 5 , .Xr crashinfo 8 , diff --git a/sbin/savecore/savecore.c b/sbin/savecore/savecore.c index 4444b845babb..d0c93e475f9b 100644 --- a/sbin/savecore/savecore.c +++ b/sbin/savecore/savecore.c @@ -68,6 +68,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -106,9 +107,11 @@ __FBSDID("$FreeBSD$"); static cap_channel_t *capsyslog; static fileargs_t *capfa; static bool checkfor, compress, uncompress, clear, force, keep; /* flags */ +static bool livecore; /* flags cont. */ static int verbose; static int nfound, nsaved, nerr; /* statistics */ static int maxdumps; +static uint8_t comp_desired; extern FILE *zdopen(int, const char *); @@ -393,6 +396,12 @@ saved_dump_remove(int savedirfd, int bounds) (void)unlinkat(savedirfd, path, 0); (void)snprintf(path, sizeof(path), "textdump.tar.%d.gz", bounds); (void)unlinkat(savedirfd, path, 0); + (void)snprintf(path, sizeof(path), "livecore.%d", bounds); + (void)unlinkat(savedirfd, path, 0); + (void)snprintf(path, sizeof(path), "livecore.%d.gz", bounds); + (void)unlinkat(savedirfd, path, 0); + (void)snprintf(path, sizeof(path), "livecore.%d.zst", bounds); + (void)unlinkat(savedirfd, path, 0); } static void @@ -408,6 +417,9 @@ symlinks_remove(int savedirfd) (void)unlinkat(savedirfd, "vmcore_encrypted.last.gz", 0); (void)unlinkat(savedirfd, "textdump.tar.last", 0); (void)unlinkat(savedirfd, "textdump.tar.last.gz", 0); + (void)unlinkat(savedirfd, "livecore.last", 0); + (void)unlinkat(savedirfd, "livecore.last.gz", 0); + (void)unlinkat(savedirfd, "livecore.last.zst", 0); } /* @@ -731,6 +743,183 @@ DoTextdumpFile(int fd, off_t dumpsize, off_t lasthd, char *buf, return (0); } +static void +DoLiveFile(const char *savedir, int savedirfd, const char *device) +{ + char infoname[32], corename[32], linkname[32], tmpname[32]; + struct mem_livedump_arg marg; + struct kerneldumpheader kdhl; + xo_handle_t *xostdout; + off_t dumplength; + uint32_t version; + int fddev, fdcore; + int bounds; + int error, status; + + bounds = getbounds(savedirfd); + status = STATUS_UNKNOWN; + + xostdout = xo_create_to_file(stdout, XO_STYLE_TEXT, 0); + if (xostdout == NULL) { + logmsg(LOG_ERR, "xo_create_to_file() failed: %m"); + return; + } + + /* + * Create a temporary file. We will invoke the live dump and its + * contents will be written to this fd. After validating and removing + * the kernel dump header from the tail-end of this file, it will be + * renamed to its definitive filename (e.g. livecore.2.gz). + * + * If any errors are encountered before the rename, the temporary file + * is unlinked. + */ + strcpy(tmpname, "livecore.tmp.XXXXXX"); + fdcore = mkostempsat(savedirfd, tmpname, 0, 0); + if (fdcore < 0) { + logmsg(LOG_ERR, "error opening temp file: %m"); + return; + } + + fddev = fileargs_open(capfa, device); + if (fddev < 0) { + logmsg(LOG_ERR, "%s: %m", device); + goto unlinkexit; + } + + bzero(&marg, sizeof(marg)); + marg.fd = fdcore; + marg.compression = comp_desired; + if (ioctl(fddev, MEM_KERNELDUMP, &marg) == -1) { + logmsg(LOG_ERR, + "failed to invoke live-dump on system: %m"); + close(fddev); + goto unlinkexit; + } + + /* Close /dev/mem fd, we are finished with it. */ + close(fddev); + + /* Seek to the end of the file, minus the size of the header. */ + if (lseek(fdcore, -(off_t)sizeof(kdhl), SEEK_END) == -1) { + logmsg(LOG_ERR, "failed to lseek: %m"); + goto unlinkexit; + } + + if (read(fdcore, &kdhl, sizeof(kdhl)) != sizeof(kdhl)) { + logmsg(LOG_ERR, "failed to read kernel dump header: %m"); + goto unlinkexit; + } + /* Reset cursor */ + (void)lseek(fdcore, 0, SEEK_SET); + + /* Validate the dump header. */ + version = dtoh32(kdhl.version); + if (compare_magic(&kdhl, KERNELDUMPMAGIC)) { + if (version != KERNELDUMPVERSION) { + logmsg(LOG_ERR, + "unknown version (%d) in dump header on %s", + version, device); + goto unlinkexit; + } else if (kdhl.compression != comp_desired) { + /* This should be impossible. */ + logmsg(LOG_ERR, + "dump compression (%u) doesn't match request (%u)", + kdhl.compression, comp_desired); + if (!force) + goto unlinkexit; + } + } else { + logmsg(LOG_ERR, "magic mismatch on live dump header"); + goto unlinkexit; + } + if (kerneldump_parity(&kdhl)) { + logmsg(LOG_ERR, + "parity error on last dump header on %s", device); + nerr++; + status = STATUS_BAD; + if (!force) + goto unlinkexit; + } else { + status = STATUS_GOOD; + } + + nfound++; + dumplength = dtoh64(kdhl.dumplength); + if (dtoh32(kdhl.dumpkeysize) != 0) { + logmsg(LOG_ERR, + "dump header unexpectedly reported keysize > 0"); + goto unlinkexit; + } + + /* Remove the vestigial kernel dump header. */ + error = ftruncate(fdcore, dumplength); + if (error != 0) { + logmsg(LOG_ERR, "failed to truncate the core file: %m"); + goto unlinkexit; + } + + if (verbose >= 2) { + printf("\nDump header:\n"); + printheader(xostdout, &kdhl, device, bounds, -1); + printf("\n"); + } + logmsg(LOG_ALERT, "livedump"); + + writebounds(savedirfd, bounds + 1); + saved_dump_remove(savedirfd, bounds); + + snprintf(corename, sizeof(corename), "livecore.%d", bounds); + if (compress) + strcat(corename, kdhl.compression == KERNELDUMP_COMP_ZSTD ? + ".zst" : ".gz"); + + if (verbose) + printf("renaming %s to %s\n", tmpname, corename); + if (renameat(savedirfd, tmpname, savedirfd, corename) != 0) { + logmsg(LOG_ERR, "renameat failed: %m"); + goto unlinkexit; + } + + snprintf(infoname, sizeof(infoname), "info.%d", bounds); + if (write_header_info(xostdout, &kdhl, savedirfd, infoname, device, + bounds, status) != 0) { + nerr++; + return; + } + + logmsg(LOG_NOTICE, "writing %score to %s/%s", + compress ? "compressed " : "", savedir, corename); + + if (verbose) + printf("\n"); + + symlinks_remove(savedirfd); + if (symlinkat(infoname, savedirfd, "info.last") == -1) { + logmsg(LOG_WARNING, "unable to create symlink %s/%s: %m", + savedir, "info.last"); + } + + snprintf(linkname, sizeof(linkname), "livecore.last"); + if (compress) + strcat(linkname, kdhl.compression == KERNELDUMP_COMP_ZSTD ? + ".zst" : ".gz"); + if (symlinkat(corename, savedirfd, linkname) == -1) { + logmsg(LOG_WARNING, "unable to create symlink %s/%s: %m", + savedir, linkname); + } + + nsaved++; + if (verbose) + printf("dump saved\n"); + + close(fdcore); + return; +unlinkexit: + funlinkat(savedirfd, tmpname, fdcore, 0); + close(fdcore); +} + static void DoFile(const char *savedir, int savedirfd, const char *device) { @@ -748,6 +937,12 @@ DoFile(const char *savedir, int savedirfd, const char *device) uint32_t dumpkeysize; bool iscompressed, isencrypted, istextdump, ret; + /* Live kernel dumps are handled separately. */ + if (livecore) { + DoLiveFile(savedir, savedirfd, device); + return; + } + bounds = getbounds(savedirfd); dumpkey = NULL; mediasize = 0; @@ -1220,9 +1415,10 @@ init_caps(int argc, char **argv) static void usage(void) { - xo_error("%s\n%s\n%s\n", + xo_error("%s\n%s\n%s\n%s\n", "usage: savecore -c [-v] [device ...]", " savecore -C [-v] [device ...]", + " savecore -L [-fvZz] [-m maxdumps] [directory]", " savecore [-fkuvz] [-m maxdumps] [directory [device ...]]"); exit(1); } @@ -1235,10 +1431,11 @@ main(int argc, char **argv) char **devs; int i, ch, error, savedirfd; - checkfor = compress = clear = force = keep = false; + checkfor = compress = clear = force = keep = livecore = false; verbose = 0; nfound = nsaved = nerr = 0; savedir = "."; + comp_desired = KERNELDUMP_COMP_NONE; openlog("savecore", LOG_PERROR, LOG_DAEMON); signal(SIGINFO, infohandler); @@ -1247,7 +1444,7 @@ main(int argc, char **argv) if (argc < 0) exit(1); - while ((ch = getopt(argc, argv, "Ccfkm:uvz")) != -1) + while ((ch = getopt(argc, argv, "CcfkLm:uvZz")) != -1) switch(ch) { case 'C': checkfor = true; @@ -1261,6 +1458,9 @@ main(int argc, char **argv) case 'k': keep = true; break; + case 'L': + livecore = true; + break; case 'm': maxdumps = atoi(optarg); if (maxdumps <= 0) { @@ -1274,8 +1474,16 @@ main(int argc, char **argv) case 'v': verbose++; break; + case 'Z': + /* No on-the-fly compression with zstd at the moment. */ + if (!livecore) + usage(); + compress = true; + comp_desired = KERNELDUMP_COMP_ZSTD; + break; case 'z': compress = true; + comp_desired = KERNELDUMP_COMP_GZIP; break; case '?': default: @@ -1289,6 +1497,8 @@ main(int argc, char **argv) usage(); if (compress && uncompress) usage(); + if (livecore && (checkfor || clear || uncompress || keep)) + usage(); argc -= optind; argv += optind; if (argc >= 1 && !checkfor && !clear) { @@ -1301,7 +1511,15 @@ main(int argc, char **argv) argc--; argv++; } - if (argc == 0) + if (livecore) { + if (argc > 0) + usage(); + + /* Always need /dev/mem to invoke the dump */ + devs = malloc(sizeof(char *)); + devs[0] = strdup("/dev/mem"); + argc++; + } else if (argc == 0) devs = enum_dumpdevs(&argc); else devs = devify(argc, argv); @@ -1314,6 +1532,9 @@ main(int argc, char **argv) (void)cap_rights_init(&rights, CAP_CREATE, CAP_FCNTL, CAP_FSTATAT, CAP_FSTATFS, CAP_PREAD, CAP_SYMLINKAT, CAP_FTRUNCATE, CAP_UNLINKAT, CAP_WRITE); + if (livecore) + cap_rights_set(&rights, CAP_RENAMEAT_SOURCE, + CAP_RENAMEAT_TARGET); if (caph_rights_limit(savedirfd, &rights) < 0) { logmsg(LOG_ERR, "cap_rights_limit(): %m"); exit(1);