From nobody Tue May 23 18:58:27 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQk9w6Ppyz4Cp9x for ; Tue, 23 May 2023 18:58:44 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb36.google.com (mail-yb1-xb36.google.com [IPv6:2607:f8b0:4864:20::b36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQk9v6KsFz3L3m for ; Tue, 23 May 2023 18:58:43 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=B7uEzEwP; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::b36) smtp.mailfrom=tomek@cedro.info; dmarc=none Received: by mail-yb1-xb36.google.com with SMTP id 3f1490d57ef6-ba818eb96dcso69224276.0 for ; Tue, 23 May 2023 11:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1684868322; x=1687460322; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=nB+KETuhNWgh2rcrgLzJEn5b97eKXdyGdxRX3fKaMCc=; b=B7uEzEwP8SQldtubkIrU6NtvitAadiDiEB685dR5n5F2HlelFlg7a4YpTR/WNDvp+D ESrK4fRWMXshH+b38oaZR4fOWAwF6lFNAJHyRONeImAO7FPuA10RI9LZ7xLQnPfT1DdN +rfjpfYy8FBLbF4qemBVRMYYyM1NblaYqOJJJJywuNXTd5BJjhesPO0crz4eRwpFqYtP wqZnNHD9UmsKU5jkKdOO2StwJdbV0qz9lQ2omlrilsODgT12mumvqx8Ene0vQ7hyAJBe JPCAF6Eo/LHOY4+o3P2RCUjXeSeUCBLmbj68ifuJecXSNTY/FYwIpxCluZ73yQBDC7sE a29w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684868322; x=1687460322; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nB+KETuhNWgh2rcrgLzJEn5b97eKXdyGdxRX3fKaMCc=; b=hze0VMoYIM5+LmpAVrX6oXcj8Xwxl7dJ6ABaVK6YveNFq7ZPoORkdxQjHdF+Bkkh/Z q1O/ONpUJGBsrK/lxveEuZvitZ3MuhcYwJP5V9/8ux8TunUGLh3fMZyzsHEJQZ1Uu11L cUaGR/AqTCvdAu71cH1l2yi8n9vn6lJYoAr59h+nZoi+y62O1QY0fWhP2qAcLpufk6wk ld9fl90S1JdL6C/tQQAGbLnvC2GFPERYRpncnqbqUXeVprlg40du/IzYX0mllRR2ppni g0epyv4s0J5myGBTZN3Qx76gclEEp4+Z5JCEGMh+YFHBa0zygB+C5lXBMkIdqaJpQ9yF E0eg== X-Gm-Message-State: AC+VfDym4lCe/r9C8OYjzvcM0V+W+6VU/cnjbaEYu/3MMX4eFI5vp6xn kA9O26KH3hTGjO5/0CdRCeEpncFz8bsWiw1Td5g= X-Google-Smtp-Source: ACHHUZ4LaXPVxWPuZHqIBC38l4A+rOX47kIxSJJIHSOA/49K+n6SJjMf6M0uBrGBl2RC8PDKmw4aNQ== X-Received: by 2002:a25:6ac2:0:b0:b72:4ca:b3ce with SMTP id f185-20020a256ac2000000b00b7204cab3cemr14863980ybc.16.1684868322509; Tue, 23 May 2023 11:58:42 -0700 (PDT) Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com. [209.85.219.171]) by smtp.gmail.com with ESMTPSA id b185-20020a0df2c2000000b0056507de3d82sm1669836ywf.104.2023.05.23.11.58.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 23 May 2023 11:58:42 -0700 (PDT) Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-ba6d024a196so46993276.2; Tue, 23 May 2023 11:58:42 -0700 (PDT) X-Received: by 2002:a81:4e4a:0:b0:55a:985e:8ad1 with SMTP id c71-20020a814e4a000000b0055a985e8ad1mr16906355ywb.33.1684868321800; Tue, 23 May 2023 11:58:41 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> In-Reply-To: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> From: Tomek CEDRO Date: Tue, 23 May 2023 20:58:27 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Vitaliy Gusev Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; MIME_GOOD(-0.10)[text/plain]; BLOCKLISTDE_FAIL(0.00)[209.85.219.171:server fail,2607:f8b0:4864:20::b36:server fail]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b36:from,209.85.219.171:received]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; DKIM_TRACE(0.00)[cedro.info:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TAGGED_RCPT(0.00)[]; DMARC_NA(0.00)[cedro.info]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4QQk9v6KsFz3L3m X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Tue, May 23, 2023 at 6:06=E2=80=AFPM Vitaliy Gusev wrote: > Hi, > Here is a proposal for bhyve snapshot/checkpoint image format improvement= s. > It implies moving snapshot code to nvlist engine. Hey there Vitaliy :-) bhyve getting more and more traction, I am new user of bhyve and no expert, but new and missing features are welcome I guess.. there was a discussion on the mailing lists recently on better snapshots mechanism :-) > Current snapshot implementation has disadvantages: > 3 files per snapshot: .meta, .kern, vram No problem, unless new single file will be protected against corruption (filesystem, transfer, application crash) and possible to be easily and cheaply modified in place? > Binary Stream format of data. This is small and fast? Will new format too? > Adding optional variable - breaks resume > Removing variable - breaks resume > Changing saved order of variables - breaks resume Obviously need improvement :-) > Hard to get information about what is saved and decode. > Hard to debug if somethings goes wrong Additional tools missing? Will new format allow text editor interaction? > No versions. If change code, resume of an old images can be > passed, but with UB. Is new format future proof and provides backward compatibility? > New nvlist implementation should solve all things above. The first step - > improve snapshot/checkpoint saving format. It eliminates three files usag= e > per a snapshot. > > 1. BHYVE SNAPSHOT image format: > > +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ > | HEADER PHYS - 4096 BYTES | > +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ > | | > | DATA | > | | > +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ > (..) > Predefined sections: =E2=80=9Cconfig=E2=80=9D, =E2=80=9Cdevices=E2= =80=9D, =E2=80=9Ckernel=E2=80=9D, =E2=80=9Cmemory=E2=80=9D. > 4. EXAMPLE: > IDENT STRING: > "BHYVE CHECKPOINT IMAGE VERSION 1" > NVLIST HEADER: > [config] > config.offset =3D 0x1000 (4096) > config.size =3D 0x1f6 (502) > config.type =3D "text" > [kernel] > kernel.offset =3D 0x11f6 (4598) > kernel.size =3D 0x19a7 (6567) > kernel.type =3D =E2=80=9Cnvlist" > (..) So this will be new text config based format with variable =3D value and se= ctions? How much bigger will be the overal file size increase? How much longer it will take do decode/encode/process files? What is the possibility of format change and backward/foward compatibility? Have you considered efficiency comparison of current format, proposed format, and maybe using SQLITE or JSON storage/parsers? For instance sqlite would be blazingly fast but hard to migrate. json would be most versatile but more time/memory consuming? Maybe EFL approach of storing configuration files for limited resources embedded system storage that use binary storage data but can be decompressed in chunks that can be replaced in place? https://www.enlightenment.org/develop/efl/start Sorry for asking those questions but there may be already good and verified solutions out there not to reinvent the wheel? :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info