From nobody Sat Apr 26 19:21:44 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZlKMY29HKz5tfBx; Sat, 26 Apr 2025 19:21:45 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZlKMX5jjvz3Fp7; Sat, 26 Apr 2025 19:21:44 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745695304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4upHzEOXFZZUY6bO3TURyp/sz7JMv7CsEDf0vOKedKQ=; b=iAVNfnU+RmQZYwIBNLn28KQ2Bu2nZSc/Sr8BOF7GKgob9QWeURV1JiTWqqeYUl3nQvCdzK Nn3t+HEsrFMFX8xr/xQz0QDyeYRswxbXZzC6iV6zrt0JRfHRy4ENxGpEngPamdb0wxK8i0 kKaLjCUj6OaaDsVQ0+UIav55tCwCVPUgFfOUcgadoOezhCuMTCkxP5Dh8g2I2SJjMoPSEV QOti7lQO4eKuSlZbhCwhtLVA9BDLiQ4HM+I/dB3TZ0Oa6lUnElWWJkRJKw3/bxOeR4yVhH WTfLd5YC6D0ay0zANJrniyBcMB4urjkCKJbKV8MIr4WlijxZFbsL6+bePtM6rA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1745695304; a=rsa-sha256; cv=none; b=HGI9/o/hrPTvsflx0gMLNJFtue+QFGsf4TqHxih++36X3ux6wEoWjYw2i5/SATrjf4hULD WdwN/eebq98tHSbHgYjTIZ0eCJ1pqVSWgHd2fr1nKmHdCfFeiPyIPSdyka0Q5HrwF0lQxt p7Il/VCHcI5sf65yGcJoBrwmfCtl52zZsCOZLNzy1oanjYvQelVzGUHV2YMOW0yNCFaBoN Cgt1Jp7/Jj3uBC0/9aQDOBk1UyKpDjhSflGax8n/jGPvGDFkPgVAIHTsAywVqN8Kr74K5r 5bqJRENzrVc6dtVxACdBgBMSPq5FzrK47zwwtZ7ZSj0ywsWAqN6VxJbvcNkvTQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745695304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4upHzEOXFZZUY6bO3TURyp/sz7JMv7CsEDf0vOKedKQ=; b=ojSXUJPR9Kh/SxPxsX+3i8WZ5Olc4iaCRje0iJMvTnByFoz01CQJrV9afTIjTzvygV31Wq MzE/zQqRiUnbe+G4vgFkWS/wEmhXuSjrb+yfOsh9/nLXZQ9g2zY7LaPLICZIEJwXebYnNo KOVHNCjulDtDj2APwgsz/JGR1fXuT3MWIJj2FG5XX3O7gR46Rp3n33lzT2v150Huvo9Krb 6NM6SGLxIGGA26vkLr4IJ272n2obe4+UnwGCGHAniZpCEGtwOVv7VYiZyFUZ7VC0jF/4+B VmCp2CwwyqSswvIFeILeJCUFMCmnZ2ZtWnKx7W+6+lvrh8O3NkVYSWy7Br4woQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZlKMX57BLzZM5; Sat, 26 Apr 2025 19:21:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53QJLiDt009489; Sat, 26 Apr 2025 19:21:44 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53QJLiU4009486; Sat, 26 Apr 2025 19:21:44 GMT (envelope-from git) Date: Sat, 26 Apr 2025 19:21:44 GMT Message-Id: <202504261921.53QJLiU4009486@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Colin Percival Subject: git: 925ec43c1525 - stable/14 - EC2: Add AMI Builder AMI building List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: cperciva X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 925ec43c152566b28cf00f8ec0d11715263a6c3a Auto-Submitted: auto-generated The branch stable/14 has been updated by cperciva: URL: https://cgit.FreeBSD.org/src/commit/?id=925ec43c152566b28cf00f8ec0d11715263a6c3a commit 925ec43c152566b28cf00f8ec0d11715263a6c3a Author: Colin Percival AuthorDate: 2025-04-20 16:38:48 +0000 Commit: Colin Percival CommitDate: 2025-04-26 19:17:51 +0000 EC2: Add AMI Builder AMI building Starting in 2015 I have published "AMI Builder AMIs" for FreeBSD/EC2: These boot into a memory disk, extract a "clean" copy of FreeBSD onto the root disk, mount it at /mnt, and allow the user to SSH in to make customizations before creating a new AMI from the "running" instance (in fact, from the FreeBSD installation which is not running but is mounted on /mnt). This provides a much cleaner mechanism for building customized FreeBSD AMIs than the traditional Linux approach of "launch an EC2 instance, SSH in and configure it, then try to wipe logs and credentials before creating an AMI"; and it's easier than building a customized AMI ab initio by modifying the FreeBSD release-building code. This commit brings that functionality into the FreeBSD src tree and into the collection of images built by the release engineering team: The EC2 "BUILDER" flavour AMI is essentially a "SMALL" flavour AMI with a compressed "BASE" flavour disk image, plus an init script which juggles disks around (rerooting into a memory disk and extracting the "BASE" image onto disk). Polished by: bz, emaste MFC after: 1 week Sponsored by: Amazon Differential Revision: https://reviews.freebsd.org/D49930 (cherry picked from commit 58426589030308cd632477d328b9536b1634c54d) --- release/Makefile.vm | 13 ++++++-- release/tools/ec2-builder.conf | 57 ++++++++++++++++++++++++++++++++ release/tools/mkami.sh | 65 +++++++++++++++++++++++++++++++++++++ release/tools/rc.amibuilder | 74 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 207 insertions(+), 2 deletions(-) diff --git a/release/Makefile.vm b/release/Makefile.vm index 7058d8be5987..7de07c6da714 100644 --- a/release/Makefile.vm +++ b/release/Makefile.vm @@ -35,8 +35,9 @@ BASIC-CLOUDINIT_FSLIST?= ufs zfs BASIC-CLOUDINIT_DESC?= Images for VM with cloudinit disk config support EC2_FORMAT= raw EC2_FSLIST?= ufs zfs -EC2_FLAVOURS?= BASE CLOUD-INIT SMALL +EC2_FLAVOURS?= BASE BUILDER CLOUD-INIT SMALL EC2-BASE_DESC= Amazon EC2 image +EC2-BUILDER_DESC= Amazon EC2 AMI Builder image EC2-CLOUD-INIT_DESC= Amazon EC2 Cloud-Init image EC2-SMALL_DESC= Amazon EC2 small image GCE_FORMAT= raw @@ -112,12 +113,20 @@ CLEANFILES+= ${_CW:tl}.${_FS}.${_FMT}.img \ ${_CW:tl}.${_FS}.${_FMT}.raw ${_CW:tu}${_FS:tu}${_FMT:tu}IMAGE= ${_CW:tl}.${_FS}.${_FMT} +# Special handling: EC2 "AMI Builder" images need a copy of the "base" disk +# built first. +.if ${_CW} == EC2-BUILDER +cw-${_CW:tl}-${_FS}-${_FMT}: cw-ec2-base-${_FS}-${_FMT} +.endif + cw-${_CW:tl}-${_FS}-${_FMT}: ${QEMUTGT} mkdir -p ${.OBJDIR}/${.TARGET} env TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} SWAPSIZE=${SWAPSIZE} \ QEMUSTATIC=${QEMUSTATIC} \ + EC2BASEIMG=${.OBJDIR}/${EC2-BASE${_FS:tu}${_FMT:tu}IMAGE} \ ${.CURDIR}/scripts/mk-vmimage.sh \ - -C ${.CURDIR}/tools/vmimage.subr -d ${.OBJDIR}/${.TARGET} -F ${_FS} \ + -C ${.CURDIR}/tools/vmimage.subr -d ${.OBJDIR}/${.TARGET} \ + -F ${"${_CW:MEC2-BUILDER}" != "":?ufs:${_FS}} \ -i ${.OBJDIR}/${_CW:tl}.${_FS}.${_FMT}.img -s ${VMSIZE} -f ${_FMT} \ -S ${WORLDDIR} -o ${.OBJDIR}/${${_CW:tu}${_FS:tu}${_FMT:tu}IMAGE} -c ${${_CW:tu}CONF} touch ${.TARGET} diff --git a/release/tools/ec2-builder.conf b/release/tools/ec2-builder.conf new file mode 100644 index 000000000000..7927fc13a74d --- /dev/null +++ b/release/tools/ec2-builder.conf @@ -0,0 +1,57 @@ +#!/bin/sh + +. ${WORLDDIR}/release/tools/ec2.conf + +# Build with a 7.9 GB partition; this is enough for our stripped-down +# base system plus the compressed ec2-base image. +export VMSIZE=8000m + +# Flags to installworld/kernel: We don't want debug symbols (kernel or +# userland), 32-bit libraries, tests, or the debugger. +export INSTALLOPTS="WITHOUT_DEBUG_FILES=YES WITHOUT_KERNEL_SYMBOLS=YES \ + WITHOUT_LIB32=YES WITHOUT_TESTS=YES WITHOUT_LLDB=YES" + +# Packages to install into the image we're creating. In addition to packages +# present on all EC2 AMIs, we install: +# * ec2-scripts, which provides a range of EC2ification startup scripts, +# * isc-dhcp44-client, used for IPv6 network setup, and +# * py-awscli, to make it easier for users to create AMIs. +export VM_EXTRA_PACKAGES="${VM_EXTRA_PACKAGES} ec2-scripts \ + isc-dhcp44-client devel/py-awscli" + +# Services to enable in rc.conf(5). +export VM_RC_LIST="${VM_RC_LIST} ec2_configinit ec2_ephemeral_swap \ + ec2_fetchkey ec2_loghostkey sshd" + +vm_extra_pre_umount() { + # Any EC2 ephemeral disks seen when the system first boots will + # be "new" disks; there is no "previous boot" when they might have + # been seen and used already. + touch ${DESTDIR}/var/db/ec2_ephemeral_diskseen + + # Configuration common to all EC2 AMIs + ec2_common + + # Standard FreeBSD network configuration + ec2_base_networking + + # Grab a copy of the ec2-base disk image, and compress it + zstd < ${EC2BASEIMG} > ${DESTDIR}/image.zst + + # Disable fortune so we don't have extra noise at login + chmod a-x ${DESTDIR}/usr/bin/fortune + + # Install the AMI-building script + install -m 755 ${WORLDDIR}/release/tools/mkami.sh ${DESTDIR}/bin/mkami + + # Install an /etc/rc which juggles disks around for us + install -m 755 ${WORLDDIR}/release/tools/rc.amibuilder ${DESTDIR}/etc + + # We want to mount from the UFS disk and juggle disks first + cat >> ${DESTDIR}/boot/loader.conf <<-EOF + vfs.root.mountfrom="ufs:/dev/gpt/rootfs" + init_script="/etc/rc.amibuilder" + EOF + + return 0 +} diff --git a/release/tools/mkami.sh b/release/tools/mkami.sh new file mode 100644 index 000000000000..cfbbcd3bc8a9 --- /dev/null +++ b/release/tools/mkami.sh @@ -0,0 +1,65 @@ +#!/bin/sh -e +# +# Copyright (c) 2015 Colin Percival +# +# SPDX-License-Identifier: BSD-2-Clause +# +# mkami.sh: Create an AMI from the currently running EC2 instance. +# + +export PATH=$PATH:/usr/local/bin + +NAME=$1 +if [ -z "$NAME" ]; then + echo "usage: mkami []" + exit 1 +fi +DESC=$2 +if ! [ -z "$DESC" ]; then + DESCOPT="--description '$DESC'" +fi + +# Get the instance ID and region from the EC2 Instance Metadata Service: +# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html +TMPFILE=`mktemp` +fetch -qo $TMPFILE http://169.254.169.254/latest/dynamic/instance-identity/document +INST=`awk -F \" '/"instanceId"/ { print $4 }' $TMPFILE` +REGION=`awk -F \" '/"region"/ { print $4 }' $TMPFILE` +rm $TMPFILE +CMD="aws --region $REGION ec2 create-image --instance-id $INST --output text --no-reboot --name '$NAME' $DESCOPT" + +# Unmount the new system image +if mount -p | grep -q '/mnt.*ufs'; then + echo -n "Unmounting new system image..." + sync + umount /mnt + sync + sleep 5 + sync + echo " done." +elif mount -p | grep -q '/mnt.*zfs'; then + echo -n "Unmounting new system image..." + sync + zfs umount -a + zfs umount zroot/ROOT/default + sync + sleep 5 + sync + echo " done." +fi + +if eval "$CMD" --dry-run 2>&1 | + grep -qE 'UnauthorizedOperation|Unable to locate credentials'; then + echo "This EC2 instance does not have permission to create AMIs." + echo "Launch an AMI-builder instance with an appropriate IAM Role," + echo "create an AMI from this instance via the AWS Console, or run" + echo "the following command from a system with the necessary keys:" + echo + echo "$CMD" + exit +fi + +echo -n "Creating AMI..." +AMINAME=`eval "$CMD"` +echo " done." +echo "AMI created in $REGION: $AMINAME" diff --git a/release/tools/rc.amibuilder b/release/tools/rc.amibuilder new file mode 100644 index 000000000000..b25e8facb597 --- /dev/null +++ b/release/tools/rc.amibuilder @@ -0,0 +1,74 @@ +#!/bin/sh +# +# Copyright (c) 2025 Colin Percival +# +# SPDX-License-Identifier: BSD-2-Clause +# +# rc.amibuilder: Juggle disks to reroot into a memory disk and install a clean +# copy of FreeBSD onto the root disk. +# + +# Do nothing if init(8) is in the middle of rerooting +if ps -p 1 -o command | grep -q reroot; then + exit 0 +fi + +# Figure out which partition we boot from +BOOTPART=$(sysctl -n kern.geom.conftxt | + grep -E 'PART|gpt/rootfs' | + grep -B 1 gpt/rootfs | + awk '{ print $3 }' | + head -1) +BOOTDISK=${BOOTPART%%p*} +BOOTPARTNUM=${BOOTPART##*p} + +# First pass: Copy ourselves into a memory disk and reroot into it +if ! [ -c /dev/md0 ]; then + # Create a memory disk of appropriate size and copy the disk + echo "Copying FreeBSD into memory disk..." + DISKBYTES=$(diskinfo ${BOOTDISK} | awk '{print $3}') + mdconfig -a -t swap -s ${DISKBYTES}b -u 0 + dd if=/dev/${BOOTDISK} of=/dev/md0 bs=1M + + # Reboot into the memory disk we just created + echo "Rebooting into memory disk..." + kenv vfs.root.mountfrom="ufs:/dev/md0p${BOOTPARTNUM}" + reboot -r + + # Lose a race against init + sleep 10 + exit 1 +fi + +# Second pass: Extract a clean copy of FreeBSD onto the physical disk +echo "Installing base FreeBSD image..." +sysctl kern.geom.debugflags=16 +zstdcat < /image.zst | dd bs=1M of=/dev/${BOOTDISK} + +# Mount the clean image +if gpart show ${BOOTDISK} | grep -q freebsd-ufs; then + mount /dev/${BOOTPART} /mnt +else + zpool import -aNR /mnt + zfs mount zroot/ROOT/default + zfs mount -a +fi + +# Provide instructions for when the user logs in +mount -w / +cat >/etc/motd.template < [] + +to create the AMI. Don't forget to shut down this instance when +you're done! + +EOF +mount -o ro / + +# After we exit, the boot proceeds with init spawning /etc/rc normally +exit 0