Instance of 11.3 got "pkg: repository meta has wrong version 2"

[Li-Wen Hsu]

This is a stripped version from several private discussion. I think it
will be useful to post here.
CC bapt as he helped answer lots of my pkg questions, all the correct
things following are from him, and hope he can correct if I have
misunderstands below. :-)

The background is some users reported that when launch instance of
11.3, pkg doesn't work at all and the error message is: "pkg:
repository meta has wrong version 2"

In short, this is due to the pkg installed (already bootstraped when
creating the image) in the VM cannot handle the repository metadata
created by newer pkg.  This is safe and recommended to do a `pkg
bootstrap -f` on the first boot of a VM created from an image. The
`pkg update` is also recommended but not necessary, as the pkg will
check if the index needs updating automatically.

Long version:

"pkg: repository meta has wrong version 2" means that pkg on client
cannot handle the repository created by newer pkg on server. There are
multiple versions of the repositories to allow backward compatibility.
After a certain amount of time, we change the default version and
deprecated the old versions on the server side.

The problem we had is that the repository version 2 is very old
(likely pkg 1.10), and the repository created by it was deprecated
when pkg 1.12 was released (and latest version is 1.14 now).  In most
cases, the machines got maintained should already receive the latest
pkg since it checks for self-update automatically.

When the 11.3 image was created, the pkg was 1.9 and because we need
to install utilities like agent for the cloud provider, the pkg was
already bootstraped in the system, and got "fixed" in the image.

A vanilla installed 11.3 also doesn't have this problem because when
the system just installed, it has no "real" pkg bootstraped, the one
comes with base system is /usr/sbin/pkg , a.k.a pkg(7), its
responsibility is bootstraping pkg(8) from package repository,
installed as /usr/local/sbin/pkg .

Finally, a `pkg bootstrap -f` is recommended on the first boot of
every VM created from an pre-built image, to ensure it gets the latest
pkg before performing all the other actions with pkg.  It is also safe
because pkg will take care of the upgrading local pkg database (which
stores the information about installed packages on the system.)

I will check if doing `ASSUME_ALWAYS_YES=yes pkg bootstrap -f; pkg
update` is feasible to make this as a default action in the
"cloudware" release target, like we run freebsd-update on firstboot.

Best,
Li-Wen