what next for the pkg_install rewrite

[Garrett Cooper]

On Thu, Aug 19, 2010 at 3:10 PM, Ivan Voras <ivoras at freebsd.org> wrote:
> On 19/08/2010, jhell <jhell at dataix.net> wrote:
>>       Adding to this I would like to see a central database created for
>> packages that have been removed like in Slackware Linux. They keep a
>> file in /var/log/preserved_packages with a flat text format with the
>> file name looking like:
>>
>> ${PORTNAME}-${PORTVERSION}${PORTREVISION}-`date +%Y%m%d%H%M%S`

Please no. We need another ad hoc text format like we need holes in our head.

> Ah yes, you reminded me of this other thing: I would also suggest
> getting rid of text files carrying rich information in ad-hoc formats
> :)
>
> I'm not saying XML should be the only choice, but it *is* well
> supported - expat is even in base as libbsdxml.
>
> While suggesting nebulous things I know will be hard to pass near a
> lot of people: sqlite is *the* choice for any record-based file
> databases today. The single most important thing I'll promote with it
> is its transaction capabilities and ACID - these would get much use if
> parallel operations (upgrades / installs) are to be supported. There
> are a ton of other reasons too.
>
> I started writing this a long time ago but abandoned it because of
> strong opposition: http://wiki.freebsd.org/PortsUsingSQLite - maybe it
> would help at this time.

There are a lot of ideas going around, but unless the current patches
get polished up and portmgr (points in flz's general direction)
actually accept ideas, ideas are nothing more than ideas and will
essentially be vaporware. That has been the chokepoint for patches
I've added to the PR database.

Also, if it breaks backwards compatibility used everywhere over the
map and is actually functionality that's used in ports and src
packages without equivalent functionality, it's a non-starter.

If any work needs to be done, we should be getting _rid_ of features
which aren't in use, or can be easily replaced with equivalent
methods. Stuff I've identified over the past year that can be worked
on are:

========
pkg_install
========

Locking:
- We need concurrent package installation.

Packing list:

@dirrmtry (needs to be replaced with scripting infrastructure called
via +INSTALL/+DEINSTALL). See @exec/@unexec (similar problem).
@exec/@unexec (needs to be replaced with scripting infrastructure
using +INSTALL/+DEINSTALL).
@group/@mode/@owner (needs to be replaced with scripting infrastructure).
@srcdir really shouldn't be in a plist (someone can script the call
easily with pkg_create).

At the end of the day, it would be really nice if there was one small
file with the package metadata, and the other file was checksums via
an mtree file. mtree files will solve a lot of the problems with the
ad hoc plist `format'.

@noinst should be checked to see whether or not it's widely used in
ports. It might be a good idea to implement, but I'm not confident in
that statement.

Dependencies:

- We should be using reverse dependencies, not forward dependencies.
There's less value in forward dependencies, because I can uninstall
things, and I don't have an accurate idea of what the current state is
on the system. Reverse dependencies actually tell us _what_ we depend
on, and should be expressed in terms of origins, not package versions.
Whenever the version for the origins change, the package should be
bumped. This would (IMO) reduce a lot of headaches with packages.

INDEX*:
- We should depend on INDEX* for package data, and it should be
distributed with the base system, not ports, and ports should build
off of this data.

*libpkg*!!:
- I emphasize this, because it's extremely important... David/Florent
are already doing work to make this a reality, BUT we shouldn't be
duplicating their work. It would be nice if the work being done by
both Florent and David was more transparent, could be tasked out to
various individuals, etc... but that's not how it is today.

========
Ports
========

bsd.*.mk:
- Needs to use pkg_add / pkg_create out of the box instead of
installing things directly into <DESTDIR>/<LOCALBASE>. This would
avoid a lot of issues with corrupted installs, etc.
- [Nice to have] should be trimmed wherever possible. There's a lot of
cruft that actually could be moved elsewhere to reduce the amount of
work in terms of parsing and evaluating statements for everyday ports
users.

In general:
- Ports need to be fixed so that BUILD_DEPENDS and RUN_DEPENDS don't
end up in the package dependency list. We suffer from this in areas,
Redhat suffers from this, etc, and it's not an easy task to do.
However, if done properly, this will 1) speed up package building, 2)
package installation, 3) reduce installed package count, etc.


Just for the sake of not repeating past discussions:

1. SQLite was killed before because of complexity and because it was
needs another package in base that isn't BSD licensed. That's why
everyone in the know has been pushing for BDB 1.8x (in base). People
suggested that to me back when I was trying to get off the ground in
GSoC 2006, they did it to you before Ivan, and I'm sure they've done
it before in the past. We need to implement things in terms of BDB
first, but make the APIs generic enough so that it _could_ be extended
to SQLite if people chose to do the work later on.
2. XML is a bad idea. Great in theory, wonderful in my browser, but a
bloated plaintext file with a lot of complexity. I would prefer a
database solution that could be dumped to a simple text file format if
needed.

Again (and I can't overemphasize this): no matter what we do, say,
etc, unless we have buy-in from portmgr beforehand on anything here,
the work will never see a ports/src CVS/svn repo, etc. This includes
work done for past GSoCs, and current GSoCs.

Thanks,
-Garrett