INCLUDE_CONFIG_FILE in GENERIC

[Bruce Evans]

On Thu, 14 Jan 2010, M. Warner Losh wrote:

> In message: <86625798-F339-4863-8F97-63B5232A6CF7 at freebsd.org>
>            "Robert N. M. Watson" <rwatson at freebsd.org> writes:
> : I agree. I see two kinds of users:
> :
> : - Desktop/server users who want their system to work without any
> : special tuning or magic, and likely feel the comments they put in
> : configuration files are important
> : - Users who care a about a few hundred bytes of comments due to the
> : nature of their special embedded hardware target environment

"Stripping" comments has nothing to do with saving space.  It is because
it is technically difficult, and not implemented, to not strip them,
except in the old limited code that just preserves the unprocessed
top-level-only config file.

> : As such, I see a reasonable "default" -- i.e., i386/amd64 GENERIC --
> : be to fully preserve the configuration and its comments. For the

Your code to implement this is welcome :-).  Even GENERIC is not quite
complete, since it is missing the implicit include of DEFAULTS.  OTOH,
completing the config by merging with DEFAULTS, as I think the processed
version must do, may give an unusable config by repeating things in
DEFAULTS.  The merge should probably have no processing at all, with
include files concatenated and not replacing include directives, and
DEFAULTS not included.

> I think you are confusing two different things.
>
> The embedded folks omit this entirely.  That's not the issue at all
> here.  Space is not the concern.  We're putting it into GENERIC.
>
> Why are there two ways?  Because there are two points in the process
> one can squirt the data out of config.  Let me explain (sorry for the
> length, but it is necessary to clear up the confusion):
>
> The first way is "preserve the FILE the user used to create the
> kernel" on the assumption that the FILE is what is important.

No, it is on the non-assumption that the include directive didn't exist
when the INCLUDE_CONFIG_FILE option was implemented.

> This
> will preserve the comments, but assumes that every single included
> file from that file is recoverable in a trivial manner (eg, from cvs,
> svn, p4, release ISOs, etc).

This assumption is false, so this model became just broken when the
include directive was implemented.  History:
     INCLUDE_CONFIG_FILE option: 1996
     include directive:          2001
     processed output and -C:    2007
The -C option just preserves the breakage at its 2001-2007 level.

> This model works well for the casual
> user.

Casual users presumably didn't notice the problem (until comments were lost
in 2007) since they use GENERIC as is or with minor editing, and GENERIC
doesn't use the include directive.  Non-casual users don't notice since
they know what is in their config files and don't use the
INCLUDE_CONFIG_FILE option.  (I use a combination of 3 levels of include
files and 2 levels of symlinks to handle about 6 configurations times 2
arches times 4 FreeBSD versions.)

> In this model, we're therefore able to preserve the kernel
> config by copying the file, verbatim, into the kernel.  I think this
> is the model that best fits most user's needs, since they EITHER take
> GENERIC and hack on it (in which case we preserve all that), OR they
> include GENERIC and opt in/out of things based on that default.
> Either way, this is the best way for users of releases to preserve the
> data they need to recreate the kernel without losing data that's
> important to the user, but not config (comments, spacing, order, etc).

This may be enough, but it is a historical accident that config "works"
this way.

> The other way of preserving the config file is to say "I want
> EVERYTHING applied, and I want a copy of that!"  To get that, you have
> to walk through config's internal data structures.  By the time we get
> to *THAT* point in the processing, comments are but a distant memory.

Getting output from cpp has the same problems:
- you can get fully preprocessed output fairly easily
- cpp normally discards comments early, so you can't get them easily.  I
   think gnu cpp now has an option to not discard them.
- so you usually get only the fully preprocessed output
- but sometimes you don't want so much preprocessing.  None might be best.
   I don't know of any option in gnu cpp to get none.
- the problem is harder in cpp than in config due to ifdefed out C code
   normally being completely removed, while config doesn't even have ifdef.
   Teaching cpp to preserve (only some) ifdefed out code, and presenting
   the results nicely (need to preserve the original ifdef structure)
   would be difficult.  Except it might not be too difficult for either
   cpp or config to just concatenate all the files used.

> Those are discarded early, along with the variations in spacing, blank
> lines.  By this point in the process, even the original ordering is
> long gone, as is the origin of each of the remaining items in config's
> internal data structures.  However, the resulting config file that we
> generate here is GUARANTEED to recreate the same kernel from the same
> sources modulo any silly time/date/path stuff we encode.  There's a

I checked what happens with repeating DEFAULTS.  Repeating it using
"include" results in many warnings but the same generated files.

> If space isn't an issue, we could save BOTH.  That would be a bigger
> patch, since we'd have to alter config to extract both kinds of data.
> All I did was to move the -C option from an obscure src.conf variable
> to be a full-fledged kernel option, leaving the rest of the
> infrastructure intact.  The advantage is that we cover both bases and
> could export both views as a sysctl (right now we overload the two
> different views with one sysctl).

ISTR a long discussion about the -C option when it was implemented.  Why
didn't you complain back then? :-).  I looked at my old mail about this.
I wasn't involved with the initial discussion, but imp was :-).  My mail
was after the initial commit.  I reported the following problems:
- DEFAULTS wasn't processed
- several ordering problems.  Ones that reordered directives of the form:
      options FOO=bar
      nooptions FOO
      options FOO=baz
   made the generated file unusable.
- many formatting problems in the generated file, making it a bad example.

Bruce