[Bug 227922] man -k/-f does not search man pages/mandoc.db from packages/ports (MANPATH problem)
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Wed May 2 12:39:09 UTC 2018
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227922
Bug ID: 227922
Summary: man -k/-f does not search man pages/mandoc.db from
packages/ports (MANPATH problem)
Product: Base System
Version: 11.1-STABLE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Many People
Priority: ---
Component: bin
Assignee: bugs at FreeBSD.org
Reporter: jdc at koitsu.org
It appears that man -k (a.k.a. apropos) and man -f (a.k.a. whatis) have been
broken for several years now. Rather than just give some terse explanation,
I'm going to provide proof/evidence first, followed by the fixes last.
Looks broken:
$ man -k IPC::Open3
apropos: nothing appropriate
$ man -f IPC::Open3
whatis: nothing appropriate
$ apropos -k IPC::Open3
apropos: nothing appropriate
$ whatis -f IPC::Open3
whatis: nothing appropriate
Man page does exist:
$ find /usr/local/lib/perl5 -name "IPC::Open3*" -print
/usr/local/lib/perl5/5.26/perl/man/man3/IPC::Open3.3.gz
What does manpath say?
$ manpath
/usr/share/man:/usr/local/man:/usr/share/openssl/man:/usr/local/lib/perl5/site_perl/man:/usr/local/lib/perl5/5.26/perl/man
Looks correct. Double verification that the perl pkg did the right thing:
$ cat /usr/local/etc/man.d/perl5.conf
MANPATH /usr/local/lib/perl5/site_perl/man
MANPATH /usr/local/lib/perl5/5.26/perl/man
And mandoc.db from makewhatis? Looks OK...
$ find /usr/local -name "mandoc.db" -print 2>/dev/null
/usr/local/lib/perl5/site_perl/man/mandoc.db
/usr/local/lib/perl5/5.26/perl/man/mandoc.db
/usr/local/man/mandoc.db
What if we specify the MANPATH directly? Wow, it works:
$ apropos -M /usr/local/lib/perl5/5.26/perl/man -k IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using
open3()
$ whatis -M /usr/local/lib/perl5/5.26/perl/man -f IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using
open3()
Okay, let's dig around with truss and see what's happening in apropos and
whatis, since they're C programs (man is not -- man is a shell script):
$ file /usr/bin/man /usr/bin/apropos /usr/bin/whatis
/usr/bin/man: POSIX shell script, ASCII text executable
/usr/bin/apropos: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD),
dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1
(1101515), FreeBSD-style, not stripped
/usr/bin/whatis: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD),
dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1
(1101515), FreeBSD-style, not stripped
$ truss -f -- apropos -k IPC::Open3 2>&1 | grep man.d
$ truss -f -- whatis -f IPC::Open3 2>&1 | grep man.d
$ truss -f -- whatis -f IPC::Open3 2>&1 | grep opendir
$ truss -f -- apropos -k IPC::Open3 2>&1 | grep opendir
$ truss -f -- apropos -k IPC::Open3 2>&1 | egrep 'chdir|mandoc.db'
6011: chdir("/usr/share/man") = 0 (0x0)
6011: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) = 3 (0x3)
6011: chdir("/usr/local/man") = 0 (0x0)
6011: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) = 3 (0x3)
6011: chdir("/home/jdc") = 0 (0x0)
$ truss -f -- whatis -f IPC::Open3 2>&1 | egrep 'chdir|mandoc.db'
6018: chdir("/usr/share/man") = 0 (0x0)
6018: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) = 3 (0x3)
6018: chdir("/usr/local/man") = 0 (0x0)
6018: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) = 3 (0x3)
6018: chdir("/home/jdc") = 0 (0x0)
$ truss -f -- apropos -k IPC::Open3 2>&1 | grep \.conf
5594: lstat("/etc/libmap.conf",{ mode=-rw-r--r--
,inode=1926215,size=107,blksize=32768 }) = 0 (0x0)
5594: openat(AT_FDCWD,"/etc/libmap.conf",O_RDONLY|O_CLOEXEC,00) = 3 (0x3)
5594: readlink("/etc/malloc.conf",0x7fffffffd520,1024) ERR#2 'No such file or
directory'
5594: open("/etc/man.conf",O_RDONLY,0666) ERR#2 'No such file or
directory'
$ truss -f -- whatis -f IPC::Open3 2>&1 | grep \.conf
5623: lstat("/etc/libmap.conf",{ mode=-rw-r--r--
,inode=1926215,size=107,blksize=32768 }) = 0 (0x0)
5623: openat(AT_FDCWD,"/etc/libmap.conf",O_RDONLY|O_CLOEXEC,00) = 3 (0x3)
5623: readlink("/etc/malloc.conf",0x7fffffffd520,1024) ERR#2 'No such file or
directory'
5623: open("/etc/man.conf",O_RDONLY,0666) ERR#2 'No such file or
directory'
I see no evidence that apropos or whatis understand
/usr/local/etc/man.d/*.conf. The man pages for these utilities confirm it:
$ man apropos
...
NAME
apropos, whatis – search manual page databases
...
By default, apropos searches for makewhatis(8) databases in the default
paths stipulated by man(1) and uses case-insensitive substring matching
(the = operator) over manual names and descriptions (the Nm and Nd macro
keys). Multiple terms imply pairwise -o.
...
FILES
mandoc.db name of the makewhatis(8) keyword database
/etc/man.conf default man(1) configuration file
...
What about man's own man page?
$ man man
...
-d Print extra debugging information. Repeat for increased
verbosity. Does not display the manual page.
...
FILES
/etc/man.conf
System configuration file.
/usr/local/etc/man.d/*.conf
Local configuration files.
...
So it should be working, but isn't. Let's try that debugging flag:
$ man -dddddd -k IPC::Open3
apropos: nothing appropriate
$ man -dddddd -f IPC::Open3
whatis: nothing appropriate
Not helpful.
At this stage I'm thinking: do I delve into 20KB+ of shell script for man, or
do I start looking at C code for apropos and whatis? I decided to look at man
first.
What I found:
1. Use of man -k or -f ends up calling do_man (because argv[0] is man, not
apropos or whatis like it used to be in old days).
2. do_man immediately calls man_parse_args, which uses shell getopt to parse
argv flags and so on. Within this same code: if -f is set, run do_whatis with
arguments. If -k is set, run do_apropos with arguments.
3. Looking at do_apropos we find this magical beast:
957 do_apropos() {
958 [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ]
&& \
959 exec apropos "$@"
960 search_whatis apropos "$@"
961 }
That's... interesting. What is with the stat calls? Why are we comparing
inode numbers of these two binaries, and then call exec on apropos if they
don't match?
213317 gordon do_apropos() {
285836 bapt [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i
/usr/bin/apropos) ] && \
285835 bapt exec apropos "$@"
213317 gordon search_whatis apropos "$@"
213317 gordon }
What are these two commits?
r285836 | bapt | 2015-07-24 02:20:02 -0700 (Fri, 24 Jul 2015) | 4 lines
inode should be different to actually mean mandocdb is in used
Sponsored by: gandi.net
------------------------------------------------------------------------
r285835 | bapt | 2015-07-24 02:10:03 -0700 (Fri, 24 Jul 2015) | 9 lines
Fix man -k with mandocdb
If apropos(1) and whatis(1) are not hardlinks to man(1) that means the system
is
using mandocdb, then man -k should spawn apropos(1) and/or whatis(1) directly
Reported by: kevlo
Tested by: kevlo
Sponsored by: gandi.net
So they happened over 3 years ago. There's no PR/GNATS/bug number in the
commit. Should I search mailing lists? At this point I opt to say no. The
commit message says that the intended goal was to "fix man -k", so did this
break things or did something else cause the breakage (ex.
/usr/local/etc/man.d/*.conf introduction?). My guess is the Gandi folks are
building their systems with WITHOUT_MANDOCDB=true or WITHOUT_MAN_UTILS=true
(not sure which) in /etc/src.conf. Anyway, let's look into the man.1 man page
to see if we can find out when THAT feature got added.
213317 gordon .Sh FILES
213317 gordon .Bl -tag -width indent -compact
213317 gordon .It Pa /etc/man.conf
213317 gordon System configuration file.
213317 gordon .It Pa /usr/local/etc/man.d/*.conf
213317 gordon Local configuration files.
213317 gordon .El
Long before r285835 and r285836, so /usr/local/etc/man.d/*.conf is not
responsible for this. No wonder I didn't see this breakage in stable/9.
So now that we know who to blame and understand WHY there's breakage, let's
figure out how to fix it.
apropos and whatis both comprehend the MANPATH environment variable if present.
On FreeBSD it isn't by default. FreeBSD has the manpath command, esp. manpath
-q which is used in periodic scripts like weekly/320.whatis. man's man page
says it comprehends /usr/local/etc/man.d/*.conf, so surely it has the smarts
somewhere, but it isn't getting called when using man -k or man -f. I changed
#! /bin/sh to #! /bin/sh -x and witnessed proof of this.
So back to the man shell script we go, starting from step 4...
4. In do_man(), if man -k or man -f aren't used, the do_apropos and do_whatis
aren't called. So what happens in that case? Interesting function name:
969 man_setup
5. man_setup() does some magical things relating to environment variables
within the script itself (NOTHING is export-ed). Near the end of it, I see:
628 build_manpath
629 man_setup_locale
630 man_setup_width
Hey, that build_manpath() looks like it might be relevant.
6. build_manpath() parse all kinds of fun stuff. Most importantly, it calls a
function called parse_configs, which does in fact comprehend
/usr/local/etc/man.d/*.conf and so on. Near the end, build_manpath() sets the
env var MANPATH to what should be a proper manpath.
What if we were to call build_manpath ourselves and export MANPATH? Let's try
this code in do_apropos() and then try man -k:
957 do_apropos() {
958 build_manpath
959 export MANPATH
960 [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ]
&& \
961 exec apropos "$@"
962 search_whatis apropos "$@"
963 }
$ ./man.sh -k IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using
open3()
It works! Mostly.
There is some locale-based support in man.sh but it's mainly used by manpath
(which is also this script) and the OLD whatis used to do: call search_whatis()
back in the old days.
7. search_whatis() does this right at the start:
817 search_whatis() {
...
825 build_manpath
826 build_manlocales
827 setup_pager
Very similar. build_manlocales() sets an env var called MANLOCALES that is
only used within search_whatis() itself. setup_pager() isn't relevant to man
-k output.
So while I can't get the locale bits to work for this because some code from
search_whatis() would need to be made its own function (or maybe apropos/whatis
modified to support MANLOCALE env var), what I CAN do is get man -f and man -k
working with this simple change:
$ svnlite diff
Index: man.sh
===================================================================
--- man.sh (revision 333170)
+++ man.sh (working copy)
@@ -955,6 +955,8 @@ whatis_usage() {
# Supported commands
do_apropos() {
+ build_manpath
+ export MANPATH
[ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] && \
exec apropos "$@"
search_whatis apropos "$@"
@@ -992,6 +994,8 @@ do_manpath() {
}
do_whatis() {
+ build_manpath
+ export MANPATH
[ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/whatis) ] && \
exec whatis "$@"
search_whatis whatis "$@"
Confirmation:
$ ./man.sh -k IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using
open3()
$ ./man.sh -f IPC::Open3
IPC::Open3(3) - open a process for reading, writing, and error handling using
open3()
In summary: I'm amazed that man -k/-f has essentially been broken for almost 3
years (for ports/pkgs, not base system) and nobody's complained. Like, TRULY
amazed/baffled.
So what are apropos/whatis now? Well, they're from OpenBSD, and the source is
in src/contrib/mdocml. This is code for mandoc, makewhatis, apropos, and
whatis. I just don't see a native way to make this code work with
/usr/local/etc/man.d/*.conf so man (the shell script) will have to be the
"gateway" for now.
Also: I just want to give a shoutout to Bug 223555 because it looks like
apropos/whatis may on stable/10 been the old shell script version which users
have come to rely on. Maybe the tail end of this PR will help explain that.
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-bugs
mailing list