"Unprintable" 8-bit characters

[Robert Bonomi]

On Tue, 8 Nov 2011 18:42:36 -0600, "Conrad J. Sabatier" wrote:
>
> I've been trying to understand what the deal is with regards to the
> displaying of the "extended" 8-bit character set, i.e., 8-bit characters
> with the MSB set.

Quite simply Unix dates from the days where the 8th bit was used as a 'parity'
bit.  Allowing detection of *all* single-bit errors -- especially over the
notoriously un-reliable connections known as 'serial ports'.
>
> More specifically, I'm trying to figure out how to get the "ls" command
> to properly display filenames containing characters in this extended
> set.  I have some MP3 files, for instance, whose names contain certain
> European characters, such as the lowercase "u" with umlaut (code 0xfc
> in the Latin set, according to gucharmap), that I just can't get ls to
> display properly.  These characters seem to be considered by ls as
> "unprintable", and the best I've been able to produce in the ls
> output is backslash interpretations of the characters using either the
> -B or -b options, otherwise the default "?" is displayed in their place.
>
> The strange thing is that these characters will display just fine in
> xterm, gnome-terminal, etc.  I can copy and paste them from the
> gucharmap utility into a shell command line or other application, and
> they appear as they should, but ls simply refuses to display them.  I
> can print them using the printf command, even bash's builtin echo seems
> to have no problem with them.  Only ls appears to have this problem.
>
> I've experimented with using various locales, using the LC_*
> variables, as well as the LANG variable (as documented in the
> environment section of the ls man page), all to no avail.

Obviously you never read as far as the '-w' switch.  <grin>

> Is this an inherent limitation of ls, 

It is -not- a limitation; rather it is a _desired_ behavior -- so that 
one can _tell_ where there is an 'unprintable' character (like \r, or\b)
in a filename.  There are *good*reasons*(TM) why -q is the default behavior
for 'terminal' output.

>                                       or is there some workaround or
> other solution?  Do we need a new en_*.UTF-16 locale?  Should we
> consider extending the ls command to handle these characters?

There _are_ "improved" versions of ls that do understand the 'locale'
environment variables -- but those programs introduce a whole bunch of
*other* 'not necessarily desired' behaviors -- like sorting upper-case and
lower-case letters as 'equals', rather than regarding any upper-case as 
sorting before any lowercase.

>                                                                Or is
> there just something about all of this that I'm just not "getting"?
>
> As an additional note, I notice that in the text console, this same
> character code (0xfc) produces an entirely different character (a
> lowercase n in a raised position, as for the exponent in a mathematical
> expression).  Is there, in fact, no standardization re: the
> representation of these "high bit" characters?

"The nice thing about standards is that there are so many to choose from"
applies.  WITH A VENGANCE!!

There are at least FIFTEEN different sets of glyphs for the 'high bit set'
byte codes *JUST* for the 'iso-8859' base charset.  Plus 'utf-8'  And not 
counting the various bastardiztions (e.g. 'CP-1252', etc.) that Microsoft 
has introduced.

> Thanks to anyone who can help clear up this long-standing mystery for
> me.

<R>eading <t>he <f>ine <m>anpage -- with particular attention to the '-q'
and '-w' options should provie some enlightenment.