From nobody Fri Feb 03 16:31:55 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7h2Y3V6qz3kV5j for ; Fri, 3 Feb 2023 16:29:57 +0000 (UTC) (envelope-from eivinde@terraplane.org) Received: from smtp.domeneshop.no (smtp.domeneshop.no [IPv6:2a01:5b40:0:3006::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7h2Y1Xz6z3qx2 for ; Fri, 3 Feb 2023 16:29:57 +0000 (UTC) (envelope-from eivinde@terraplane.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=terraplane.org; s=ds202212; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=S7Hz8ARJ9fTDxV1Qe+3zA3E7qEgU3dPAOrCx8McYTgQ=; b=IJ7dQMjQKskn8cCzZ7+s/85mzO BDshtjEKAdo8rEJyyMqYQEdgN2p10fBGqJFhtaFijrz3Nl7+O7Sp1t0QImBLodKAxwt+eBt8MW8jn 2pOGwLHy8sPovY/kMIBa78GISH4ekgiJ21ZJazN9dBNq5hM/iBv8tH/Is9e57WWOV6T3urciNTHHt m1KyQWcujmW9IJsDpKzYYCbFCGVnomLzbV4RPqBYBFQetlvWQievDdSCL+mAxq4vUJ7MfiSsKgmQY onLp0BqEOn0IJU3NMS1IhoE/HBqcqbgNOTpgQkly1stAj/b2V9JX0n2zP2lwC/n3dCPdpx+1GYLvm P3sLR1fw==; Received: from ti0027q160-0136.bb.online.no ([37.200.21.137]:27596 helo=elg.hjerdalen.lokalnett) by smtp.domeneshop.no with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pNywj-004ERR-EW; Fri, 03 Feb 2023 17:29:54 +0100 Date: Fri, 3 Feb 2023 17:31:55 +0100 From: Eivind Nicolay Evensen To: Tomoaki AOKI Cc: stable@freebsd.org Subject: Re: Grep with non-ascii Message-ID: <20230203173155.179902a4@elg.hjerdalen.lokalnett> In-Reply-To: <20230204010605.4874609f80eed28543407807@dec.sakura.ne.jp> References: <20230203110642.70e4a076@elg.hjerdalen.lokalnett> <819a4336-9689-bdbe-a90d-8f1d7b842662@grosbein.net> <20230203151853.02732bd6@elg.hjerdalen.lokalnett> <20230204010605.4874609f80eed28543407807@dec.sakura.ne.jp> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4P7h2Y1Xz6z3qx2 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:12996, ipnet:2a01:5b40::/48, country:NO] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Den Sat, 4 Feb 2023 01:06:05 +0900 skrev Tomoaki AOKI : > On Fri, 3 Feb 2023 15:18:53 +0100 > Eivind Nicolay Evensen wrote: > > > Den Fri, 3 Feb 2023 19:12:32 +0700 > > skrev Eugene Grosbein : > > > > > 03.02.2023 17:06, Eivind Nicolay Evensen wrote: > > > > Hello. > > > > > > > > I just noticed this today: > > > > > > > > elg!ene[~]> printf "bø\nhei\nøl\n" | grep ø > > > > grep: trailing backslash (\) > > > > elg!ene[~]> echo $LC_CTYPE $LANG > > > > nb_NO.ISO8859-1 nb_NO.ISO8859-1 > > > > > > > > While I have the result I envisioned with gnugrep: > > > > > > > > elg!ene[~]> printf "bø\nhei\nøl\n" | ggrep ø > > > > bø > > > > øl > > > > > > > > Also, on OpenIndiana, linux and Netbsd, grep gives the proper > > > > result. > > > > > > > > Is lib/libc/regex the right place to look into this if I > > > > find the time, or does anybody know this enough to know the > > > > problem? > > > > > > Try single quotes instead of double quotes. > > > And pleace specify system version and shell name, and shell > > > version if its not in base system. > > > > This is > > elg!ene[~]> uname -a > > FreeBSD elg.hjerdalen.lokalnett 13.2-PRERELEASE FreeBSD > > 13.2-PRERELEASE #1: Tue Jan 31 11:23:29 CET 2023 > > ene@elg.hjerdalen.lokalnett:/usr/obj/usr/src/amd64.amd64/sys/ENE-spurv > > amd64 > > > > Using the tcsh that comes with it. But I don't think the quotes > > matter much because of this: > > > > elg!ene[~]> grep ø > > grep: trailing backslash (\) > > > > The output was more just to have something to look for, like > > with ggrep but anyway: > > > > elg!ene[~]> printf 'bø\nhei\nøl\n' |grep ø > > grep: trailing backslash (\) > > > > And obviously: > > > > elg!ene[~]> printf 'bø\nhei\nøl\n' > > bø > > hei > > øl > > > > And it seems to be the same for any 8859-1 character not part > > of ascii: > > > > elg!ene[~]> grep ä > > grep: trailing backslash (\) > > elg!ene[~]> grep ß > > grep: trailing backslash (\) > > elg!ene[~]> grep ç > > grep: trailing backslash (\) > > > > -- > > Eivind Nicolay Evensen > > I recalled very, very old problem on Japanese characters. > Does the characters you mentioned include 0x5c in nb_NO.ISO8859-1 > charset? > > In dirty, ugly DOS era, Shift-JIS (CP932) was the mainstream in Japan. > In this charset, some 2bytes kanji characters have 0x5c in its second > byte. > > This caused imported, non-Japanese-aware softwares mis-handle Japanese > texts, and the workaround was to add excessive 0x5c after problematic > characters. :-( > > For example, ?? in Shift-JIS bytestream was 0x95 0x5c 0x8e 0xa6, and > as 0x5c was usually considered as backslash, escape character, it was > modified to 0x95 0x8e 0xa6 in non-Japanese softwares. > As this mis-conversion often happened recussively, the required > numbers of excessive 0x5c varied, varied and varied!!!!! Crazily. > > If this is the case like above, the only solution is to move to > character set containing ALL characters all over the world. > > AFAIK, the only candidates are only two, TRON code [1] and Unicode > (UCS, ISO/IEC 10646) [2]. And TRON code is very rarely used, actual > candidate would be Unicode only. > Note that Unicode is usually encoded to any of UTF-8, UTF-16 or UTF-32 > for data transfer (sometimes raw UCS-2?). > > > [1] https://en.wikipedia.org/wiki/TRON_(encoding) > [2] https://en.wikipedia.org/wiki/Unicode > > P.S. > On UTF-8, character ø was encoded to UTF-8: 0xC3 0xB8. So it should be > OK. In 8859-1, "ø" is: elg!ene[~]> printf ø |hexdump -C 00000000 f8 |ø| 00000001 so this does not seem to be the problem here. And all those characters I tried are one-byte (all 8859-1 are): elg!ene[~]> printf "äßç" |hexdump -C 00000000 e4 df e7 |äßç| 00000003 So I do not believe this is the same problem. I did, however, find it interesting that multi-byte character sets may have been in use longer than I imagined. -- Eivind Nicolay Evensen