bin/150727: diff on UTF-8 text files thinks they are binary - regression from 7.X

[Pete French]

>Number:         150727
>Category:       bin
>Synopsis:       diff on UTF-8 text files thinks they are binary - regression from 7.X
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Sep 19 17:20:01 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Pete French
>Release:        FreeBSD 8.1-STABLE amd64
>Organization:
>Environment:
System: FreeBSD skerry.drayhouse 8.1-STABLE FreeBSD 8.1-STABLE #0: Sat Sep 18 14:03:44 BST 2010 pete at skerry.drayhouse:/usr/obj/usr/src/sys/GENERIC amd64


>Description:
	When give two files containing valid UTF-8 text characters, diff
	simply reports "Files x and y differ". Note that this happens
	even if your locale is correctly set as UTF-8. Diff from 7.X
	does not do this. This breaks scripts which are relying on diff
	to produce the differences in the text files. In the test I tried
	the actualldiffering lines were plain ASCII - the UTF-8 characters
	were elsewhere in the file.

>How-To-Repeat:
	Create two UTF-8 files, change one, diff the pair and observe
	that diff does produce the differences.

>Fix:
	'diff -a' provides a workaround

>Release-Note:
>Audit-Trail:
>Unformatted: