kern/162373: VESA framebuffer memory mapping fails with EINVAL for certain modes

[Mikhail Kupchik]

>Number:         162373
>Category:       kern
>Synopsis:       VESA framebuffer memory mapping fails with EINVAL for certain modes
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 08 08:40:05 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Mikhail Kupchik
>Release:        FreeBSD-8.2-RELEASE
>Organization:
>Environment:
FreeBSD virtual-bsd32.localdomain 8.2-RELEASE FreeBSD-8.2-RELEASE #3: Sun Nov 6 00:25:20 EET 2011  root at virtual-bsd32.localdomain:/usr/obj/usr/src/sys/PRIMEEX  i386
>Description:
Memory mapping of framebuffer managed by vesa syscons driver fails with EINVAL in some video modes, while working as expected in other ones.

This is caused by erroneous range check in kernel mode code.
>How-To-Repeat:
1) Install FreeBSD 8.2 Release on VmWare (6.5.3/Win64 in my environment, but I don't think it does really matter)

2) Enable VESA framebuffer support in kernel (add the following lines to kernel config file in /usr/src/sys/i386/conf):
options SC_PIXEL_MODE
options VESA

3) Rebuild kernel, install new kernel, reboot

4) Dump list of available video modes with vidcontrol(1)
vidcontrol -i mode | more
Make sure 1024x768x32 and 800x600x32 are there. Write down mode numbers.
Under VmWare they are 321 and 320.

5) Make sure both modes work:
vidcontrol MODE_321
vidcontrol MODE_320

6) Prepare small program to test video mode mapping:
#include <sys/mman.h>
#include <stdio.h>
int main( int argc, char* argv[] ) {
    void*  p;
    if( argc >= 3 ) {
        p = mmap( 0, atoi(argv[1])*atoi(argv[2])*4, PROT_READ | PROT_WRITE,
                  MAP_SHARED, 1 /*stdout*/, 0 );
        if( !p )
            perror("mmap");
        else
            printf("mmapped OK\n");
    }
    return 0;
}
# gcc test_mmap.c -o test_mmap

7) Test memory mapping in 1024x768:
$ vidcontrol MODE_321
$ test_mmap 1024 768
mmapped OK

8) Test memory mapping in 800x600:
$ vidcontrol MODE_320
$ test_mmap 800 600
mmap: invalid argument

>Fix:
In file /usr/src/sys/dev/fb/vesa.c
in function vesa_mmap()
comment out the following lines:
      /* va_window_size == va_buffer_size/vi_planes */
      /* XXX: is this correct? */
      if (offset > adp->va_window_size - PAGE_SIZE)
          return (-1);
Then recompile kernel, install it and reboot.

It's a temporary workaround just to localize the cause of the problem.
Proper range check needs to be inserted in that place. Probably:
      if( offset >= adp->va_info.vi_buffer_size )
          return (-1);


>Release-Note:
>Audit-Trail:
>Unformatted: