kern/156545: mv could break UFS on SMP systems

[Konstantin]

>Number:         156545
>Category:       kern
>Synopsis:       mv could break UFS on SMP systems
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 21 10:20:08 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Konstantin
>Release:        8.X/7.X
>Organization:
Kasperksy Lab
>Environment:
# uname -a
FreeBSD test-host 8.2-RELEASE FreeBSD 8.2-RELEASE #1: Thu Apr  7 14:23:20 UTC 2011     root at host:/usr/obj/amd64/usr/src/sys/GENERIC  amd64

>Description:
We have found out a strange mv(1) behaviour on SMP systems with UFS. If two or more processes are trying to move the same dir from one source, it could break UFS.
It works on 7.X/8.X amd64 systems. Actually, i think, it works on all systems with SMP on all platforms.
I guess that the problem is somewhere in ./sys/ufs/ufs/ufs_vnops.c because i can't repeat it for an example on ZFS partition.

After UFS break you will see this: 

# mount
/dev/da0p2 on / (ufs, local, noatime)
devfs on /dev (devfs, local, multilabel)
/dev/da0p4 on /data (ufs, local, noatime, soft-updates)

# cd /data/test

# ls -lai
total 40
11540480 drwxr-xr-x  5 root  wheel    512 Apr 21 17:30 .
       2 drwxr-xr-x  8 root  wheel    512 Apr 21 17:29 ..
11543482 drwxr-xr-x  3 root  wheel  16384 Apr 21 17:37 dst1
11543483 drwxr-xr-x  3 root  wheel    512 Apr 21 17:37 dst2
11540481 drwxr-xr-x  2 root  wheel  16384 Apr 21 17:37 src
       4 -rw-r--r--  1 root  wheel    738 Apr 21 17:33 test.pl

# ls -lai dst1
total 20
11543482 drwxr-xr-x  3 root  wheel  16384 Apr 21 17:37 .
11540480 drwxr-xr-x  5 root  wheel    512 Apr 21 17:30 ..
11541475 drwx------  3 root  wheel    512 Apr 21 17:36 03qOT

# ls -lai dst2
total 6
11543483 drwxr-xr-x  3 root  wheel  512 Apr 21 17:37 .
11540480 drwxr-xr-x  5 root  wheel  512 Apr 21 17:30 ..
11541475 drwx------  3 root  wheel  512 Apr 21 17:36 03qOT

# rm -rf dst1/03qOT
rm: dst1/03qOT: Invalid argument

# rm -rf dst2/03qOT
rm: dst2/03qOT: Invalid argument

The only way to fix it is to unmount the partition and run the fsck on it. 

# fsck -vy /data
start /data wait fsck_ufs -y /dev/da0p4
** /dev/da0p4
** Last Mounted on /data
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
/test/dst1/03qOT IS AN EXTRANEOUS HARD LINK TO DIRECTORY /test/dst1/03qOT

REMOVE? yes

BAD INODE NUMBER FOR '..'  I=11541475  OWNER=root MODE=40700
SIZE=512 MTIME=Apr 21 17:36 2011
DIR=/test/dst2/03qOT

UNEXPECTED SOFT UPDATE INCONSISTENCY

FIX? yes

** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
LINK COUNT DIR I=11541475  OWNER=root MODE=40700
SIZE=512 MTIME=Apr 21 17:36 2011  COUNT 3 SHOULD BE 2
ADJUST? yes

LINK COUNT DIR I=11543482  OWNER=root MODE=40755
SIZE=16384 MTIME=Apr 21 17:37 2011  COUNT 3 SHOULD BE 2
ADJUST? yes

** Phase 5 - Check Cyl groups
7757 files, 87291 used, 134202352 free (256 frags, 16775262 blocks, 0.0% fragmentation)

***** FILE SYSTEM IS CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****
>How-To-Repeat:
You can use this script to repeat the problem:

#!/usr/local/bin/perl
use strict;

my $iterations = 50;
my $dirs = 1000;

mkdir 'src';

for(1..$iterations)
{
  print "Iterations $_\n";
  do_test();
}

sub do_test
{
  print "Creating $dirs sample directories...\n";

  for(1..$dirs)
  {
    my $dirname = `mktemp -d src/XXXXX`;
    chomp $dirname;
    system("touch $dirname/data && touch $dirname/meta");
    print "$_ ";
    }

    print "done\n";

    my ($pid1, $pid2) = (do_job('src', 'dst1'), do_job('src', 'dst2'));
    waitpid($pid1, 0);
    waitpid($pid2, 0);
}

sub do_job
{
  my ($src, $dst) = @_;
  
  my $pid = fork();
  return $pid if $pid != 0;
  
  system("mkdir -p $dst && ls $src | xargs -I _ mv -n $src/_ $dst");
  system("rm -rf $dst/*") == 0 or die;
  
  exit(0);
}
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: