re0 device txcsum issue

[John-Mark Gurney]

Sean McNeil wrote this message on Thu, Sep 23, 2004 at 16:08 -0700:
> On Thu, 2004-09-23 at 15:52, John-Mark Gurney wrote:
> > Sean McNeil wrote this message on Thu, Sep 23, 2004 at 15:20 -0700:
> > > Is anyone willing to work with me to help trace down this problem?  It
> > > has been outstanding for a long time and I would dearly like it fixed. 
> > > I'm willing to help in any capacity to trace down the culprit.
> > > 
> > > To recap, on the re0 device (possibly others) running -current on an
> > > amd64 processor there are times when udp packets get improper checksum
> > > calculations with txcsum set for the interface.  This causes a deadlock
> > > in nfs as the client just contunuously requests this packet and it is
> > > rejected because of the checksum being bad.
> > 
> > I have recently been working with the re driver, so I'm somewhat familar
> > with the driver.  I haven't seen any issues, but I also don't have an
> > AMD64 system to test with.
> > 
> > Have you tried to find out if it is packet size related?  are you
> > trying to use jumbo frames?  rwatson committed netsend to the src/tools
> > tree that could help this, and I have attached udpcheck.py which is
> > a client/server script to test/verify packet sizes of difference
> > sizes.
> 
> My method of testing has been to just do an "ls -lR" from a large
> directory structure.  With txcsum set, it consistently locks up.  This
> is on clients ranging from x86 linux, BSD, a sparc solaris2, and an hppa
> HPUX machine.  If I turn off txcsum (i.e. ifconfig re0 -txcsum) I have
> never had any problems.
> 
> I tried your program with txcsum and it just hangs.  Without txcsum, 

You can provide a -v to get more detailed information on what is going
on...

Yes, the naming (in the python script) is a bit confusing since it is
from the server's poing of view..  I'm assuming you run the server side
(-s 1234) on another box and the client (server 1234) on the box w/ the
re driver?  so, the errors below means that the client sent a udp packet
and it didn't match...

> I'm not sure what the output here means, but this is what I got:

w/o txcsum you got the following errors?  This is very worrysome as
it means that you're getting udp packet corruption even w/o checksuming..

Ok, what you can do is add the line:
                                open('r.%d' % i, 'w').write(rdata)
just before the line:
                                print 'packet send mismatch at:', i, 'got:', rlen

and then run the client as:
python udpcheck.py server 1234 -s 1792 -e 1795

You will then have a set of r.179[2-5] files... They are the contents
of the packet that was received by the server...  if you could email
them to me in private mail, it might shed light on the problem..

Thanks.

> [sean at office mahi]$ python ~/Desktop/Downloads/udpcheck.py server 1234
> packet send mismatch at: 1792
> packet send mismatch at: 1793
> packet send mismatch at: 1794
> packet send mismatch at: 1795
> packet send mismatch at: 1796
> packet send mismatch at: 1797
> packet send mismatch at: 1798
> packet send mismatch at: 1799
> packet send mismatch at: 1800
> packet send mismatch at: 1801
> packet send mismatch at: 1802
> packet send mismatch at: 1803
> packet send mismatch at: 1804
> packet send mismatch at: 1805
> packet send mismatch at: 1806
> packet send mismatch at: 1807
> packet send mismatch at: 1808
> packet send mismatch at: 1809
> packet send mismatch at: 1810
> packet send mismatch at: 1811
> packet send mismatch at: 1812
> packet send mismatch at: 1813
> packet send mismatch at: 1814
> packet send mismatch at: 1815
> packet send mismatch at: 1816
> packet send mismatch at: 1817
> packet send mismatch at: 1818
> packet send mismatch at: 1819
> packet send mismatch at: 1820
> packet send mismatch at: 1821
> packet send mismatch at: 1822
> packet send mismatch at: 1823
> packet send mismatch at: 1824
> packet send mismatch at: 1825
> packet send mismatch at: 1826
> packet send mismatch at: 1827
> packet send mismatch at: 1828
> packet send mismatch at: 1829
> packet send mismatch at: 1830
> packet send mismatch at: 1831
> packet send mismatch at: 1832
> packet send mismatch at: 1833
> packet send mismatch at: 1834
> packet send mismatch at: 1835
> packet send mismatch at: 1836
> packet send mismatch at: 1837
> packet send mismatch at: 1838
> packet send mismatch at: 1839
> packet send mismatch at: 1840
> packet send mismatch at: 1841
> packet send mismatch at: 1842
> packet send mismatch at: 1843
> packet send mismatch at: 1844
> packet send mismatch at: 1845
> packet send mismatch at: 1846
> packet send mismatch at: 1847
> packet send mismatch at: 1848
> packet send mismatch at: 1849
> packet send mismatch at: 1850
> packet send mismatch at: 1851
> packet send mismatch at: 1852
> packet send mismatch at: 1853
> packet send mismatch at: 1854
> packet send mismatch at: 1855
> packet send mismatch at: 1856
> packet send mismatch at: 1857
> packet send mismatch at: 1858
> packet send mismatch at: 1859
> packet send mismatch at: 1860
> packet send mismatch at: 1861
> packet send mismatch at: 1862
> packet send mismatch at: 1863
> packet send mismatch at: 1864
> packet send mismatch at: 1865
> packet send mismatch at: 1866
> packet send mismatch at: 1867
> packet send mismatch at: 1868
> packet send mismatch at: 1869
> packet send mismatch at: 1870
> packet send mismatch at: 1871
> packet send mismatch at: 1872
> packet send mismatch at: 1873
> packet send mismatch at: 1874
> packet send mismatch at: 1875
> packet send mismatch at: 1876
> packet send mismatch at: 1877
> packet send mismatch at: 1878
> packet send mismatch at: 1879
> packet send mismatch at: 1880
> packet send mismatch at: 1881
> packet send mismatch at: 1882
> packet send mismatch at: 1883
> packet send mismatch at: 1884
> packet send mismatch at: 1885
> packet send mismatch at: 1886
> packet send mismatch at: 1887
> packet send mismatch at: 1888
> packet send mismatch at: 1889
> packet send mismatch at: 1890
> packet send mismatch at: 1891
> packet send mismatch at: 1892
> packet send mismatch at: 1893
> packet send mismatch at: 1894
> packet send mismatch at: 1895
> packet send mismatch at: 1896
> packet send mismatch at: 1897
> packet send mismatch at: 1898
> packet send mismatch at: 1899
> packet send mismatch at: 1900
> packet send mismatch at: 1901
> packet send mismatch at: 1902
> packet send mismatch at: 1903
> packet send mismatch at: 1904
> packet send mismatch at: 1905
> packet send mismatch at: 1906
> packet send mismatch at: 1907
> packet send mismatch at: 1908
> packet send mismatch at: 1909
> packet send mismatch at: 1910
> packet send mismatch at: 1911
> packet send mismatch at: 1912
> packet send mismatch at: 1913
> packet send mismatch at: 1914
> packet send mismatch at: 1915
> packet send mismatch at: 1916
> packet send mismatch at: 1917
> packet send mismatch at: 1918
> packet send mismatch at: 1919
> packet send mismatch at: 1920
> packet send mismatch at: 1921
> packet send mismatch at: 1922
> packet send mismatch at: 1923
> packet send mismatch at: 1924
> packet send mismatch at: 1925
> packet send mismatch at: 1926
> packet send mismatch at: 1927
> packet send mismatch at: 1928
> packet send mismatch at: 1929
> packet send mismatch at: 1930
> packet send mismatch at: 1931
> packet send mismatch at: 1932
> packet send mismatch at: 1933
> packet send mismatch at: 1934
> packet send mismatch at: 1935
> packet send mismatch at: 1936
> packet send mismatch at: 1937
> packet send mismatch at: 1938
> packet send mismatch at: 1939
> packet send mismatch at: 1940
> packet send mismatch at: 1941
> packet send mismatch at: 1942
> packet send mismatch at: 1943
> packet send mismatch at: 1944
> packet send mismatch at: 1945
> packet send mismatch at: 1946
> packet send mismatch at: 1947
> packet send mismatch at: 1948
> packet send mismatch at: 1949
> packet send mismatch at: 1950
> packet send mismatch at: 1951
> packet send mismatch at: 1952
> packet send mismatch at: 1953
> packet send mismatch at: 1954
> packet send mismatch at: 1955
> packet send mismatch at: 1956
> packet send mismatch at: 1957
> packet send mismatch at: 1958
> packet send mismatch at: 1959
> packet send mismatch at: 1960
> packet send mismatch at: 1961
> packet send mismatch at: 1962
> packet send mismatch at: 1963
> packet send mismatch at: 1964
> packet send mismatch at: 1965
> packet send mismatch at: 1966
> packet send mismatch at: 1967
> packet send mismatch at: 1968
> packet send mismatch at: 1969
> packet send mismatch at: 1970
> packet send mismatch at: 1971
> packet send mismatch at: 1972
> packet send mismatch at: 1973
> packet send mismatch at: 1974
> packet send mismatch at: 1975
> packet send mismatch at: 1976
> packet send mismatch at: 1977
> packet send mismatch at: 1978
> packet send mismatch at: 1979
> packet send mismatch at: 1980
> packet send mismatch at: 1981
> packet send mismatch at: 1982
> packet send mismatch at: 1983
> packet send mismatch at: 1984
> packet send mismatch at: 1985
> packet send mismatch at: 1986
> packet send mismatch at: 1987
> packet send mismatch at: 1988
> packet send mismatch at: 1989
> packet send mismatch at: 1990
> packet send mismatch at: 1991
> packet send mismatch at: 1992
> packet send mismatch at: 1993
> packet send mismatch at: 1994
> packet send mismatch at: 1995
> packet send mismatch at: 1996
> packet send mismatch at: 1997
> packet send mismatch at: 1998
> packet send mismatch at: 1999
> packet send mismatch at: 2000
> packet send mismatch at: 2001
> packet send mismatch at: 2002
> packet send mismatch at: 2003
> packet send mismatch at: 2004
> packet send mismatch at: 2005
> packet send mismatch at: 2006
> packet send mismatch at: 2007
> packet send mismatch at: 2008
> packet send mismatch at: 2009
> packet send mismatch at: 2010
> packet send mismatch at: 2011
> packet send mismatch at: 2012
> packet send mismatch at: 2013
> packet send mismatch at: 2014
> packet send mismatch at: 2015
> packet send mismatch at: 2016
> packet send mismatch at: 2017
> packet send mismatch at: 2018
> packet send mismatch at: 2019
> packet send mismatch at: 2020
> packet send mismatch at: 2021
> packet send mismatch at: 2022
> packet send mismatch at: 2023
> packet send mismatch at: 2024
> packet send mismatch at: 2025
> packet send mismatch at: 2026
> packet send mismatch at: 2027
> packet send mismatch at: 2028
> packet send mismatch at: 2029
> packet send mismatch at: 2030
> packet send mismatch at: 2031
> packet send mismatch at: 2032
> packet send mismatch at: 2033
> packet send mismatch at: 2034
> packet send mismatch at: 2035
> packet send mismatch at: 2036
> packet send mismatch at: 2037
> packet send mismatch at: 2038
> packet send mismatch at: 2039
> packet send mismatch at: 2040
> packet send mismatch at: 2041
> packet send mismatch at: 2042
> packet send mismatch at: 2043
> packet send mismatch at: 2044
> packet send mismatch at: 2045
> packet send mismatch at: 2046
> packet send mismatch at: 2047
> [('r', 1792), ('r', 1793), ('r', 1794), ('r', 1795), ('r', 1796), ('r',
> 1797), ('r', 1798), ('r', 1799), ('r', 1800), ('r', 1801), ('r', 1802),
> ('r', 1803), ('r', 1804), ('r', 1805), ('r', 1806), ('r', 1807), ('r',
> 1808), ('r', 1809), ('r', 1810), ('r', 1811), ('r', 1812), ('r', 1813),
> ('r', 1814), ('r', 1815), ('r', 1816), ('r', 1817), ('r', 1818), ('r',
> 1819), ('r', 1820), ('r', 1821), ('r', 1822), ('r', 1823), ('r', 1824),
> ('r', 1825), ('r', 1826), ('r', 1827), ('r', 1828), ('r', 1829), ('r',
> 1830), ('r', 1831), ('r', 1832), ('r', 1833), ('r', 1834), ('r', 1835),
> ('r', 1836), ('r', 1837), ('r', 1838), ('r', 1839), ('r', 1840), ('r',
> 1841), ('r', 1842), ('r', 1843), ('r', 1844), ('r', 1845), ('r', 1846),
> ('r', 1847), ('r', 1848), ('r', 1849), ('r', 1850), ('r', 1851), ('r',
> 1852), ('r', 1853), ('r', 1854), ('r', 1855), ('r', 1856), ('r', 1857),
> ('r', 1858), ('r', 1859), ('r', 1860), ('r', 1861), ('r', 1862), ('r',
> 1863), ('r', 1864), ('r', 1865), ('r', 1866), ('r', 1867), ('r', 1868),
> ('r', 1869), ('r', 1870), ('r', 1871), ('r', 1872), ('r', 1873), ('r',
> 1874), ('r', 1875), ('r', 1876), ('r', 1877), ('r', 1878), ('r', 1879),
> ('r', 1880), ('r', 1881), ('r', 1882), ('r', 1883), ('r', 1884), ('r',
> 1885), ('r', 1886), ('r', 1887), ('r', 1888), ('r', 1889), ('r', 1890),
> ('r', 1891), ('r', 1892), ('r', 1893), ('r', 1894), ('r', 1895), ('r',
> 1896), ('r', 1897), ('r', 1898), ('r', 1899), ('r', 1900), ('r', 1901),
> ('r', 1902), ('r', 1903), ('r', 1904), ('r', 1905), ('r', 1906), ('r',
> 1907), ('r', 1908), ('r', 1909), ('r', 1910), ('r', 1911), ('r', 1912),
> ('r', 1913), ('r', 1914), ('r', 1915), ('r', 1916), ('r', 1917), ('r',
> 1918), ('r', 1919), ('r', 1920), ('r', 1921), ('r', 1922), ('r', 1923),
> ('r', 1924), ('r', 1925), ('r', 1926), ('r', 1927), ('r', 1928), ('r',
> 1929), ('r', 1930), ('r', 1931), ('r', 1932), ('r', 1933), ('r', 1934),
> ('r', 1935), ('r', 1936), ('r', 1937), ('r', 1938), ('r', 1939), ('r',
> 1940), ('r', 1941), ('r', 1942), ('r', 1943), ('r', 1944), ('r', 1945),
> ('r', 1946), ('r', 1947), ('r', 1948), ('r', 1949), ('r', 1950), ('r',
> 1951), ('r', 1952), ('r', 1953), ('r', 1954), ('r', 1955), ('r', 1956),
> ('r', 1957), ('r', 1958), ('r', 1959), ('r', 1960), ('r', 1961), ('r',
> 1962), ('r', 1963), ('r', 1964), ('r', 1965), ('r', 1966), ('r', 1967),
> ('r', 1968), ('r', 1969), ('r', 1970), ('r', 1971), ('r', 1972), ('r',
> 1973), ('r', 1974), ('r', 1975), ('r', 1976), ('r', 1977), ('r', 1978),
> ('r', 1979), ('r', 1980), ('r', 1981), ('r', 1982), ('r', 1983), ('r',
> 1984), ('r', 1985), ('r', 1986), ('r', 1987), ('r', 1988), ('r', 1989),
> ('r', 1990), ('r', 1991), ('r', 1992), ('r', 1993), ('r', 1994), ('r',
> 1995), ('r', 1996), ('r', 1997), ('r', 1998), ('r', 1999), ('r', 2000),
> ('r', 2001), ('r', 2002), ('r', 2003), ('r', 2004), ('r', 2005), ('r',
> 2006), ('r', 2007), ('r', 2008), ('r', 2009), ('r', 2010), ('r', 2011),
> ('r', 2012), ('r', 2013), ('r', 2014), ('r', 2015), ('r', 2016), ('r',
> 2017), ('r', 2018), ('r', 2019), ('r', 2020), ('r', 2021), ('r', 2022),
> ('r', 2023), ('r', 2024), ('r', 2025), ('r', 2026), ('r', 2027), ('r',
> 2028), ('r', 2029), ('r', 2030), ('r', 2031), ('r', 2032), ('r', 2033),
> ('r', 2034), ('r', 2035), ('r', 2036), ('r', 2037), ('r', 2038), ('r',
> 2039), ('r', 2040), ('r', 2041), ('r', 2042), ('r', 2043), ('r', 2044),
> ('r', 2045), ('r', 2046), ('r', 2047)]
> 

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."