From matthias.apitz at oclc.org  Tue May  6 17:48:44 2008
From: matthias.apitz at oclc.org (Matthias Apitz)
Date: Tue May  6 17:48:48 2008
Subject: porting "nozomi" driver (Option N.V. GlobeTrotter 3G+ UMTS
	datacard) to FreeBSD 7.0R
Message-ID: <20080506174359.GA2814@rebelion.Sisis.de>


Hello,

I'm on the way to port the above mentioned driver to FreeBSD 7.0-REL;
the work is based on the Linux driver of this card and of some help I've
got in frebsd-mobile, see thread:

http://groups.google.com/group/muc.lists.freebsd.mobile/browse_thread/thread/6e4b18d5d292b0a7/0b1f42404c982b8e

the current state of the work is:

- driver attaches fine to the card on insert;
- devices come up as /dev/cuaN0...4;
- serial communication with, for example, kermit to /dev/cuaN0 is fine;
- PPPD can chat with AT-cmd's into UMTS network and sign on;
- LCP layer is fine, IP comes up, routing, etc;
- with real TCP traffic the communication gets stuck, for example with
  SSH from the PPPD-laptop to some server in my network;

I've watched with TCPDUMP the interface ppp0 of the laptop and the eth0 of
the server at the same time, here are both traces; you will see that the PPP
outbound package (marked with ^^^^^^^^^^^^^^^^^) does not reach the server
(and perhaps does not go out to UMTS) and the server is sending its
SSH-good-morning string again and again because of missing ACK:


TCPDUMP on PPPD site:

14:03:54.890785 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: S 3468691149:3468691149(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3018001 0>
14:03:54.993387 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: S 1963911431:1963911431(0) ack 3468691150 win 5792 <mss 1420,sackOK,timestamp 1008400094 3018001,nop,wscale 2>
14:03:54.993445 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: . ack 1 win 8272 <nop,nop,timestamp 3018104 1008400094>
14:03:55.142728 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008400244 3018104>
14:03:55.142976 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: P 1:40(39) ack 24 win 8272 <nop,nop,timestamp 3018254 1008400244>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
14:03:55.513520 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008400610 3018104>
14:03:55.513569 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: P 40:792(752) ack 24 win 8272 <nop,nop,timestamp 3018624 1008400610>
14:03:56.243482 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008401342 3018104>
14:03:56.243513 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: . ack 24 win 8272 <nop,nop,timestamp 3019354 1008401342>
14:03:57.460382 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: P 1:792(791) ack 24 win 8272 <nop,nop,timestamp 3020572 1008401342>
14:03:57.714044 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008402806 3018104>
14:03:57.714125 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: . ack 24 win 8272 <nop,nop,timestamp 3020825 1008402806>
14:04:00.643527 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008405734 3018104>
14:04:00.643564 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: . ack 24 win 8272 <nop,nop,timestamp 3023755 1008405734>
14:04:01.895706 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: P 1:792(791) ack 24 win 8272 <nop,nop,timestamp 3025008 1008405734>
14:04:03.594932 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: F 792:792(0) ack 24 win 8272 <nop,nop,timestamp 3026707 1008405734>
14:04:06.493335 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008411590 3018104>
14:04:06.493370 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: F 792:792(0) ack 24 win 8272 <nop,nop,timestamp 3029606 1008411590>



TCPDUMP on SSHD site (193.31.xxx.xxx):

15:03:21.929767 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: S 3468691149:3468691149(0) win 65535 <mss 1420,nop,wscale 3,sackOK,timestamp 3018001 0>
15:03:21.930193 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: S 1963911431:1963911431(0) ack 3468691150 win 5792 <mss 1460,sackOK,timestamp 1008400094 3018001,nop,wscale 2>
15:03:22.050901 IP 77.25.44.116.58392 > 193.31.xxx.xxx.22: . ack 1 win 8272 <nop,nop,timestamp 3018104 1008400094>
15:03:22.079679 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008400244 3018104>
15:03:22.444768 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008400610 3018104>
15:03:23.176653 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008401342 3018104>
15:03:24.640438 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008402806 3018104>
15:03:27.567996 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008405734 3018104>
15:03:33.423109 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008411590 3018104>
15:03:45.133330 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008423302 3018104>
15:04:08.553772 IP 193.31.xxx.xxx.22 > 77.25.44.116.58392: P 1:24(23) ack 1 win 1448 <nop,nop,timestamp 1008446726 3018104>

I've instructed the source with printf's in the sending code, which
looks like this: 

        buf = malloc(sizeof(struct fifo_buf), M_DEVBUF, M_NOWAIT);
        memcpy(buf->data, data, cnt < sizeof(buf->data) ? cnt : sizeof(buf->data));
        buf->size = cnt;
        printf("nzdebug: nzstart() -> STAILQ_INSERT_TAIL() of %d bytes\n", cnt); 
        STAILQ_INSERT_TAIL(&fifo_head, buf, fifo_bufs);
        ndflush(&tty->t_outq, cnt);
        intr_ul(sc, pidx, ENABLE);

and the resulting log is this (I've marked what the output contains):

May  6 14:03:21 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 1 bytes   <-- PPPD chat to network
May  6 14:03:21 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 1 bytes   <-- PPPD chat to network
May  6 14:03:21 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 1 bytes   <-- PPPD chat to network
May  6 14:03:21 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 1 bytes   <-- PPPD chat to network
May  6 14:03:21 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 1 bytes   <-- PPPD chat to network
May  6 14:03:21 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 1 bytes   <-- PPPD chat to network
May  6 14:03:27 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 45 bytes  <-- LCP
May  6 14:03:27 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 55 bytes  <-- LCP
May  6 14:03:27 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:27 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 41 bytes  <-- LCP
May  6 14:03:27 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:28 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:29 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:30 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:31 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:32 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:32 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 16 bytes  <-- LCP
May  6 14:03:32 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 10 bytes  <-- LCP
May  6 14:03:54 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 65 bytes  <-- TCP SYN
May  6 14:03:55 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 57 bytes  <-- TCP ACK
May  6 14:03:55 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 96 bytes  <-- TCP 39 byte which does not go out
May  6 14:03:55 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 108 bytes
May  6 14:03:55 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 108 bytes
May  6 14:03:56 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 108 bytes
May  6 14:03:56 Rebelion kernel: nzdebug: STAILQ_INSERT_TAIL() inserting 108 bytes

the full source and its Makefile are here:

http://www.unixarea.de/nozomi/nozomi.c
http://www.unixarea.de/nozomi/Makefile

Any idea about what could be wrong is pretty much welcome; thanks in advance;
The source is GPL'ed and can not be taken in this form into FreeBSD;

	matthias

-- 
Matthias Apitz
Manager Technical Support - OCLC GmbH
Gruenwalder Weg 28g - 82041 Oberhaching - Germany
t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211
e <matthias.apitz@oclc.org> - w http://www.oclc.org/ http://www.UnixArea.de/
b http://gurucubano.blogspot.com/
Don't top-post, read RFC1855 http://www.faqs.org/rfcs/rfc1855.html
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on Usenet and in e-mail?
From guru at Sisis.de  Fri May  9 12:47:27 2008
From: guru at Sisis.de (Matthias Apitz)
Date: Fri May  9 12:47:31 2008
Subject: porting "nozomi" driver (Option N.V. GlobeTrotter 3G+ UMTS
	datacard) to FreeBSD 7.0R
In-Reply-To: <20080506174359.GA2814@rebelion.Sisis.de>
References: <20080506174359.GA2814@rebelion.Sisis.de>
Message-ID: <20080509123521.GA11366@rebelion.Sisis.de>

El d?a Tuesday, May 06, 2008 a las 07:43:59PM +0200, Matthias Apitz escribi?:

> 
> Hello,
> 
> I'm on the way to port the above mentioned driver to FreeBSD 7.0-REL;
> the work is based on the Linux driver of this card and of some help I've
> got in frebsd-mobile, see thread:
> 
> http://groups.google.com/group/muc.lists.freebsd.mobile/browse_thread/thread/6e4b18d5d292b0a7/0b1f42404c982b8e
> 
> the current state of the work is:
> 
> - driver attaches fine to the card on insert;
> - devices come up as /dev/cuaN0...4;
> - serial communication with, for example, kermit to /dev/cuaN0 is fine;
> - PPPD can chat with AT-cmd's into UMTS network and sign on;
> - LCP layer is fine, IP comes up, routing, etc;
> - with real TCP traffic the communication gets stuck, for example with
>   SSH from the PPPD-laptop to some server in my network;
> 
> I've watched with TCPDUMP the interface ppp0 of the laptop and the eth0 of
> the server at the same time, ...

Hello,

without going into the details of the TCPDUMP, it shows that the
three-way-handshake (SYN-SYN-ACK) is fine, the 1st data pkg from SSH
server arrives, but the answer from PPPD with the ACK is only to be seen
in the TCPDUMP on the PPPD site and does not arrive at server;

I've instructed the driver to log a lot of stuff, and as well the
contents of the buffers send to the card ... and compared them against
the PPP specs I have handy in the TCP/IP Illustraded, Volume I, from
Stevens;

for the LCP packages the frame look like this:

7e ff 03 c0 21 09 00 00 08 b7 b4 30 0b 54 b9 7e

7e    -- flag
ff    -- addr, always ff
03    -- control, always 03
c0 21 -- protocol, here LCP

and the frame ends with 7e, ok;

for the IP datagram containing the 1st (outgoing) SYN it looks like:

7e 21 45 00 00 3c 00 a7 40 00 40 06 90 5d 4d 19 8f b7 c1 1f 0b c8 cc
fe 00 16 16 b9 37 7d 5d 00 00 00 00 a0 02 ff ff ef 58 00 00 02 04 05
b4 01 03 03 03 04 02 08 0a 00 1a 93 8e 00 00 00 00 79 f2 7e


7e    -- flag
21    -- ??? (could be part of 0021 indicating IP datagram)
45 00 00 3c 00 a7 40 00 40 06 90 5d 4d 19 8f b7 c1 1f 0b c8 -- IP header already
                                    4d 19 8f b7             -- src addr 77.25.143.183
				                c1 1f 0b c8 -- dst addr

as well the ACK of SYN-SYN-ACK goes out like this:

7e 21 45 00 00 34 00 aa 40 00 40 06 90 62 4d 19 8f b7 c1 1f 0b c8 ...

and also the 1st real data (containing the protocol string of the SSH client
looks like this:

7e 21 45 00 00 5b 00 af 40 00 40 06 90 36 4d 19 8f b7 c1 1f 0b c8 cc
fe 00 16 16 b9 37 7d 5e c5 c8 fd cc 80 18 20 50 04 fd 00 00 01 01 08
0a 00 1a 95 9e 4b 36 fd 6c 53 53 48 2d 32 2e 30 2d 4f 70 65 6e 53 53
48 5f 34 2e 35 70 31 20 46 72 65 65 42 53 44 2d 32 30 30 36 31 31 31
30 0a f1 62 7e

shouldn't they start with

7e ff 03 00 21 ...

Stevens explains further more that client and server could handshake to
omit the constant flag (7e) and adress field (ff) and reduce the
protocol field (0021) to one byte (21), but in the above packages 'flag' is there
while 'addr' and 'control' are missing?

any PPP expert here to explain this? could this be the reason that the
connection gets stuck?

Thx

	matthias


-- 
Matthias Apitz
Manager Technical Support - OCLC GmbH
Gruenwalder Weg 28g - 82041 Oberhaching - Germany
t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211
e <matthias.apitz@oclc.org> - w http://www.oclc.org/ http://www.UnixArea.de/
b http://gurucubano.blogspot.com/
Don't top-post, read RFC1855 http://www.faqs.org/rfcs/rfc1855.html
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on Usenet and in e-mail?
From smithi at nimnet.asn.au  Fri May  9 15:27:08 2008
From: smithi at nimnet.asn.au (Ian Smith)
Date: Fri May  9 15:27:11 2008
Subject: porting "nozomi" driver (Option N.V. GlobeTrotter 3G+ UMTS
	datacard) to FreeBSD 7.0R
In-Reply-To: <20080509123521.GA11366@rebelion.Sisis.de>
Message-ID: <Pine.BSF.3.96.1080510002225.18341A-100000@gaia.nimnet.asn.au>

On Fri, 9 May 2008, Matthias Apitz wrote:
 > El d?a Tuesday, May 06, 2008 a las 07:43:59PM +0200, Matthias Apitz escribi?:
 > 
 > > 
 > > Hello,
 > > 
 > > I'm on the way to port the above mentioned driver to FreeBSD 7.0-REL;
 > > the work is based on the Linux driver of this card and of some help I've
 > > got in frebsd-mobile, see thread:
 > > 
 > > http://groups.google.com/group/muc.lists.freebsd.mobile/browse_thread/thread/6e4b18d5d292b0a7/0b1f42404c982b8e
 > > 
 > > the current state of the work is:
 > > 
 > > - driver attaches fine to the card on insert;
 > > - devices come up as /dev/cuaN0...4;
 > > - serial communication with, for example, kermit to /dev/cuaN0 is fine;
 > > - PPPD can chat with AT-cmd's into UMTS network and sign on;
 > > - LCP layer is fine, IP comes up, routing, etc;
 > > - with real TCP traffic the communication gets stuck, for example with
 > >   SSH from the PPPD-laptop to some server in my network;
 > > 
 > > I've watched with TCPDUMP the interface ppp0 of the laptop and the eth0 of
 > > the server at the same time, ...
 > 
 > Hello,
 > 
 > without going into the details of the TCPDUMP, it shows that the
 > three-way-handshake (SYN-SYN-ACK) is fine, the 1st data pkg from SSH
 > server arrives, but the answer from PPPD with the ACK is only to be seen
 > in the TCPDUMP on the PPPD site and does not arrive at server;
 > 
 > I've instructed the driver to log a lot of stuff, and as well the
 > contents of the buffers send to the card ... and compared them against
 > the PPP specs I have handy in the TCP/IP Illustraded, Volume I, from
 > Stevens;
 > 
 > for the LCP packages the frame look like this:
 > 
 > 7e ff 03 c0 21 09 00 00 08 b7 b4 30 0b 54 b9 7e
 > 
 > 7e    -- flag
 > ff    -- addr, always ff
 > 03    -- control, always 03
 > c0 21 -- protocol, here LCP
 > 
 > and the frame ends with 7e, ok;
 > 
 > for the IP datagram containing the 1st (outgoing) SYN it looks like:
 > 
 > 7e 21 45 00 00 3c 00 a7 40 00 40 06 90 5d 4d 19 8f b7 c1 1f 0b c8 cc
 > fe 00 16 16 b9 37 7d 5d 00 00 00 00 a0 02 ff ff ef 58 00 00 02 04 05
 > b4 01 03 03 03 04 02 08 0a 00 1a 93 8e 00 00 00 00 79 f2 7e
 > 
 > 
 > 7e    -- flag
 > 21    -- ??? (could be part of 0021 indicating IP datagram)
 > 45 00 00 3c 00 a7 40 00 40 06 90 5d 4d 19 8f b7 c1 1f 0b c8 -- IP header already
 >                                     4d 19 8f b7             -- src addr 77.25.143.183
 > 				                c1 1f 0b c8 -- dst addr
 > 
 > as well the ACK of SYN-SYN-ACK goes out like this:
 > 
 > 7e 21 45 00 00 34 00 aa 40 00 40 06 90 62 4d 19 8f b7 c1 1f 0b c8 ...
 > 
 > and also the 1st real data (containing the protocol string of the SSH client
 > looks like this:
 > 
 > 7e 21 45 00 00 5b 00 af 40 00 40 06 90 36 4d 19 8f b7 c1 1f 0b c8 cc
 > fe 00 16 16 b9 37 7d 5e c5 c8 fd cc 80 18 20 50 04 fd 00 00 01 01 08
 > 0a 00 1a 95 9e 4b 36 fd 6c 53 53 48 2d 32 2e 30 2d 4f 70 65 6e 53 53
 > 48 5f 34 2e 35 70 31 20 46 72 65 65 42 53 44 2d 32 30 30 36 31 31 31
 > 30 0a f1 62 7e
 > 
 > shouldn't they start with
 > 
 > 7e ff 03 00 21 ...
 > 
 > Stevens explains further more that client and server could handshake to
 > omit the constant flag (7e) and adress field (ff) and reduce the
 > protocol field (0021) to one byte (21), but in the above packages 'flag' is there
 > while 'addr' and 'control' are missing?
 > 
 > any PPP expert here to explain this? could this be the reason that the
 > connection gets stuck?

Probably not the sort of help you wanted, and I'm no PPP expert, but
having run ppp(8) in and out for ten years and more recently mpd(8) 
outbound, I have to ask, does this gig depend on using pppd?

Just that in that time I've seen very few people using pppd except some
people just coming over from linux, or those hoping to use kde's dialer,
and have noticed little success reported with apparent pppd bugs.

Almost invariably people seem better advised to use ppp or more lately
mpd instead, if only because quite a few people here can resolve config
and usage problems with either, whereas pppd appears to have been all
but deprecated for some years - but perhaps I do it an injustice.

HTH, Ian  (subscribed only to -net)

From guru at Sisis.de  Sat May 10 16:38:07 2008
From: guru at Sisis.de (Matthias Apitz)
Date: Sat May 10 16:38:22 2008
Subject: porting "nozomi" driver (Option N.V. GlobeTrotter 3G+ UMTS
	datacard) to FreeBSD 7.0R
In-Reply-To: <Pine.BSF.3.96.1080510002225.18341A-100000@gaia.nimnet.asn.au>
References: <20080509123521.GA11366@rebelion.Sisis.de>
	<Pine.BSF.3.96.1080510002225.18341A-100000@gaia.nimnet.asn.au>
Message-ID: <20080510163417.GA17718@rebelion.Sisis.de>

El d?a Saturday, May 10, 2008 a las 12:49:02AM +1000, Ian Smith escribi?:

>  > Stevens explains further more that client and server could handshake to
>  > omit the constant flag (7e) and adress field (ff) and reduce the
>  > protocol field (0021) to one byte (21), but in the above packages 'flag' is there
>  > while 'addr' and 'control' are missing?
>  > 
>  > any PPP expert here to explain this? could this be the reason that the
>  > connection gets stuck?
> 
> Probably not the sort of help you wanted, and I'm no PPP expert, but
> having run ppp(8) in and out for ten years and more recently mpd(8) 
> outbound, I have to ask, does this gig depend on using pppd?
	...

I've checked a short moment mpd5(8) (installed it from the ports and
checked the manual about configuration); it seems that it would me take
some time to get the CHAT part working;

because, on the other hand, my installed pppd(8) works just fine,
with older UMTS cards, I don't think that this is the problem;

I've spent some more hours as well on debugging and now I have it clear
a) what the problem is with the TCP packages
b) why LCP works just nicely
c) why CHAT works as well nicely
d) why telneting to the ECHO port works also if (and only if) you enter
   only a few characters on the line, say up to five

the problem is the size of the buffer coming down from user space in the
nzstart() function:

        ...
        data = tty->t_outq.c_cf;
        cbp = (struct cblock *) ((intptr_t) tty->t_outq.c_cf & ~CROUND);
        cnt = min((char *) (cbp+1) - tty->t_outq.c_cf, tty->t_outq.c_cc);

        if(cnt == 0)
                goto out;

        buf = malloc(sizeof(struct fifo_buf), M_DEVBUF, M_NOWAIT);
        memcpy(buf->data, data, cnt < sizeof(buf->data) ? cnt : sizeof(buf->data));
        buf->size = cnt;

        printf("nzdebug: nzstart() -> STAILQ_INSERT_TAIL() of %d bytes\n", cnt);
        STAILQ_INSERT_TAIL(&fifo_head, buf, fifo_bufs);
        ndflush(&tty->t_outq, cnt);
        intr_ul(sc, pidx, ENABLE);
        ...

I saw frames upto 108 bytes length;

and later send_data(), which puts the data into the card's buffer, picks
the data up like this:

        buf = STAILQ_FIRST(&fifo_head);
        size = buf->size;

        memcpy(sc->send_buf, buf->data, ul_size < SEND_BUF_MAX ?  ul_size : SEND_BUF_MAX );

        STAILQ_REMOVE_HEAD(&fifo_head, fifo_bufs);
        free(buf, M_DEVBUF);

        port->tx_data += size;

        /* Write length + data */
        bus_write_4(sc->res, ul_offs, size);
        bus_write_region_4(sc->res, ul_offs + 4, (u_int32_t *)sc->send_buf, size);

SEND_BUF_MAX is 1024 but the 'ul_size' (size of the up link part of the
card) is only 68!! that's why parts of the PPP frames are just thrown
away if the frame size is bigger than 64 (4 bytes are needed for the
size); I got to know this comparing the hex dumps of the buffer in the
nzstart() and send_data() parts;

at the moment I have no idea how to fix this :-((
I've put the driver here:

http://www.unixarea.de/nozomi/nozomi.c

if someone wants to have a look and give me some hint; thanks in advance;

it's Saturday evening and sunny, time for go out...

	matthias

From nike_d at cytexbg.com  Sat May 10 17:35:46 2008
From: nike_d at cytexbg.com (Niki Denev)
Date: Sat May 10 17:35:50 2008
Subject: porting "nozomi" driver (Option N.V. GlobeTrotter 3G+ UMTS
	datacard) to FreeBSD 7.0R
In-Reply-To: <20080510163417.GA17718@rebelion.Sisis.de>
References: <20080509123521.GA11366@rebelion.Sisis.de>
	<Pine.BSF.3.96.1080510002225.18341A-100000@gaia.nimnet.asn.au>
	<20080510163417.GA17718@rebelion.Sisis.de>
Message-ID: <2e77fc10805101008ybafe7b3g8b9210cb8d2955d@mail.gmail.com>

On Sat, May 10, 2008 at 12:34 PM, Matthias Apitz <guru@sisis.de> wrote:
> El d?a Saturday, May 10, 2008 a las 12:49:02AM +1000, Ian Smith escribi?:
>
>>  > Stevens explains further more that client and server could handshake to
>>  > omit the constant flag (7e) and adress field (ff) and reduce the
>>  > protocol field (0021) to one byte (21), but in the above packages 'flag' is there
>>  > while 'addr' and 'control' are missing?
>>  >
>>  > any PPP expert here to explain this? could this be the reason that the
>>  > connection gets stuck?
>>
>> Probably not the sort of help you wanted, and I'm no PPP expert, but
>> having run ppp(8) in and out for ten years and more recently mpd(8)
>> outbound, I have to ask, does this gig depend on using pppd?
>        ...
>
> I've checked a short moment mpd5(8) (installed it from the ports and
> checked the manual about configuration); it seems that it would me take
> some time to get the CHAT part working;
>
> because, on the other hand, my installed pppd(8) works just fine,
> with older UMTS cards, I don't think that this is the problem;
>
> I've spent some more hours as well on debugging and now I have it clear
> a) what the problem is with the TCP packages
> b) why LCP works just nicely
> c) why CHAT works as well nicely
> d) why telneting to the ECHO port works also if (and only if) you enter
>   only a few characters on the line, say up to five
>
> the problem is the size of the buffer coming down from user space in the
> nzstart() function:
>
>        ...
>        data = tty->t_outq.c_cf;
>        cbp = (struct cblock *) ((intptr_t) tty->t_outq.c_cf & ~CROUND);
>        cnt = min((char *) (cbp+1) - tty->t_outq.c_cf, tty->t_outq.c_cc);
>
>        if(cnt == 0)
>                goto out;
>
>        buf = malloc(sizeof(struct fifo_buf), M_DEVBUF, M_NOWAIT);
>        memcpy(buf->data, data, cnt < sizeof(buf->data) ? cnt : sizeof(buf->data));
>        buf->size = cnt;
>
>        printf("nzdebug: nzstart() -> STAILQ_INSERT_TAIL() of %d bytes\n", cnt);
>        STAILQ_INSERT_TAIL(&fifo_head, buf, fifo_bufs);
>        ndflush(&tty->t_outq, cnt);
>        intr_ul(sc, pidx, ENABLE);
>        ...
>
> I saw frames upto 108 bytes length;
>
> and later send_data(), which puts the data into the card's buffer, picks
> the data up like this:
>
>        buf = STAILQ_FIRST(&fifo_head);
>        size = buf->size;
>
>        memcpy(sc->send_buf, buf->data, ul_size < SEND_BUF_MAX ?  ul_size : SEND_BUF_MAX );
>
>        STAILQ_REMOVE_HEAD(&fifo_head, fifo_bufs);
>        free(buf, M_DEVBUF);
>
>        port->tx_data += size;
>
>        /* Write length + data */
>        bus_write_4(sc->res, ul_offs, size);
>        bus_write_region_4(sc->res, ul_offs + 4, (u_int32_t *)sc->send_buf, size);
>
> SEND_BUF_MAX is 1024 but the 'ul_size' (size of the up link part of the
> card) is only 68!! that's why parts of the PPP frames are just thrown
> away if the frame size is bigger than 64 (4 bytes are needed for the
> size); I got to know this comparing the hex dumps of the buffer in the
> nzstart() and send_data() parts;
>
> at the moment I have no idea how to fix this :-((
> I've put the driver here:
>
> http://www.unixarea.de/nozomi/nozomi.c
>
> if someone wants to have a look and give me some hint; thanks in advance;
>
> it's Saturday evening and sunny, time for go out...
>
>        matthias
>

I can see that this is the initial port of the Linux nozomi driver that i did
back in 2006, and i got stuck exactly with the same problem,
ppp connection get's established, but only small packets are passed.
I will see if I can get my hands on this HSDPA/UMTS card
(i don't have it anymore) and help.

Niki
From matthias.apitz at oclc.org  Mon May 12 15:18:10 2008
From: matthias.apitz at oclc.org (Matthias Apitz)
Date: Mon May 12 15:18:13 2008
Subject: porting "nozomi" driver (Option N.V. GlobeTrotter 3G+ UMTS
	datacard) to FreeBSD 7.0R
In-Reply-To: <2e77fc10805101008ybafe7b3g8b9210cb8d2955d@mail.gmail.com>
References: <20080509123521.GA11366@rebelion.Sisis.de>
	<Pine.BSF.3.96.1080510002225.18341A-100000@gaia.nimnet.asn.au>
	<20080510163417.GA17718@rebelion.Sisis.de>
	<2e77fc10805101008ybafe7b3g8b9210cb8d2955d@mail.gmail.com>
Message-ID: <20080512151013.GA15571@rebelion.Sisis.de>

El d?a Saturday, May 10, 2008 a las 01:08:00PM -0400, Niki Denev escribi?:

> I can see that this is the initial port of the Linux nozomi driver that i did
> back in 2006, and i got stuck exactly with the same problem,
> ppp connection get's established, but only small packets are passed.
> I will see if I can get my hands on this HSDPA/UMTS card
> (i don't have it anymore) and help.

Niki,

I've solved the problem of the TCP packages by putting in nzstart() what
is coming down from PPPD in pieces of fifo_buf of not more then 68
bytes, i.e. if 108 bytes is to queued it will be splited, malloc'ed and
nzstart() calls STAILQ_INSERT_TAIL() with two pieces; this works fine now;

remaining is what I've posted in the forum of the Linux driver:

http://www.pharscape.org/component/option,com_forum/Itemid,68/page,viewtopic/p,1745/#1745

...
this makes SSH now happy and a log with TCPDUMP on both ends (in the
PPP interface and the ETH0 interface of the SSH server) proofes that the
data is flooding fine, nearly fine; I could already sing-on to the SSH
server two times; 
 
 the problem is that upstream data is not send fast enough, which let's
the server missing the ACK which should come in with that data and
resent its packages again; this occures if the TCP packages are big,
let's say some 700 bytes; it never occurs with short packages, nor with
the short LCP, and once you are logged in you can type chars fine and
the output of 'ls -lR /' is coming down at normal speed; 
 
 I've a full log, showing the send and received data, together with the
interrupts and as well I've inserted what the TCPDUMP on the PPPD side
sees, i.e. which TCP it sees outcoming from the driver and which it
wants to send through the driver; 
 
 I've put it here, if someone could help me with a look and some hint: 
 http://www.unixarea.de/nozomi/messages+tcp.txt 
 
 What could be the reason for not sending fast enough the data? As I
said, downstream all is fine. Is there some technical documente
available for that card Option N.V. GlobeTrotter 3G+? 
 
 Thx in advance for any hint 
 
 Matthias

-- 
Matthias Apitz
Manager Technical Support - OCLC GmbH
Gruenwalder Weg 28g - 82041 Oberhaching - Germany
t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211
e <matthias.apitz@oclc.org> - w http://www.oclc.org/ http://www.UnixArea.de/
b http://gurucubano.blogspot.com/
Don't top-post, read RFC1855 http://www.faqs.org/rfcs/rfc1855.html
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on Usenet and in e-mail?
From sbruno at miralink.com  Tue May 13 18:17:12 2008
From: sbruno at miralink.com (Sean Bruno)
Date: Tue May 13 18:17:15 2008
Subject: Realtek Ethernet Cards(new PCI IDS)?
Message-ID: <4829DB26.4070306@miralink.com>

I seem to have a motherboard (Jetway ?7K4K1G5D-LF) that has a Realtek 
ethernet controller. The driver is not picking up this controller under 
RELENG_63?



none0@pci0:9:0: class=0x020000 card=0x10ec16f3 chip=0x816710ec rev=0x10 
hdr=0x00 vendor = 'Realtek Semiconductor'
device = 'RTL8169/8110 Family Gigabit Ethernet NIC'
class = network
subclass = ethernet
none1@pci0:11:0: class=0x020000 card=0x10ec16f3 chip=0x816710ec rev=0x10 
hdr=0x00
vendor = 'Realtek Semiconductor'
device = 'RTL8169/8110 Family Gigabit Ethernet NIC'
class = network
subclass = ethernet

When I load the if_rl module, I get the following:

Source # kldload if_rl
pci0: driver added
found-> vendor=0x10ec, dev=0x8167, revid=0x10
bus=0, slot=9, func=0
class=02-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0007, statreg=0x02b0, cachelnsz=16 (dwords)
lattimer=0x20 (960 ns), mingnt=0x20 (8000 ns), maxlat=0x40 (16000 ns)
intpin=a, irq=18
powerspec 2 supports D0 D1 D2 D3 current D0
pci0:9:0: reprobing on driver added
found-> vendor=0x10ec, dev=0x8167, revid=0x10
bus=0, slot=11, func=0
class=02-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0007, statreg=0x02b0, cachelnsz=16 (dwords)
lattimer=0x20 (960 ns), mingnt=0x20 (8000 ns), maxlat=0x40 (16000 ns)
intpin=a, irq=19
powerspec 2 supports D0 D1 D2 D3 current D0
pci0:11:0: reprobing on driver added
pci1: driver added
found-> vendor=0x1106, dev=0x3344, revid=0x01
bus=1, slot=0, func=0
class=03-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0007, statreg=0x0230, cachelnsz=0 (dwords)
lattimer=0x20 (960 ns), mingnt=0x02 (500 ns), maxlat=0x00 (0 ns)
intpin=a, irq=16
powerspec 2 supports D0 D1 D2 D3 current D0
pci1:0:0: reprobing on driver added



From sbruno at miralink.com  Tue May 13 18:20:55 2008
From: sbruno at miralink.com (Sean Bruno)
Date: Tue May 13 18:20:59 2008
Subject: Realtek Ethernet Cards(new PCI IDS)?
In-Reply-To: <4829DB26.4070306@miralink.com>
References: <4829DB26.4070306@miralink.com>
Message-ID: <4829DC05.2040108@miralink.com>

Or, I am unable to read man pages and should use the "re" driver not the 
"rl" driver.

Sorry for the noise.

Sean

Sean Bruno wrote:
> I seem to have a motherboard (Jetway ?7K4K1G5D-LF) that has a Realtek 
> ethernet controller. The driver is not picking up this controller 
> under RELENG_63?
>
>
>
> none0@pci0:9:0: class=0x020000 card=0x10ec16f3 chip=0x816710ec 
> rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor'
> device = 'RTL8169/8110 Family Gigabit Ethernet NIC'
> class = network
> subclass = ethernet
> none1@pci0:11:0: class=0x020000 card=0x10ec16f3 chip=0x816710ec 
> rev=0x10 hdr=0x00
> vendor = 'Realtek Semiconductor'
> device = 'RTL8169/8110 Family Gigabit Ethernet NIC'
> class = network
> subclass = ethernet
>
> When I load the if_rl module, I get the following:
>
> Source # kldload if_rl
> pci0: driver added
> found-> vendor=0x10ec, dev=0x8167, revid=0x10
> bus=0, slot=9, func=0
> class=02-00-00, hdrtype=0x00, mfdev=0
> cmdreg=0x0007, statreg=0x02b0, cachelnsz=16 (dwords)
> lattimer=0x20 (960 ns), mingnt=0x20 (8000 ns), maxlat=0x40 (16000 ns)
> intpin=a, irq=18
> powerspec 2 supports D0 D1 D2 D3 current D0
> pci0:9:0: reprobing on driver added
> found-> vendor=0x10ec, dev=0x8167, revid=0x10
> bus=0, slot=11, func=0
> class=02-00-00, hdrtype=0x00, mfdev=0
> cmdreg=0x0007, statreg=0x02b0, cachelnsz=16 (dwords)
> lattimer=0x20 (960 ns), mingnt=0x20 (8000 ns), maxlat=0x40 (16000 ns)
> intpin=a, irq=19
> powerspec 2 supports D0 D1 D2 D3 current D0
> pci0:11:0: reprobing on driver added
> pci1: driver added
> found-> vendor=0x1106, dev=0x3344, revid=0x01
> bus=1, slot=0, func=0
> class=03-00-00, hdrtype=0x00, mfdev=0
> cmdreg=0x0007, statreg=0x0230, cachelnsz=0 (dwords)
> lattimer=0x20 (960 ns), mingnt=0x02 (500 ns), maxlat=0x00 (0 ns)
> intpin=a, irq=16
> powerspec 2 supports D0 D1 D2 D3 current D0
> pci1:0:0: reprobing on driver added
>
>
>
> _______________________________________________
> freebsd-drivers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-drivers
> To unsubscribe, send any mail to 
> "freebsd-drivers-unsubscribe@freebsd.org"

From matthias.apitz at oclc.org  Wed May 14 10:25:08 2008
From: matthias.apitz at oclc.org (Matthias Apitz)
Date: Wed May 14 10:25:12 2008
Subject: question about tty->t_outq && clist && cblock
Message-ID: <20080514102503.GA5862@rebelion.Sisis.de>


Hello,

I've some small questions about the struct 'clist', 'cblock' and 'tty'

/usr/src/sys/sys/tty.h says about 'clist'

/*
 * Clists are character lists, which is a variable length linked list
 * of cblocks, with a count of the number of characters in the list.
 */
struct clist {
        int     c_cc;           /* Number of characters in the clist. */
        int     c_cbcount;      /* Number of cblocks. */
        int     c_cbmax;        /* Max # cblocks allowed for this clist.  */
        int     c_cbreserved;   /* # cblocks reserved for this clist. */
        char    *c_cf;          /* Pointer to the first cblock. */
        char    *c_cl;          /* Pointer to the last cblock. */
};

and about 'tty'

struct tty {
        struct  clist t_rawq;           /* Device raw input queue. */
        long    t_rawcc;                /* Raw input queue statistics.  */
        struct  clist t_canq;           /* Device canonical queue. */
        long    t_cancc;                /* Canonical queue statistics.  */
        struct  clist t_outq;           /* Device output queue. */
        ...
}

and /usr/src/sys/sys/clist.h about 'cblock'

struct cblock {
        struct cblock *c_next;                  /* next cblock in queue */
        unsigned char c_quote[CBQSIZE];         /* quoted characters */
        unsigned char c_info[CBSIZE];           /* characters */
};

In the driver it seems that tty->t_outq.c_cf is pointing to the data
bytes itself which the driver should send to the device and not to
a 'cblock' (the comment in 'clist' says *c_cf; is Pointer to the first
cblock);

is this just a small error in the comment and t_outq.c_cf is really
pointing to cblock.c_info[]?

what would be the correct way to pick up the next 'cblock' in that chain
if there is already more data to deliver when for example
tty->t_outq.c_cc says 810 bytes? picking up the 4 byte pointer at
t_outq.c_cf - 20?

thanks in advance for some clarification


	matthias

-- 
Matthias Apitz
http://www.UnixArea.de/
From gaspar_i at yahoo.com  Fri May 16 15:57:05 2008
From: gaspar_i at yahoo.com (n4ti1us)
Date: Fri May 16 15:57:09 2008
Subject: Spin locks and software IRQs.
Message-ID: <17277776.post@talk.nabble.com>


Hi All,

I was looking at the linux kernel, and in the implementation of spinlocks,
one can use different functions. One of them is spin_lock_bh(). What it does
is it creates a spin lock and disables the software irqs.

In FreeBSD on the other hand, the implementation of spinlocks is done by
mtx_lock_spin (which is actually a macro that calls mtx_lock_spin_flags())
but the problem is that is disables all the irq vector (at the bottom of the
stack it executes an "cli"). Is there any way one can implement spinlocks
that only disable the bottom half (sof irq)?

Alternatively, what would be the best way to manualy disable the soft irqs?

And all this makes me think... Why has nobody implemented a "_bh" version so
far? Is this an architectural decision, or is it just me being blind?
-- 
View this message in context: http://www.nabble.com/Spin-locks-and-software-IRQs.-tp17277776p17277776.html
Sent from the freebsd-drivers mailing list archive at Nabble.com.

From jhb at freebsd.org  Mon May 19 15:01:00 2008
From: jhb at freebsd.org (John Baldwin)
Date: Mon May 19 15:01:04 2008
Subject: Spin locks and software IRQs.
In-Reply-To: <17277776.post@talk.nabble.com>
References: <17277776.post@talk.nabble.com>
Message-ID: <200805190915.00902.jhb@freebsd.org>

On Friday 16 May 2008 11:37:46 am n4ti1us wrote:
> Hi All,
>
> I was looking at the linux kernel, and in the implementation of spinlocks,
> one can use different functions. One of them is spin_lock_bh(). What it
> does is it creates a spin lock and disables the software irqs.
>
> In FreeBSD on the other hand, the implementation of spinlocks is done by
> mtx_lock_spin (which is actually a macro that calls mtx_lock_spin_flags())
> but the problem is that is disables all the irq vector (at the bottom of
> the stack it executes an "cli"). Is there any way one can implement
> spinlocks that only disable the bottom half (sof irq)?
>
> Alternatively, what would be the best way to manualy disable the soft irqs?
>
> And all this makes me think... Why has nobody implemented a "_bh" version
> so far? Is this an architectural decision, or is it just me being blind?

Generally one should use a regular mutex to protect data.  Spin locks in 
FreeBSD are only used by scheduler internals or for a driver that has to 
interact with an INTR_FAST handler (or filter in 7.x+).

-- 
John Baldwin
From freebsd at sopwith.solgatos.com  Tue May 20 18:29:15 2008
From: freebsd at sopwith.solgatos.com (Dieter)
Date: Tue May 20 18:29:19 2008
Subject: Open Graphics Project is now taking pre-orders
Message-ID: <200805201826.SAA25413@sopwith.solgatos.com>

The Open Graphics Project is now taking pre-orders for
the OGD1 development board.  This PCI-X board has 2 FPGA chips
and video outputs (2 heads).  List price is $1500, the
first 100 orders get a $100 discount.  There are developer
discounts available, so someone writing a BSD device driver,
or working on X11 support, etc. could get a board for $700-1000.
There is a variety of development work that needs to be done.
Profits from the OGD1 board will be used to bootstrap other FLOSS
friendly hardware projects (not limited to graphics).  If you are
tired of undocumented hardware, here is a chance to do something
about it.

The product info page:
http://www.traversaltech.com/products.phtml

The FAQ:
http://www.traversaltech.com/ogd1p_faq2.phtml

The order page:
http://www.traversaltech.com/store.phtml

For a developer discount, contact the Open Hardware Foundation:
http://www.openhardwarefoundation.org/
From solarux at hotmail.com  Fri May 23 07:55:01 2008
From: solarux at hotmail.com (Rick Nekus)
Date: Fri May 23 07:55:08 2008
Subject: Open Graphics Project is now taking pre-orders
In-Reply-To: <200805201826.SAA25413@sopwith.solgatos.com>
References: <200805201826.SAA25413@sopwith.solgatos.com>
Message-ID: <BAY113-W28235BF47DBDA5B351CDC4A1C10@phx.gbl>


Hi,
 I'm just curious here, but why PCI-X, why not PCI-EXpress ? or is this obviously for development reasons.
> To: freebsd-drivers@freebsd.org; freebsd-hardware@freebsd.org; freebsd-multimedia@freebsd.org> Date: Tue, 20 May 2008 11:26:31 +0100> From: freebsd@sopwith.solgatos.com> CC: > Subject: Open Graphics Project is now taking pre-orders> > The Open Graphics Project is now taking pre-orders for> the OGD1 development board. This PCI-X board has 2 FPGA chips> and video outputs (2 heads). List price is $1500, the> first 100 orders get a $100 discount. There are developer> discounts available, so someone writing a BSD device driver,> or working on X11 support, etc. could get a board for $700-1000.> There is a variety of development work that needs to be done.> Profits from the OGD1 board will be used to bootstrap other FLOSS> friendly hardware projects (not limited to graphics). If you are> tired of undocumented hardware, here is a chance to do something> about it.> > The product info page:> http://www.traversaltech.com/products.phtml> > The FAQ:> http://www.traversaltech.com/ogd1p_faq2.phtml> > The order page:> http://www.traversaltech.com/store.phtml> > For a developer discount, contact the Open Hardware Foundation:> http://www.openhardwarefoundation.org/> _______________________________________________> freebsd-multimedia@freebsd.org mailing list> http://lists.freebsd.org/mailman/listinfo/freebsd-multimedia> To unsubscribe, send any mail to "freebsd-multimedia-unsubscribe@freebsd.org"
_________________________________________________________________
If you like crossword puzzles, then you'll love Flexicon, a game which combines four overlapping crossword puzzles into one!
http://g.msn.ca/ca55/208
From freebsd at sopwith.solgatos.com  Fri May 23 15:22:45 2008
From: freebsd at sopwith.solgatos.com (Dieter)
Date: Fri May 23 15:22:51 2008
Subject: Open Graphics Project is now taking pre-orders 
In-Reply-To: Your message of "Fri, 23 May 2008 07:43:01 -0000."
	<BAY113-W28235BF47DBDA5B351CDC4A1C10@phx.gbl> 
Message-ID: <200805231515.PAA10055@sopwith.solgatos.com>

> I'm just curious here, but why PCI-X, why not PCI-EXpress ?
> or is this obviously for development reasons.

According to the FAQ,

	While most graphics cards now fit in PCI Express slots,
	PCI is more popular with users of FPGA kits. We have
	identified the parts necessary to support PCI Express,
	so if there is demand, we can build them.  In the mean
	time, we need to sell the PCI version OGD1 to bootstrap
	our efforts.

	PCI-X is backward compatible with your 32-bit 33MHz PCI slots,
	and OGD1 has been tested with several PC motherboards. Note
	that some PC motherboards may position components so that they
	interfere with the "extended" part of the PCI card edge for
	some slots.  OGD1 is 6.875in long and 4.2in wide
	(17.46cm x 10.67cm).  Some small form-factor systems do not
	provide enough room in one or both dimensions.

There is a development advantage to PCI.  PCI is a bus, so it is easy
to sniff the bus if things aren't working.  There has been some
discussion about writing software to turn the OGD1 into a PCI bus
sniffer, for developing other boards.

If you are interested in buying a PCIe version of OGD1, OGP needs to
hear about it.
From koitsu at FreeBSD.org  Fri May 23 15:59:32 2008
From: koitsu at FreeBSD.org (Jeremy Chadwick)
Date: Fri May 23 15:59:34 2008
Subject: Open Graphics Project is now taking pre-orders
In-Reply-To: <200805231515.PAA10055@sopwith.solgatos.com>
References: <BAY113-W28235BF47DBDA5B351CDC4A1C10@phx.gbl>
	<200805231515.PAA10055@sopwith.solgatos.com>
Message-ID: <20080523153937.GA40374@eos.sc1.parodius.com>

On Fri, May 23, 2008 at 08:15:53AM +0100, Dieter wrote:
> > I'm just curious here, but why PCI-X, why not PCI-EXpress ?
> > or is this obviously for development reasons.
> 
> According to the FAQ,
> 
> 	While most graphics cards now fit in PCI Express slots,
> 	PCI is more popular with users of FPGA kits. We have
> 	identified the parts necessary to support PCI Express,
> 	so if there is demand, we can build them.  In the mean
> 	time, we need to sell the PCI version OGD1 to bootstrap
> 	our efforts.
> 
> 	PCI-X is backward compatible with your 32-bit 33MHz PCI slots,
> 	and OGD1 has been tested with several PC motherboards. Note
> 	that some PC motherboards may position components so that they
> 	interfere with the "extended" part of the PCI card edge for
> 	some slots.  OGD1 is 6.875in long and 4.2in wide
> 	(17.46cm x 10.67cm).  Some small form-factor systems do not
> 	provide enough room in one or both dimensions.
> 
> There is a development advantage to PCI.  PCI is a bus, so it is easy
> to sniff the bus if things aren't working.  There has been some
> discussion about writing software to turn the OGD1 into a PCI bus
> sniffer, for developing other boards.
> 
> If you are interested in buying a PCIe version of OGD1, OGP needs to
> hear about it.

I think what people (not just here, but folks on Slashdot as well) want
to know is: is the PCI-X choice *purely* for development reasons, e.g.
will the retail/non-development version (read: transistor-based, no
FPGA) of the card be PCIe?

If the manufacturer plans on keeping everything PCI-X -- dev cards and
commercial/retail cards -- they are making a *huge* mistake.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

From freebsd at sopwith.solgatos.com  Fri May 23 16:54:58 2008
From: freebsd at sopwith.solgatos.com (Dieter)
Date: Fri May 23 16:55:01 2008
Subject: Open Graphics Project is now taking pre-orders 
In-Reply-To: Your message of "Fri, 23 May 2008 08:39:37 PDT."
	<20080523153937.GA40374@eos.sc1.parodius.com> 
Message-ID: <200805231652.QAA05864@sopwith.solgatos.com>

> I think what people (not just here, but folks on Slashdot as well) want
> to know is: is the PCI-X choice *purely* for development reasons, e.g.
> will the retail/non-development version (read: transistor-based, no
> FPGA) of the card be PCIe?
> 
> If the manufacturer plans on keeping everything PCI-X -- dev cards and
> commercial/retail cards -- they are making a *huge* mistake.

The plan, as I understand it, is:

	Develop GPU logic using OGD1 boards.

	Sell as many OGD1 boards as possible to raise money needed
	(estimated at US$2M) to fabricate ASIC.

	Probably do other FLOSS friendly hardware projects to
	raise money for ASIC.  (Ideas welcome)

	Build OGC boards with ASIC, no FPGA.

The ASICs are expected to sell in *much* larger quantities than the
FPGA boards.

I'm 99.9% sure that the ASIC based OGC boards will be available as
PCIe.  Possibly PCI or PCI-X and AGP as well, and the bare ASIC chips
will be available for embedded applications (PDAs, kiosks, etc.).

The big hurdle is raising the US$2M needed to fab the ASIC.
From gpalmer at freebsd.org  Fri May 23 17:11:25 2008
From: gpalmer at freebsd.org (Gary Palmer)
Date: Fri May 23 17:11:33 2008
Subject: Open Graphics Project is now taking pre-orders
In-Reply-To: <20080523153937.GA40374@eos.sc1.parodius.com>
References: <BAY113-W28235BF47DBDA5B351CDC4A1C10@phx.gbl>
	<200805231515.PAA10055@sopwith.solgatos.com>
	<20080523153937.GA40374@eos.sc1.parodius.com>
Message-ID: <20080523171124.GB1142@in-addr.com>

On Fri, May 23, 2008 at 08:39:37AM -0700, Jeremy Chadwick wrote:
> On Fri, May 23, 2008 at 08:15:53AM +0100, Dieter wrote:
> > > I'm just curious here, but why PCI-X, why not PCI-EXpress ?
> > > or is this obviously for development reasons.
> > 
> > According to the FAQ,
> > 
> > 	While most graphics cards now fit in PCI Express slots,
> > 	PCI is more popular with users of FPGA kits. We have
> > 	identified the parts necessary to support PCI Express,
> > 	so if there is demand, we can build them.  In the mean
> > 	time, we need to sell the PCI version OGD1 to bootstrap
> > 	our efforts.
> > 
> > 	PCI-X is backward compatible with your 32-bit 33MHz PCI slots,
> > 	and OGD1 has been tested with several PC motherboards. Note
> > 	that some PC motherboards may position components so that they
> > 	interfere with the "extended" part of the PCI card edge for
> > 	some slots.  OGD1 is 6.875in long and 4.2in wide
> > 	(17.46cm x 10.67cm).  Some small form-factor systems do not
> > 	provide enough room in one or both dimensions.
> > 
> > There is a development advantage to PCI.  PCI is a bus, so it is easy
> > to sniff the bus if things aren't working.  There has been some
> > discussion about writing software to turn the OGD1 into a PCI bus
> > sniffer, for developing other boards.
> > 
> > If you are interested in buying a PCIe version of OGD1, OGP needs to
> > hear about it.
> 
> I think what people (not just here, but folks on Slashdot as well) want
> to know is: is the PCI-X choice *purely* for development reasons, e.g.
> will the retail/non-development version (read: transistor-based, no
> FPGA) of the card be PCIe?
> 
> If the manufacturer plans on keeping everything PCI-X -- dev cards and
> commercial/retail cards -- they are making a *huge* mistake.

This is answered on their website.  The reason they chose PCI-X was
that they can slap a PCIe to PCI-X bridge on it without having to
choose new FPGAs and respin the boards (which would delay things even
more)

http://wiki.opengraphics.org/tiki-index.php?page=OGPN20#PCIe

(amongst many others)

http://www.google.com/search?q=site%3Awiki.opengraphics.org+pcie

Regards,

Gary

P.S. isn't this thread violating the crossposting rules?
From sbruno at miralink.com  Sun May 25 23:21:25 2008
From: sbruno at miralink.com (Sean Bruno)
Date: Sun May 25 23:21:28 2008
Subject: ATA patch for RELENG_6 ... a patch looking for a good home
Message-ID: <4839F473.6070109@miralink.com>

I got this patch a while ago and I don't see it appearing in RELENG_6
yet.  Can someone "sheperd" this along or point out why it's not acceptable?

This patch was generated by a failure to boot correctly off of a compact
flash IDE module from Transcend.

Index: dev/ata/ata-chipset.c
===================================================================
--- dev/ata/ata-chipset.c       (.../FreeBSD_RELENG_6_13APR07/src/sys)
(revision 5436)
+++ dev/ata/ata-chipset.c       (.../miralink.FreeBSD.6/src/sys)
(revision 5436)
@@ -2059,7 +2059,8 @@
            atadev->mode = ATA_SA150;
     }
     else {
-       mode = ata_limit_mode(dev, mode, ATA_UDMA5);
+       /*mode = ata_limit_mode(dev, mode, ATA_UDMA5);*/
+       mode = ata_check_80pin(dev, ATA_UDMA5);
        if (!ata_controlcmd(dev, ATA_SETFEATURES, ATA_SF_SETXFER, 0, mode))
            atadev->mode = mode;
     }

-- 
Sean Bruno
MiraLink Corporation
6015 NE 80th Ave, Ste 100
Portland, OR 97218
Phone 503-621-5143
Fax 503-621-5199


From aopopov at yahoo.com  Tue May 27 09:59:15 2008
From: aopopov at yahoo.com (Alexander Popov)
Date: Tue May 27 09:59:22 2008
Subject: Synchronization in drivers (after SMP improvements)
Message-ID: <730729.32628.qm@web51410.mail.re2.yahoo.com>

Hello all,

I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done.

Previously, i.e. in 4.4 BSD, the following statements were true:

1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode;
2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver).
   
This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever.

Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE.

It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used:
int driver_read(...)
{
   mtx_lock(&sc->mtx);
   
   if (no data is available) => tsleep(..)
   
   mtx_unlock(&sc->mtx);
}

First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex).

Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel...

The bottom line is:
1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this);
2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed;
3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed.

Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver?

Of course, I might be missing something here, please help.

Thanks in advance,

Alexander.


      
From ilya at po4ta.com  Tue May 27 23:21:25 2008
From: ilya at po4ta.com (Ilya Bobir)
Date: Tue May 27 23:21:30 2008
Subject: Synchronization in drivers (after SMP improvements)
In-Reply-To: <730729.32628.qm@web51410.mail.re2.yahoo.com>
References: <730729.32628.qm@web51410.mail.re2.yahoo.com>
Message-ID: <483C926D.50104@po4ta.com>

Alexander Popov wrote:
> Hello all,
>
> [...]
>
> It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used:
> int driver_read(...)
> {
>    mtx_lock(&sc->mtx);
>    
>    if (no data is available) => tsleep(..)
>    
>    mtx_unlock(&sc->mtx);
> }
>
> First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex).
> [...]
>   

Do you really need to hold the mutex while sleeping?
Maybe you need to mtx_sleep instead?
From Benjamin.Close at clearchain.com  Tue May 27 23:24:22 2008
From: Benjamin.Close at clearchain.com (Benjamin Close)
Date: Tue May 27 23:24:27 2008
Subject: Synchronization in drivers (after SMP improvements)
In-Reply-To: <730729.32628.qm@web51410.mail.re2.yahoo.com>
References: <730729.32628.qm@web51410.mail.re2.yahoo.com>
Message-ID: <483C9493.3070607@clearchain.com>

Alexander Popov wrote:
> Hello all,
>
> I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done.
>
> Previously, i.e. in 4.4 BSD, the following statements were true:
>
> 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode;
> 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver).
>    
> This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever.
>
> Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE.
>
> It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used:
> int driver_read(...)
> {
>    mtx_lock(&sc->mtx);
>    
>    if (no data is available) => tsleep(..)
>    
>    mtx_unlock(&sc->mtx);
> }
>
> First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex).
>
> Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel...
>
> The bottom line is:
> 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this);
> 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed;
> 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed.
>
> Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver?
>
> Of course, I might be missing something here, please help.
>
> Thanks in advance,
>
>   
Hi Alexander,
    As you've noticed, sleeping using tlseep whilst holding a mutex can 
cause you grief due to preemption. This is why mtx_sleep was introduced. 
It drops the lock before the sleep and reaquires it before continuing. 
Hence you can then be preempted without issues. Under 7 and above, man 9 
locking is a very useful page. In general drivers tend to use mutexes or 
in some cases rwlocks. Be aware though, you should not sleep in an 
interrupt handler.

Cheers,
    Benjamin
From aopopov at yahoo.com  Wed May 28 12:25:07 2008
From: aopopov at yahoo.com (Alexander Popov)
Date: Wed May 28 12:25:11 2008
Subject: Synchronization in drivers (after SMP improvements)
Message-ID: <600706.33129.qm@web51403.mail.re2.yahoo.com>

Hi, Benjamin, also Ilya,


Thanks for your quick responses. mtx_sleep() would
definitely help. Does it mean that user process executing in kernel space is
guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then
I would get a sleeping thread with
non-sleepable mutex anyway... or have I been in application
development for too long? :-)
Regards,

Alexander.


----- Original Message ----
From: Benjamin Close <Benjamin.Close@clearchain.com>
To: Alexander Popov <aopopov@yahoo.com>
Cc: freebsd-drivers@freebsd.org
Sent: Wednesday, May 28, 2008 1:09:07 AM
Subject: Re: Synchronization in drivers (after SMP improvements)

Alexander Popov wrote:
> Hello all,
>
> I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done.
>
> Previously, i.e. in 4.4 BSD, the following statements were true:
>
> 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode;
> 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver).
>    
> This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever.
>
> Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE.
>
> It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used:
> int driver_read(...)
> {
>    mtx_lock(&sc->mtx);
>    
>    if (no data is available) => tsleep(..)
>    
>    mtx_unlock(&sc->mtx);
> }
>
> First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex).
>
> Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel...
>
> The bottom line is:
> 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this);
> 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed;
> 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed.
>
> Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver?
>
> Of course, I might be missing something here, please help.
>
> Thanks in advance,
>
>  
Hi Alexander,
    As you've noticed, sleeping using tlseep whilst holding a mutex can 
cause you grief due to preemption. This is why mtx_sleep was introduced. 
It drops the lock before the sleep and reaquires it before continuing. 
Hence you can then be preempted without issues. Under 7 and above, man 9 
locking is a very useful page. In general drivers tend to use mutexes or 
in some cases rwlocks. Be aware though, you should not sleep in an 
interrupt handler.

Cheers,
    Benjamin



      
From ilya at po4ta.com  Wed May 28 13:49:34 2008
From: ilya at po4ta.com (Ilya Bobir)
Date: Wed May 28 13:49:39 2008
Subject: Synchronization in drivers (after SMP improvements)
In-Reply-To: <600706.33129.qm@web51403.mail.re2.yahoo.com>
References: <600706.33129.qm@web51403.mail.re2.yahoo.com>
Message-ID: <483D62D0.6070800@po4ta.com>

Alexander Popov wrote:
> Hi, Benjamin, also Ilya,
>
>
> Thanks for your quick responses. mtx_sleep() would
> definitely help. Does it mean that user process executing in kernel space is
> guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then
> I would get a sleeping thread with
> non-sleepable mutex anyway... or have I been in application
> development for too long? :-)
> Regards,
>
> Alexander.
>
>   

According to locking(9) (" Context mode table." at the very bottom) the 
only type of mutex you can use in your interrupt handler is a spin 
mutex.  And you are trying to synchronize with an interrupt handler.  
So, the sc->mtx ought to be a spin mutex.  While a spin mutex is held 
all interrupts on the current CPU are blocked or deferred (mutex(9), 
DESCRIPTION fifth paragraph) and the thread holding the mutex will not 
be preempted.

And you should use msleep_spin().
From imp at bsdimp.com  Wed May 28 14:19:23 2008
From: imp at bsdimp.com (M. Warner Losh)
Date: Wed May 28 14:19:27 2008
Subject: Synchronization in drivers (after SMP improvements)
In-Reply-To: <483D62D0.6070800@po4ta.com>
References: <600706.33129.qm@web51403.mail.re2.yahoo.com>
	<483D62D0.6070800@po4ta.com>
Message-ID: <20080528.081746.1683967403.imp@bsdimp.com>

In message: <483D62D0.6070800@po4ta.com>
            Ilya Bobir <ilya@po4ta.com> writes:
: Alexander Popov wrote:
: > Hi, Benjamin, also Ilya,
: >
: >
: > Thanks for your quick responses. mtx_sleep() would
: > definitely help. Does it mean that user process executing in kernel space is
: > guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then
: > I would get a sleeping thread with
: > non-sleepable mutex anyway... or have I been in application
: > development for too long? :-)
: > Regards,
: >
: > Alexander.
: >
: >   
: 
: According to locking(9) (" Context mode table." at the very bottom) the 
: only type of mutex you can use in your interrupt handler is a spin 
: mutex.  And you are trying to synchronize with an interrupt handler.  
: So, the sc->mtx ought to be a spin mutex.  While a spin mutex is held 
: all interrupts on the current CPU are blocked or deferred (mutex(9), 
: DESCRIPTION fifth paragraph) and the thread holding the mutex will not 
: be preempted.

That's only true of 'fast' interrupt handlers.  Normal Ithread
interrupt hanlders, mutexes are fine since you aren't really in an
interrupt context.  If this is indeed a usb device driver, then the
context for a callback routine for an interrupt endpoint really is a
taskqueue...

Warner
From imp at bsdimp.com  Wed May 28 14:53:21 2008
From: imp at bsdimp.com (M. Warner Losh)
Date: Wed May 28 14:53:24 2008
Subject: Synchronization in drivers (after SMP improvements)
In-Reply-To: <730729.32628.qm@web51410.mail.re2.yahoo.com>
References: <730729.32628.qm@web51410.mail.re2.yahoo.com>
Message-ID: <20080528.085051.-1253012495.imp@bsdimp.com>

In message: <730729.32628.qm@web51410.mail.re2.yahoo.com>
            Alexander Popov <aopopov@yahoo.com> writes:
: I'm currently writing a FreeBSD USB driver and have a few questions
: about synchronization between bottom and top half of the
: driver.

By USB driver, I'm assuming you mean "a driver for a usb device that's
plugged into the system" as opposed to "a driver for a usb host
adapter."

: This is the strategy that is followed by most of the (USB) drivers
: currently. However, after the SMP improvements have been
: implemented, the splxxx family of functions has been successfully
: stubbed out, so now these functions do absolutely nothing, which
: (correct me if am wrong) leaves most of the existing USB drivers
: without any synchronization with their bottom halves whatsoever.

Right now, the USB stack is Giant locked, so this stuff works.

: Next, in the scope of SMP project, interrupt handlers have been
: replaced with (lightweight?) threads, which makes it possible to use
: synchronization primitives in the interrupt handlers to synchronize
: them with driver's top half. However, this does not apply to USB
: stack, which, as I understand, still uses Giant lock. So, from this
: perspective, I can not make interrupts in my USB driver MPSAFE.

Well, you could, but it is really hard to do since you have to
understand the current USB code and be able to acquire Giant at the
right times, etc.  So theoretically possible, but I don't know of
anybody that's succeed in doing so.

: It has been suggested everywhere that mutexes shall be used to
: protect top and bottom halves of the driver. So I've tried. The
: following is the pseudo-code that I've used:
: int driver_read(...)
: {
:    mtx_lock(&sc->mtx);
:    
:    if (no data is available) => tsleep(..)
:    
:    mtx_unlock(&sc->mtx);
: }

As others have pointed out, the fundamental problem here is that you
are using tsleep while holding a mutex.  This is not allowed because
it leads to bad things happening.  Not always, but sometimes as you've
discovered.

: First it worked very well, but under relatively heavy load I started
: getting kernel panic, all the time related to one error: a thread
: holding a non-sleepable lock. Which means that user process has been
: suspended somewhere during execution in the read() function
: (probably awaiting data, but not necessary) and its thread has been
: holding mutex that I've used for synchronization, but that it
: apparently not allowed (look in man page on mutex).

The correct solution here is to just use msleep.

: Then I found that in SMP project an additional flag was added to
: mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping
: threads in kernel that hold mutexes. But, apparently, this flag has
: been recently removed from kernel...

This isn't what you want...

: Question is: what is now a synchronization model for modern kernels,
: i.e. FreeBSD 7.0? How shall I properly implement synchronization in
: my driver?

I think you can use msleep.  Alternatively, you could use
condvars for synchronization.  And I'd make the if () statement a
while () statement above to guard against races.

Warner

From aopopov at yahoo.com  Fri May 30 15:34:09 2008
From: aopopov at yahoo.com (Alexander Popov)
Date: Fri May 30 15:34:13 2008
Subject: Synchronization in drivers (after SMP improvements)
Message-ID: <987362.36986.qm@web51404.mail.re2.yahoo.com>

Hi, Warner,

Thanks, I appreciate your help.
Yes, indeed I am writing a USB device driver. I have done some tests,
and results are interesting. First of all, when I am trying to use default mutex in top-haft of the driver, under heavy load it panics anyway, even though there is no tsleep or alike functions used while holding the mutex. Basically what I do is this:

void driver_open()
{
   mtx_lock(&sc->mtx)
   // open USB pipes to device, etc.
   mtx_unlock(&sc->mtx)
}

void driver_write(...)
{
   mtx_lock(&sc->mtx)
   // write to device (I do not sleep here!)
   mtx_unlock(&sc->mtx)
}

then, one process opens device, and starts repeatedly write to it, constantly acquiring and leaving mutex; then, I start another process, that simply tries to open the same device => if it happens to be while first process is holding the mutex => kernel panics with message "sleeping thread with non-sleepable lock". 

When I change default mutex to spin type, it works well, and I was not able to crash it.  As Ilya pointed out, locking(9) says that Giant lock must be acquired before any other locks, which might be the reason why my code crashes with default mutex - USB drivers are Giant locked.
It is interesting, because I've seen USB drivers in the default FreeBSD distribution that use default mutexes...

However, when I try to lock spin mutex from interrupt handler, everything blocks and after a while I get panic with a message that "spin mutex has been locked for too long". Could not find explanation for this one yet...

Regards,

Alexander.

----- Original Message ----
From: M. Warner Losh <imp@bsdimp.com>
To: aopopov@yahoo.com
Cc: freebsd-drivers@FreeBSD.org
Sent: Wednesday, May 28, 2008 4:50:51 PM
Subject: Re: Synchronization in drivers (after SMP improvements)

In message: <730729.32628.qm@web51410.mail.re2.yahoo.com>
            Alexander Popov <aopopov@yahoo.com> writes:
: I'm currently writing a FreeBSD USB driver and have a few questions
: about synchronization between bottom and top half of the
: driver.

By USB driver, I'm assuming you mean "a driver for a usb device that's
plugged into the system" as opposed to "a driver for a usb host
adapter."

: This is the strategy that is followed by most of the (USB) drivers
: currently. However, after the SMP improvements have been
: implemented, the splxxx family of functions has been successfully
: stubbed out, so now these functions do absolutely nothing, which
: (correct me if am wrong) leaves most of the existing USB drivers
: without any synchronization with their bottom halves whatsoever.

Right now, the USB stack is Giant locked, so this stuff works.

: Next, in the scope of SMP project, interrupt handlers have been
: replaced with (lightweight?) threads, which makes it possible to use
: synchronization primitives in the interrupt handlers to synchronize
: them with driver's top half. However, this does not apply to USB
: stack, which, as I understand, still uses Giant lock. So, from this
: perspective, I can not make interrupts in my USB driver MPSAFE.

Well, you could, but it is really hard to do since you have to
understand the current USB code and be able to acquire Giant at the
right times, etc.  So theoretically possible, but I don't know of
anybody that's succeed in doing so.

: It has been suggested everywhere that mutexes shall be used to
: protect top and bottom halves of the driver. So I've tried. The
: following is the pseudo-code that I've used:
: int driver_read(...)
: {
:    mtx_lock(&sc->mtx);
:    
:    if (no data is available) => tsleep(..)
:    
:    mtx_unlock(&sc->mtx);
: }

As others have pointed out, the fundamental problem here is that you
are using tsleep while holding a mutex.  This is not allowed because
it leads to bad things happening.  Not always, but sometimes as you've
discovered.

: First it worked very well, but under relatively heavy load I started
: getting kernel panic, all the time related to one error: a thread
: holding a non-sleepable lock. Which means that user process has been
: suspended somewhere during execution in the read() function
: (probably awaiting data, but not necessary) and its thread has been
: holding mutex that I've used for synchronization, but that it
: apparently not allowed (look in man page on mutex).

The correct solution here is to just use msleep.

: Then I found that in SMP project an additional flag was added to
: mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping
: threads in kernel that hold mutexes. But, apparently, this flag has
: been recently removed from kernel...

This isn't what you want...

: Question is: what is now a synchronization model for modern kernels,
: i.e. FreeBSD 7.0? How shall I properly implement synchronization in
: my driver?

I think you can use msleep.  Alternatively, you could use
condvars for synchronization.  And I'd make the if () statement a
while () statement above to guard against races.

Warner


      
From ilya at po4ta.com  Sat May 31 17:57:38 2008
From: ilya at po4ta.com (Ilya Bobir)
Date: Sat May 31 17:57:42 2008
Subject: Synchronization in drivers (after SMP improvements)
In-Reply-To: <987362.36986.qm@web51404.mail.re2.yahoo.com>
References: <987362.36986.qm@web51404.mail.re2.yahoo.com>
Message-ID: <48419182.1030200@po4ta.com>

Alexander Popov wrote:
> Hi, Warner,
>
> Thanks, I appreciate your help.
> Yes, indeed I am writing a USB device driver. I have done some tests,
> and results are interesting. First of all, when I am trying to use default mutex in top-haft of the driver, under heavy load it panics anyway, even though there is no tsleep or alike functions used while holding the mutex. Basically what I do is this:
>
> void driver_open()
> {
>    mtx_lock(&sc->mtx)
>    // open USB pipes to device, etc.
>    mtx_unlock(&sc->mtx)
> }
>
> void driver_write(...)
> {
>    mtx_lock(&sc->mtx)
>    // write to device (I do not sleep here!)
>    mtx_unlock(&sc->mtx)
> }
>
> then, one process opens device, and starts repeatedly write to it, constantly acquiring and leaving mutex; then, I start another process, that simply tries to open the same device => if it happens to be while first process is holding the mutex => kernel panics with message "sleeping thread with non-sleepable lock".

The only spot in the source were I was able to find a message like this 
is in src/sys/kern/subr_turnstile.c:propagate_priority(struct thread 
*td) 
(http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/subr_turnstile.c?rev=1.173;content-type=text%2Fplain 
search for "non-sleepable").  IIUC, this message means that a thread 
that is sleeping on a lock also holds a spin lock.  Another thread 
(current thread) is trying to acquire this spin lock but finds out that 
the current owner of the spin lock is sleeping.  As sleeping while 
holding a spin lock is prohibited as it may cause a deadlock the 
function panics.  Note that the misbehaving thread's id is present in 
the message.  Chapter 11 of the "FreeBSD Developers' Handbook" explains 
how you can get a thread stack trace from a kernel dump or a running 
kernel.  But you probably already know it.

> When I change default mutex to spin type, it works well, and I was not able to crash it.  As Ilya pointed out, locking(9) says that Giant lock must be acquired before any other locks, which might be the reason why my code crashes with default mutex - USB drivers are Giant locked.
> It is interesting, because I've seen USB drivers in the default FreeBSD distribution that use default mutexes...
>   

I think you would get a lock order reversal message or some other kind 
of message if you are locking Giant after some other lock.  "Sleeping 
thread (tid %d, pid %d) owns a non-sleepable lock\n" is not about Giant.

> However, when I try to lock spin mutex from interrupt handler, everything blocks and after a while I get panic with a message that "spin mutex has been locked for too long". Could not find explanation for this one yet...
> [...]

It probably means that the mutex is already locked.  You can drop into a 
debugger and try do debug it as described in "11.9 Debugging Deadlocks" 
in the "FreeBSD Developers' Handbook".

P.S.  Note, that sleeping and preemption are different.  A thread is 
sleeping when it is waiting on a sleepable lock.  While waiting on a 
non-sleepable lock or in some other cases a thread may be preempted but 
it is not marked as sleeping.