Performing a socket accept in listen sockets' upcall - Yes or No?

[Mathur, Vedant]

Hi all,

I am working with FreeBSD in an environment where majority of my applications run in the kernel and are heavily relying on the socket upcall mechanism. All of these kernel applications use the socket upcall mechanism for socket I/O.

These applications also use the upcall mechanism to accept incoming TCP connections.  How the applications do so is by registering a receive upcall on the head socket and waiting for the upcall. When the stack makes the receive upcall on the listen socket, most of the applications defer the actual accept to a different thread using a queueing mechanism and return from the upcall.

However, I have some applications which instead of deferring the socket accept operation to a different thread want to perform the socket accept operation inline. In other words the application wants to accept the child socket in the context of the listen sockets' upcall. My questions are regarding this behavior and are as follows:

A) First and foremost, does the FreeBSD socket upcall framework even support such kind of an accept operation? Does this conceptually break FreeBSD socket accept and upcall semantics? Is the upcall mechanism to be ONLY used for socket I/O?

B) If the accept operation is supported in the context of the head sockets' upcall then, how do we handle the use after free cases where issuing a head sockets upcall can potentially make the application accept and then close the child socket which we access after the upcall returns. To be more specific, in FreeBSD 11.x the syncache_socket() code calls sonewconn() and then connects the child socket to the peer. Later it calls soisconnected() which places the child socket in head's so_comp queue and makes an upcall which can potentially free the child socket. Once syncache_socket() returns, the call stack eventually returns until tcp_input() for TCP sockets. tcp_input() now calls tcp_do_segment() which accesses the child socket and its inpcb and tcpcb which can potentially be freed memory.

If A) is possible then, I can see that commit r261242 modified the code  such that we deferred placing the child socket on the so_comp queue until the connect was complete. I believe this change is still susceptible to the use after free case described above.  If we agree then what are the potential solutions?

Thanks!
Vedant Mathur