Create a thread with a separate file descriptor table (set RFFDG flag)

[Double Tong]

Thank you Alan!

As the program's main structure is mostly done, using shared memory object to help coping with this issue requires too much work for us. We did not use any select calls. I was told the performance gain can be greater if these connections are short-lived (i.e. one connection transmit one request). I will try it later to get some data. And on Linux it is easy to do, simply a unshare(2) syscall would do it, which is why we are considering doing this on FreeBSD as well. Anyway it is something nice to have, not something that would block the project.

I am still interested my question 2 and 3, as theoretically, it feels like the idea should work.

Thanks,
-Shuangyi Tong
________________________________
From: asomers at gmail.com <asomers at gmail.com> on behalf of Alan Somers <asomers at freebsd.org>
Sent: August 16, 2018 12:38
To: Double Tong
Cc: freebsd-hackers at freebsd.org
Subject: Re: Create a thread with a separate file descriptor table (set RFFDG flag)

On Thu, Aug 16, 2018 at 10:28 AM, Double Tong <doubleble at outlook.com<mailto:doubleble at outlook.com>> wrote:
Hello,

I want to create a thread with a separate file descriptor table to have better performance with kevent(2). On Linux, I was using unshare(2) syscall to achieve this, which as far as I know there is no equivalent or similar syscall in FreeBSD.

I have posted on freebsd forums (https://forums.freebsd.org/threads/create-a-thread-with-a-separate-file-descriptor-table-set-rffdg-flag.67143/), and now I understood the following:  rfork_thread(3) is deprecated in favor of pthread_create(3). rfork_thread(3) is written in assembly language to perform stack swapping, which means if rfork_thread(3) no longer exists in the build, it can damage our program's portability if it relies on rfork_thread.

With the above consideration, I still have the following questions:

  1.  Is there an elegant way to create a thread with a separate file descriptor table?

Sort of.  The usual way to do this is to create a separate process, not a separate thread.  If you also need a shared address space, then you can probably satisfy that by creating a shared memory object, and using that for whichever data structures actually need to be shared.  See shm_open(2) for details.

  2.  If you are thinking about using rfork_thread(3) to do this, I am working on this direction. I am using waitpid to join these "threads", and the thread exits in the middle of execution with status 0x8b collected by waitpid. I guess this status means invalid page access. I wrote a tiny program (attached below) to reflect the code I am using in my program, I appreciate if you would like to take a look at it to see if there is anything I was not doing correctly.
  3.  As I was reading the code of pthread_create, it allocates a pthread struct on the top of thread, and then calls clone, which freebsd implemented its version of clone that actually calls rfork (I did not find the source of freebsd's clone, can someone provides a link?). So I believe theoretically there should be a way to achieve this in the user space. And if I am not using pthread related APIs, then missing pthread struct should be fine as well?
  4.  On Linux, after calling unshare(CLONE_FLIES), I got performance increase around 10% with 1000 concurrent TCP connections. I am instructed to implement this by my supervisor, and I do not have much details about why the performance would increase. Would this also works for freebsd as well (kevent calls)?

Does your Linux program use select(2), epoll(2), or something else?  select(2) has well-known performance problems with large file-descriptor tables, but epoll(2) and kevent(2) do not.  If your program is using select(2), then you should convert it to use epoll(2)/kevent(2) and ditch unshare(2).


Thank you for any help, comments in advance!


-Alan