python 3 subprocess performance

[Dima Pasechnik]

On Fri, Apr 12, 2019 at 9:46 AM Alexander Zagrebin <alex at zagrebin.ru> wrote:
>
> В Fri, 12 Apr 2019 09:36:13 +0200
> Dima Pasechnik <dimpase+freebsd at gmail.com> пишет:
>
> > On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin <alex at zagrebin.ru>
> > wrote:
> > >
> > > В Thu, 11 Apr 2019 17:32:42 +0200
> > > Jan Bramkamp <crest at rlwinm.de> пишет:
> > >
> > > > The reason is that that python does something stupid (tm). It
> > > > tries to close all file descriptors (except a few whitelisted
> > > > ones) up to the maximum file descriptor number. It does this by
> > > > asking the kernel for the maximum possible number and closing
> > > > everything it doesn't want to keep. Some time later someone came
> > > > up with an optimization (read the open file descriptors
> > > > from /dev/fd). All of this pain and suffering is caused by good
> > > > old Ulrich Drepper braindamage:
> > > > https://sourceware.org/bugzilla/show_bug.cgi?id=10353.
> > > >
> > > > Most Linux distros have lower default file descriptor limits than
> > > > FreeBSD making this workaround less painful. The correct solution
> > > > would be to teach python3 about closefrom(2).
> > >
> > > Thank you for hint and testing!
> > >
> > > Indeed the problem is in closing more than 400,000 file descriptors
> > > in loop. It seems that all current versions of Python are affected.
> > > Python2 uses False as default value for the close_fds parameter of
> > > the Popen constructor, so this issue is mostly not visible.
> > > Python3 has changed this default to True.
> > >
> > > As Jan Bramkamp suggested, I've wrote simple patch to fix an issue
> > > (see attached file). It seems the problem has gone.
> >
> > The attachment has been stripped out. Could you paste the diff into
> > the message?
>
> Yes, sure.
>
> --- Modules/_posixsubprocess.c.orig     2018-12-24 00:37:14.000000000
> +0300 +++ Modules/_posixsubprocess.c          2019-04-12
> 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@
> _close_fds_by_brute_force(long start_fd, }
>          start_fd = keep_fd + 1;
>      }
> +#if defined(__FreeBSD__)
> +    closefrom(start_fd);
> +#else
>      if (start_fd <= end_fd) {
>          for (fd_num = start_fd; fd_num < end_fd; ++fd_num) {
>              close(fd_num);
>          }
>      }
> +#endif
>  }
>
> > If this is a Python issue, shouldn't this be reported upstream, on
> > https://bugs.python.org ?
>
> May be. Rather, it is a FreeBSD-specific optimization.

Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)),
OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is
there, I just checked)
Anyway, FreeBSD Python maintainers will ask for an upstream PR.

I can do such a PR is noone else is willing to...

Dima


>
> --
> Alexander Zagrebin
> _______________________________________________
> freebsd-ports at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscribe at freebsd.org"