python 3 subprocess performance

[Dima Pasechnik]

On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin <alex at zagrebin.ru> wrote:
>
> В Thu, 11 Apr 2019 17:32:42 +0200
> Jan Bramkamp <crest at rlwinm.de> пишет:
>
> > The reason is that that python does something stupid (tm). It tries to
> > close all file descriptors (except a few whitelisted ones) up to the
> > maximum file descriptor number. It does this by asking the kernel for
> > the maximum possible number and closing everything it doesn't want to
> > keep. Some time later someone came up with an optimization (read the
> > open file descriptors from /dev/fd). All of this pain and suffering is
> > caused by good old Ulrich Drepper braindamage:
> > https://sourceware.org/bugzilla/show_bug.cgi?id=10353.
> >
> > Most Linux distros have lower default file descriptor limits than
> > FreeBSD making this workaround less painful. The correct solution
> > would be to teach python3 about closefrom(2).
>
> Thank you for hint and testing!
>
> Indeed the problem is in closing more than 400,000 file descriptors in
> loop. It seems that all current versions of Python are affected.
> Python2 uses False as default value for the close_fds parameter of
> the Popen constructor, so this issue is mostly not visible.
> Python3 has changed this default to True.
>
> As Jan Bramkamp suggested, I've wrote simple patch to fix an issue (see
> attached file). It seems the problem has gone.

The attachment has been stripped out. Could you paste the diff into the message?

If this is a Python issue, shouldn't this be reported upstream, on
https://bugs.python.org ?

Dima
>
> --
> Alexander Zagrebin
> _______________________________________________
> freebsd-ports at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscribe at freebsd.org"