python 3 subprocess performance

[Alexander Zagrebin]

В Fri, 12 Apr 2019 12:41:53 +0200
Dima Pasechnik <dimpase+freebsd at gmail.com> пишет:

> > > > В Thu, 11 Apr 2019 17:32:42 +0200
> > > > Jan Bramkamp <crest at rlwinm.de> пишет:
> > > >  
> > > > > The reason is that that python does something stupid (tm). It
> > > > > tries to close all file descriptors (except a few whitelisted
> > > > > ones) up to the maximum file descriptor number. It does this
> > > > > by asking the kernel for the maximum possible number and
> > > > > closing everything it doesn't want to keep. Some time later
> > > > > someone came up with an optimization (read the open file
> > > > > descriptors from /dev/fd). All of this pain and suffering is
> > > > > caused by good old Ulrich Drepper braindamage:
> > > > > https://sourceware.org/bugzilla/show_bug.cgi?id=10353.
> > > > >
> > > > > Most Linux distros have lower default file descriptor limits
> > > > > than FreeBSD making this workaround less painful. The correct
> > > > > solution would be to teach python3 about closefrom(2).  
> > > >
> > > > Thank you for hint and testing!
> > > >
> > > > Indeed the problem is in closing more than 400,000 file
> > > > descriptors in loop. It seems that all current versions of
> > > > Python are affected. Python2 uses False as default value for
> > > > the close_fds parameter of the Popen constructor, so this issue
> > > > is mostly not visible. Python3 has changed this default to True.
> > > >
> > > > As Jan Bramkamp suggested, I've wrote simple patch to fix an
> > > > issue (see attached file). It seems the problem has gone.  
> > >
> > > The attachment has been stripped out. Could you paste the diff
> > > into the message?  
> >
> > Yes, sure.
> >
> > --- Modules/_posixsubprocess.c.orig     2018-12-24
> > 00:37:14.000000000 +0300 +++ Modules/_posixsubprocess.c
> > 2019-04-12 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@
> > _close_fds_by_brute_force(long start_fd, }
> >          start_fd = keep_fd + 1;
> >      }
> > +#if defined(__FreeBSD__)
> > +    closefrom(start_fd);
> > +#else
> >      if (start_fd <= end_fd) {
> >          for (fd_num = start_fd; fd_num < end_fd; ++fd_num) {
> >              close(fd_num);
> >          }
> >      }
> > +#endif
> >  }
> >  
> > > If this is a Python issue, shouldn't this be reported upstream, on
> > > https://bugs.python.org ?  
> >
> > May be. Rather, it is a FreeBSD-specific optimization.  
> 
> Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)),
> OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is
> there, I just checked)
> Anyway, FreeBSD Python maintainers will ask for an upstream PR.
> 
> I can do such a PR is noone else is willing to...

This would be good. Thanks!

-- 
Alexander Zagrebin