rc(8) parallel tasks

[Joe Maloney]

Thanks for sharing this.  It worked great for me on FreeBSD as well with minimal services, and does indeed make a difference.  It blew up on PCBSD which has many more services running out of box.  I also noticed /tmp/init has to be removed manually in some cases when boot hangs.  Very impressive otherwise.

Joe Maloney

> On Oct 16, 2015, at 1:37 PM, Cyril Vechera <cv at jet9.net> wrote:
> 
> On 10/16/2015 07:06 PM, Mark Felder wrote:
>> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
>>> Hi there.
>>> 
>>> We've got a small launcher script (~250 loc) for parallel services
>>> start/stop etc. It is used on our embedded systems and our users
>>> containers. And I've done a proof of concept for implanting it to the
>>> FreeBSD's standard /etc/rc for execution starting scripts in parallel.
>>> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly
>>> on eliminating jams for network or other long-latency resources waiting.
>>> 
>>> The launcher is written in pure POSIX shell and uses FIFOs (named pipes)
>>> as a mutexes for synchronization. So it is embedded into /etc/rc and
>>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it
>>> guarantees topological order (strict partial order) defined by
>>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,
>>> mkfifo and a writeable file system. Due to last requirement, it can be
>>> run on the late stage or should be supplied by some kinf of writtable
>>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder
>>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change
>>> rc.d scripts
>>> 
>>> It's not a full init replacement or a kind of services supervision tool.
>>> It only starts or invokes a group of scripts in parallel with resolving
>>> and assuring execution in dependencies order.
>>> 
>>> Please take a look at the script and patch set for FreeBSD:
>>> 
>>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init
>>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd
>>> 
>>> 
>> Your first link is a 404, but this looks really nice!
> 
> In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to 'jet9-multitask-flow' to avoid confusion with naming, because it's not as it seems, from name, an init replacement, but just a parallel task launcher. And now the actual repository URL is https://github.com/cvss/jet9-multitask-flow <https://github.com/cvss/jet9-multitask-flow>
> 
> In this commit I've complete the FreeBSD compatibility. Now a script or dependency name can include minuses `-` and dots `.` (the first stone I stumbled over was ftp-proxy).  And I've cleaned up the main script code https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow <https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow> and have split it to more functions that can be redefined. So it's now easier to rewrite dependency extraction for FreeBSD rc-scripts - current implementation is too rough and takes three 'awk' runs for each rc-script. The last is not only time loss, but as DMarck mentioned before, using awk restricts parallel rc to be run only after FILESYSTEMS stage is done. Maybe it would be better to add to the rcorder(8) some new option to dump the gathered dependencies in tsort-compatible listing and insert them directly to flow execution plan.
> 
> I've also added rc.conf variable `rc_parallel` to turn on and off parallel execution. There's a risk to discover an incomplete dependency annotations in some rc-scripts that earlier were masked by serial execution. I've done some checks by enabling as much rc.conf variables as possible to start more rc-scripts, and didn't found any error. But it looks too good to be true and I'm afraid that it's just to poor testing. I think if some ordering conflict will be found, it could be worked-around with introducing a white-list for script names that must be run only in sequentially.
> 
> So it remained first to check if it really works in different conditions.
> 
> 
> 
> 
> -- 
> Cyril Vechera
> 
> _______________________________________________
> freebsd-hackers at freebsd.org <mailto:freebsd-hackers at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers <https://lists.freebsd.org/mailman/listinfo/freebsd-hackers>
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org <mailto:freebsd-hackers-unsubscribe at freebsd.org>"