freebsd-rc Digest, Vol 470, Issue 1

Sat Oct 17 12:25:19 UTC 2015

freebsd-rc mailing list submissions to
freebsd-rc at freebsd.org

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.freebsd.org/mailman/listinfo/freebsd-rc
or, via email, send a message with subject or body 'help' to
freebsd-rc-request at freebsd.org

You can reach the person managing the list at
freebsd-rc-owner at freebsd.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of freebsd-rc digest..."

Today's Topics:

1. Re: rc(8) parallel tasks (Mark Felder)
2. Re: rc(8) parallel tasks (Cyril Vechera)
3. Re: rc(8) parallel tasks (Mark Martinec)

----------------------------------------------------------------------

Message: 1
Date: Fri, 16 Oct 2015 11:06:01 -0500
From: Mark Felder <feld at FreeBSD.org>
To: Cyril Vechera <cv at jet9.net>, dougb at FreeBSD.org,
freebsd-hackers at freebsd.org, freebsd-rc at freebsd.org
Subject: Re: rc(8) parallel tasks
Message-ID:
<1445011561.1233840.412174489.688C5822 at webmail.messagingengine.com>
Content-Type: text/plain

On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
> Hi there.
> 
> We've got a small launcher script (~250 loc) for parallel services 
> start/stop etc. It is used on our embedded systems and our users 
> containers. And I've done a proof of concept for implanting it to the 
> FreeBSD's standard /etc/rc for execution starting scripts in parallel. 
> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly 
> on eliminating jams for network or other long-latency resources waiting.
> 
> The launcher is written in pure POSIX shell and uses FIFOs (named pipes) 
> as a mutexes for synchronization. So it is embedded into /etc/rc and 
> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it 
> guarantees topological order (strict partial order) defined by 
> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, 
> mkfifo and a writeable file system. Due to last requirement, it can be 
> run on the late stage or should be supplied by some kinf of writtable 
> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder 
> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change 
> rc.d scripts
> 
> It's not a full init replacement or a kind of services supervision tool. 
> It only starts or invokes a group of scripts in parallel with resolving 
> and assuring execution in dependencies order.
> 
> Please take a look at the script and patch set for FreeBSD:
> 
> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init
> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd
> 
> 

Your first link is a 404, but this looks really nice!

-- 
Mark Felder
ports-secteam member
feld at FreeBSD.org

------------------------------

Message: 2
Date: Fri, 16 Oct 2015 21:37:27 +0300
From: Cyril Vechera <cv at jet9.net>
To: Mark Felder <feld at FreeBSD.org>
Cc: marck at rinet.ru, freebsd-hackers at freebsd.org,
freebsd-rc at freebsd.org
Subject: Re: rc(8) parallel tasks
Message-ID: <562143E7.3030104 at jet9.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 10/16/2015 07:06 PM, Mark Felder wrote:
> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
>> Hi there.
>>
>> We've got a small launcher script (~250 loc) for parallel services
>> start/stop etc. It is used on our embedded systems and our users
>> containers. And I've done a proof of concept for implanting it to the
>> FreeBSD's standard /etc/rc for execution starting scripts in parallel.
>> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly
>> on eliminating jams for network or other long-latency resources waiting.
>>
>> The launcher is written in pure POSIX shell and uses FIFOs (named pipes)
>> as a mutexes for synchronization. So it is embedded into /etc/rc and
>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it
>> guarantees topological order (strict partial order) defined by
>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,
>> mkfifo and a writeable file system. Due to last requirement, it can be
>> run on the late stage or should be supplied by some kinf of writtable
>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder
>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change
>> rc.d scripts
>>
>> It's not a full init replacement or a kind of services supervision tool.
>> It only starts or invokes a group of scripts in parallel with resolving
>> and assuring execution in dependencies order.
>>
>> Please take a look at the script and patch set for FreeBSD:
>>
>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init
>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd
>>
>>
> Your first link is a 404, but this looks really nice!

In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to 
'jet9-multitask-flow' to avoid confusion with naming, because it's not 
as it seems, from name, an init replacement, but just a parallel task 
launcher. And now the actual repository URL is 
https://github.com/cvss/jet9-multitask-flow

In this commit I've complete the FreeBSD compatibility. Now a script or 
dependency name can include minuses `-` and dots `.` (the first stone I 
stumbled over was ftp-proxy). And I've cleaned up the main script code 
https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and have 
split it to more functions that can be redefined. So it's now easier to 
rewrite dependency extraction for FreeBSD rc-scripts - current 
implementation is too rough and takes three 'awk' runs for each 
rc-script. The last is not only time loss, but as DMarck mentioned 
before, using awk restricts parallel rc to be run only after FILESYSTEMS 
stage is done. Maybe it would be better to add to the rcorder(8) some 
new option to dump the gathered dependencies in tsort-compatible listing 
and insert them directly to flow execution plan.

I've also added rc.conf variable `rc_parallel` to turn on and off 
parallel execution. There's a risk to discover an incomplete dependency 
annotations in some rc-scripts that earlier were masked by serial 
execution. I've done some checks by enabling as much rc.conf variables 
as possible to start more rc-scripts, and didn't found any error. But it 
looks too good to be true and I'm afraid that it's just to poor testing. 
I think if some ordering conflict will be found, it could be 
worked-around with introducing a white-list for script names that must 
be run only in sequentially.

So it remained first to check if it really works in different conditions.

-- 
Cyril Vechera

------------------------------

Message: 3
Date: Sat, 17 Oct 2015 00:14:01 +0200
From: Mark Martinec <Mark.Martinec+freebsd at ijs.si>
To: freebsd-hackers at freebsd.org, freebsd-rc at freebsd.org
Cc: Cyril Vechera <cv at jet9.net>
Subject: Re: rc(8) parallel tasks
Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4 at mailbox.ijs.si>
Content-Type: text/plain; charset=US-ASCII; format=flowed

Tried a today's git checkout on one of our small virtual 10.2 boxes
with not many services. The installation is almost trivial and a
reboot went without any problems. Time for a reboot was shrunk by
about 10 seconds (which is understandable with a small number of
services). Very good, almost too good to be true for such fairly
small and non-intrusive piece of shell/awk scripting - thanks!

Mark

2015-10-16 20:37, Cyril Vechera wrote:
> On 10/16/2015 07:06 PM, Mark Felder wrote:
>> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
>>> Hi there.
>>> 
>>> We've got a small launcher script (~250 loc) for parallel services
>>> start/stop etc. It is used on our embedded systems and our users
>>> containers. And I've done a proof of concept for implanting it to the
>>> FreeBSD's standard /etc/rc for execution starting scripts in 
>>> parallel.
>>> It gave me a boot time reduction of rc part from 27 to 7 seconds, 
>>> mostly
>>> on eliminating jams for network or other long-latency resources 
>>> waiting.
>>> 
>>> The launcher is written in pure POSIX shell and uses FIFOs (named 
>>> pipes)
>>> as a mutexes for synchronization. So it is embedded into /etc/rc and
>>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it
>>> guarantees topological order (strict partial order) defined by
>>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,
>>> mkfifo and a writeable file system. Due to last requirement, it can 
>>> be
>>> run on the late stage or should be supplied by some kinf of writtable
>>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder
>>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to 
>>> change
>>> rc.d scripts
>>> 
>>> It's not a full init replacement or a kind of services supervision 
>>> tool.
>>> It only starts or invokes a group of scripts in parallel with 
>>> resolving
>>> and assuring execution in dependencies order.
>>> 
>>> Please take a look at the script and patch set for FreeBSD:
>>> 
>>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init
>>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd
>>> 
>>> 
>> Your first link is a 404, but this looks really nice!
> 
> In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to
> 'jet9-multitask-flow' to avoid confusion with naming, because it's not
> as it seems, from name, an init replacement, but just a parallel task
> launcher. And now the actual repository URL is
> https://github.com/cvss/jet9-multitask-flow
> 
> In this commit I've complete the FreeBSD compatibility. Now a script
> or dependency name can include minuses `-` and dots `.` (the first
> stone I stumbled over was ftp-proxy). And I've cleaned up the main
> script code
> https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and
> have split it to more functions that can be redefined. So it's now
> easier to rewrite dependency extraction for FreeBSD rc-scripts -
> current implementation is too rough and takes three 'awk' runs for
> each rc-script. The last is not only time loss, but as DMarck
> mentioned before, using awk restricts parallel rc to be run only after
> FILESYSTEMS stage is done. Maybe it would be better to add to the
> rcorder(8) some new option to dump the gathered dependencies in
> tsort-compatible listing and insert them directly to flow execution
> plan.
> 
> I've also added rc.conf variable `rc_parallel` to turn on and off
> parallel execution. There's a risk to discover an incomplete
> dependency annotations in some rc-scripts that earlier were masked by
> serial execution. I've done some checks by enabling as much rc.conf
> variables as possible to start more rc-scripts, and didn't found any
> error. But it looks too good to be true and I'm afraid that it's just
> to poor testing. I think if some ordering conflict will be found, it
> could be worked-around with introducing a white-list for script names
> that must be run only in sequentially.
> 
> So it remained first to check if it really works in different 
> conditions.

------------------------------

Subject: Digest Footer

_______________________________________________
freebsd-rc at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-rc
To unsubscribe, send any mail to "freebsd-rc-unsubscribe at freebsd.org"

------------------------------

End of freebsd-rc Digest, Vol 470, Issue 1
******************************************