DevHeads.net

Boot-related updates in raring: forwarding upstart support to Debian

Hi folks,

Now that Debian jessie is open for development, and a few packages
(lsb/insserv/sysvinit/debhelper) have been updated in saucy, at long last
there is an implementation of upstart support in Debian that's compatible
with Ubuntu.

This means that it's now possible to forward our patches for upstart job
support to Debian! When doing merges this cycle on packages that ship
upstart jobs, I would encourage you to look at whether these are in a state
to be forwarded upstream.

Before forwarding upstart jobs, there are a few things to be aware of. The
tl;dr version of this for those who just want to start fixing packages and
forwarding patches can be found at
<https://wiki.ubuntu.com/UpstartCompatibleInitScripts>.

- The new, converged upstart support drops the "upstart-job" virtual
package, which never worked as intended. Instead, as per Debian Policy
9.11.1, packages will now ship both an init script and an upstart job
instead of installing a /lib/init/upstart-job symlink in /etc/init.d.

- This means that the warning messages from /lib/init/upstart-job about not
calling "/etc/init.d/$foo" are now no longer warnings: it is an *error* to
call /etc/init.d/$foo for a service that has an upstart job. In saucy,
the invoke-rc.d and service scripts will both correctly map to either
upstart or sysvinit as appropriate. Admins should take care to use the
"service" interface (supported since the current upstart was introduced in
Ubuntu 9.10), and packages should continue to use 'invoke-rc.d'. Packages
built without using debhelper should manually add a versioned dependency
on 'sysv-rc (>= 2.88dsf-24)' for proper invoke-rc.d support.

- Also per Debian policy, and because of the previous point, init scripts
for packages that also have upstart jobs need to *do nothing* when running
under upstart:

"SysV init scripts for which an equivalent upstart job is available must
query the output of the command `initctl version' for the string
`upstart' and avoid running in favor of the native upstart job, using a
test such as this:
if [ "$1" = start ] && which initctl >/dev/null && initctl version | grep -q upstart
then
exit 1
fi
"

If your init script uses the /lib/lsb/init-functions shell library from
lsb-base, there is a new function, init_is_upstart, which you can use to
do this more simply:

if init_is_upstart; then
exit 1
fi

Init scripts should exit non-zero when called with arguments of 'start',
'restart', or 'force-reload'. When called with an argument of 'stop', it
is recommended to 'exit 0' instead.

Use of 'init_is_upstart' requires a versioned dependency on
'lsb-base (>= 4.1+Debian3)', which you will need to add manually to your
package.

- Since the new init script will be a no-op, packages which normally restart
in the postinst (instead of stopping in the prerm and starting in the
postinst) *MUST* handle stopping the service in their preinst when
upgrading from a pre-upstart-capable version. See the Debian udev package
for an example of this.

And that's it. Your assistance in reducing this delta with Debian is
appreciated. Currently in Ubuntu, we have 301 upstart jobs across 186
packages; my goal is that, by the end of June, patches will be in the Debian
BTS for any of these packages that are available in Debian.

At a later date, I will also be revisiting the question of enabling
insserv/startpar by default in Ubuntu for the remaining sysvinit scripts.
This will further reduce our delta with Debian, improve boot speed for any
systems using significant numbers of non-upstarted packages, and ensure we
avoid further bugs like bug #858122 as a result of incompatibilities with
insserv.

Cheers,

Comments

Re: Boot-related updates in raring: forwarding upstart support t

By Dmitrijs Ledkovs at 06/12/2013 - 10:58

On 19 May 2013 02:14, Steve Langasek <steve. ... at ubuntu dot com> wrote:
I've converted a non-trivial scripts into upstart compatible scripts.
And I have a question about exact requirements behind:

"Make sure that the upstart jobs and init scripts in the package have
matching names. If the names do not match, either the init script
should be split (cf. /etc/init.d/samba->/etc/init.d{s,n}mbd in the
samba package), or dummy upstart jobs should be created (cf.
/etc/init/mountall.sh.conf in the mountall package) so that the names
line up. Without this, insserv will not correctly detect that the
upstart job has started and dependency-based booting will fail."

What are the exact requirements here? one-to-one matching?
For example in util-linux package there are:
- in debian: hwclock.sh init script
- in ubuntu: hwclock and hwclock-save upstart jobs

I have added the two upstart jobs and created an "dummy" hwclock.sh
upstat job with "start on started hwclock" condition, thus to hint
insserv that upstart can handle starting "hwclock.sh" correctly.
Do I still have to introduce hwclock & hwclock-save init.d scripts as
well? Or does one only need such compat scripts if there are other
sysv init scripts that declare dependency on "hwclock.sh"?
I guess I need to read up on startpar.

Poking around dh_installinit only revealed that correct code to
transition from upstart-job symlink to a normal file is added in the
preinst.

Regards,

Dmitrijs.

Re: Boot-related updates in raring: forwarding upstart support t

By Steve Langasek at 06/12/2013 - 19:32

On Wed, Jun 12, 2013 at 04:58:00PM +0100, Dmitrijs Ledkovs wrote:
So the precise requirements for insserv compatibility are that if a SysV
init script should be skipped in favor of an upstart job when running under
upstart, there must be an upstart job with the exact same name as the init
script to be skipped.

Combined with the requirement in Policy 9.11 that packages *must* continue
to support SysV init in Debian and not be upstart-only, this means in
practice that almost all packages should have a 1:1 mapping between sysvinit
scripts and upstart jobs.

However, one key exception are packages that implement the init system
itself. As policy puts it:

An exception to this rule is scripts or jobs provided by the init
implementation itself; such jobs may be required for an
implementation-specific equivalent of the `/etc/rcS.d/' scripts and may
not have a one-to-one correspondence with the init scripts.

hwclock isn't part of an init system per se, but the hwclock.sh script *is*
shipped in /etc/rcS.d. So it's not surprising that these don't end up being
1:1.

Yep - that sounds exactly like what I did in the mountall package.

You don't need them, at all. Any init scripts that *do* declare a
dependency on 'hwclock.sh' will have this dependency satisfied by the
hwclock.sh upstart job.

Cheers,

Re: Boot-related updates in raring: forwarding upstart support t

By Colin Watson at 05/22/2013 - 06:06

On Sat, May 18, 2013 at 06:14:26PM -0700, Steve Langasek wrote:
Thanks for this. I've converted the old approaches used in openssh and
binfmt-support to this new style, simplifying the packaging in both
cases.

This isn't in the wiki page, and so I missed it the first time round; it
would be helpful to add this there as well.

Do you have a list anywhere of which packages contain jobs that need to
be forwarded, perhaps run through dd-list? It seems to me that this
would be a helpful thing to maintain centrally; and for example we could
probably deal with most orphaned packages rather quickly.

Re: Boot-related updates in raring: forwarding upstart support t

By Steve Langasek at 05/30/2013 - 17:53

On Wed, May 22, 2013 at 12:06:31PM +0100, Colin Watson wrote:

Added, thanks.

Attached. List generated with the following, then piped to dd-list:

zgrep etc/init/ ~/ubuntu/dists/saucy/Contents-i386.gz \
| sed -e's/,/ /g; s,\(^\| \)[^ ]*/, ,g' | while read job pkgs; do
for pkg in $pkgs; do
script=etc/init.d/$(basename $job .conf)
pkgfile=$(zcat ~/ubuntu/dists/saucy/*/binary-i386/Packages.gz \
| grep-dctrl -FPackage -sFilename -n -X $pkg)
dpkg -c ubuntu/$pkgfile | grep -q "$script$" || echo $pkg
done
done | grep -vE 'mountall|upstart' | sort -u

(Ignoring errors from packages not known in Debian - those of course don't
need to have changes forwarded.)

Re: Boot-related updates in raring: forwarding upstart support t

By Clint Byrum at 05/18/2013 - 20:51

On 2013-05-18 18:14, Steve Langasek wrote:
\o/

There's a side note to this which is rather complicated. Most upstart
jobs should just use 'start on runlevel [2345]' and 'stop on runlevel
[016]'. However, for those that use events, great care must be taken to
make sure those events are stable in all upstart using distributions. I
feel that we should document this fact and encourage users to submit any
new events they use to <a href="mailto:upstart- ... at lists dot ubuntu.com">upstart- ... at lists dot ubuntu.com</a>.

Perhaps we should consider moving debian/manpages/upstart-events.7 to
the upstream code base to try and curb too much diversion.

Re: Boot-related updates in raring: forwarding upstart support t

By Steve Langasek at 05/19/2013 - 00:41

Hi Clint,

On Sat, May 18, 2013 at 06:51:55PM -0700, Clint Byrum wrote:
I think that's a reasonable approach. Though FWIW, the upstart package in
Debian is kept closely in sync with the Ubuntu package, and there's not much
risk of drift.

I'm less sure of this, as currently these events are only in Debian and
Ubuntu (and derivatives) and there are other users of upstart who *don't*
use these events.