Moving /var/run and /var/lock to tmpfs in Rawhide


I hereby want to let everybody know that in the next days I will turn on
/var/run and /var/lock on tmpfs on Rawhide/F15. This is in accordance
with the following accepted F15 feature:

<a href="" title=""></a>

My current tests indicate that we will not run into too much trouble
with this and most things should continue to work just fine. However, of
course I run only a small subset of packages of the fedora archive on my
machine. So here's what might happen and which might need fixing over
the next weeks in various packages:

- Not all packages might be able to create their directory in /var/run
on start-up. Since SUSE and Ubuntu have already been shipping systems
with tmpfs on /var/run and /var/lock for quite a while I expect the
number of packages that are incapable of doing this to be very
small. If your software nonetheless fails witht this issue, then
there are two options to fix this: a) patch the program in
question, so that it is able to recreate the directories in
/var/run, or b) ship a simple drop-in file for /etc/tmpfiles.d/ which
recreates these directories on boot. (see below)

- There might be permission problems, since the rpms might have set
different perms on the subdirs of /var/run than the software itself
might apply when starting up. In this case, a drop-in file in
/etc/tmpfiles.d/ might help. (see below)

- The SELinux policy might trigger AVCs and disallow creation of the
dirs in question. In this case Dan will be of help of course, so make
sure to file a bug. And I guess I don't need to mention this but
temporarily falling back to permissive mode is a short-term workaround
for this.

- In some cases daemons might want to create more than one file/dir
below /var/run which are supposed to be labelled differently. In this
case the daemon can either be modified to fix its labels up itself, or
a drop-in file in /etc/tmpfiles.d/ might help (see below).

- Many .spec files currently own subdirs of /var/run. These need to be
updated to %ghost those dirs only, so that the automatic removal of
these files/dirs on boot doesnt cause rpm to complain. The list of packages
which own such files/subdir you find on the aforementioned feature
page. I will mass-file bugs against these packages later tonight,
requesting the %ghosting of these entries. For more information on the
%ghost directive in .spec files see this page:

<a href="" title=""></a>

Action items:

a) Lennart will mass-file bugs regarding %ghost usage tonight

b) Lennart will switch on /var/run and /var/lock on tmpfs either
tomorrow or the day after tomorrow

c) YOU need to edit your .spec file and place a %ghost where

c) YOU need to test if you package still works, and if necessary file
AVC bugs, add an /etc/tmpfiles.d drop-in file to your program, or patch
it so that it is able to recreate these directories beneath /var/run on
its own.

On /etc/tmpfiles.d:

This is a new feature of systemd, but which is apparently very much
liked by people outside of systemd, so this might actually find adoption
even on systems which will not adopt systemd any time soon, since it
actually is not specific at all to systemd. By dropping a simple
configuration file in /etc/tmpfiles.d you can ensure that volatile files
and directories are: a) created, deleted or emptied at boot b) their
permissions/ownership fixed c) its directory contents cleaned up in
regular intervals (a la tmpwatch) and d) it is properly relabeled at

As an example, here's how such a file might look like for the screen package
(name it /etc/tmpfiles.d/screen.conf):

d /var/run/screens 1777 root root 10d
d /var/run/uscreens 0755 root root 10d12h

This encodes that two directories are created under the listed names, with
automatic clean up after 10 days resp. 10 days and 12h.

For more details consult the man page:

<a href="" title=""></a>

Thank you for your attention!



Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Petr Lautrbach at 11/24/2010 - 11:45

On 11/23/2010 09:48 PM, Lennart Poettering wrote:
I think this is not needed. Files in /var/lock and /var/run are already cleaned by
rc.sysinit during boot process hence they should be already %ghost-ed now.

rpm doesn't complain on re-created directories:

# rpm -qf /var/run/abrt/
# rm -rf /var/run/abrt/
# rpm -qV abrt
missing /var/run/abrt
# mkdir /var/run/abrt/
# rpm -qV abrt

So it should be sufficient to include them in /etc/tmpfiles.d/...conf and leave them as regular dirs in package.

This would have minimal impact to changes in .spec files (no new scriplets needed) and also to configurations
without tmpfs on /var/{run,lock}


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Wouters at 11/24/2010 - 12:07

This remark makes no sense? If they "already" needed ghosting, then the mass-file should
be needed?

It would be helpful if we got a clear signal what to do with our initscripts/spec files
to minimize package changes....


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/24/2010 - 12:33

On 24/11/10 16:07, Paul Wouters wrote:
Files are directories are currently treated differently. The initscripts
clean out files from /var/lock and /var/run but leave directories alone.

So any package containing a file in these directories should already
have it marked as %ghost as it will already disappear at boot time.

However, most affected packages probably have directories rather than
files here, and *those* shouldn't need %ghost-ing because re-creating
them using a tmpfiles.d/*.conf file should be enough to keep rpm happy.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Wouters at 11/24/2010 - 12:39

Thanks for the clarification.

Is that needed if the package init script deals with this already?
(eg xl2tpd will create /var/run/xl2tpd if it does not exist)


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/24/2010 - 12:43

On 24/11/10 16:39, Paul Wouters wrote:
If the initscript already does it then that should be fine.

But Lennart prefers the tmpfiles.d approach as it's less cumbersome:
<a href="" title=""></a>


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Wouters at 11/27/2010 - 21:52

But the initscript needs the support anyway, if it is to support non-systemd
systems. And as an upstream, I'm trying to keep to a single init script where
possible. (and not have one for each branch of fedora)

Adding/supporting systemd files in /etc just for creating /var/run/packagename
does seem the ideal solution to me.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/28/2010 - 05:29

On Sat, 27 Nov 2010 20:52:13 -0500 (EST)

Yes, that's what I'm doing.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By =?ISO-8859-1?Q?... at 11/24/2010 - 09:01

On 23/11/10 20:48, Lennart Poettering wrote:
We run stateless systems here and both of the above
directories are listed in the default /etc/rwtab.
We've not noticed any issues with that
(note we disable selinux).


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Akira TAGOH at 11/24/2010 - 05:16

This may be not a question for you but just wonder..

LP> <a href="" title=""></a>

That sounds like creating a directory at the boot time
though, does this mean rebooting are required to get it
working anyway? even though it worked without rebooting
before, isn't it a kind of regression? or am I missing
something I can do in the scriptlet to create the directory

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Till Maas at 11/23/2010 - 18:02

The release notes section contains this:
| /var/run and /var/lock are now mounted from tmpfs, and hence emptied on
| reboot. Applications must ensure to recreate their own files/dirs on
| startup, and cannot rely that doing this at package installtion will
| suffice

But this does not mention tmpfiles.d which you wrote can be used instead
of creating files or dirs on startup.

Imho there should be a packaging guideline to make it clear what needs
to be done in which cases. E.g. when to %ghost files and when not.

Removing /var/run/screens after 10 days sounds wrong. Even removing the
sockets inside /var/run/screens sounds wrong. Is this just a bad example
am I not understanding the age means.

Here it says:
| If a file or directory is older than the current time minus the age
| field it is deleted.

And when are the files and dirs created? Only when the system is booted?
But then after installing an package that requires files to be created
by tmpfiles.d the system needs to be rebooted before it can be used. Or
will rpm call something that parses the appropriate tmpfiles.d file when
the package is installed / updated?


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Nathanael D. Noblet at 11/23/2010 - 19:56

On 11/23/2010 03:02 PM, Till Maas wrote:
I'm curious, one of my packages has /var/run/dspam in the specfile and
/var/lock/subsystem/dspam used in the initscript... I added a %ghost for
the second and now I'm wondering if I even need to.. Should I revert
that change? and just %ghost /var/run/dspam ?

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/24/2010 - 11:00

You probably should %ghost all dirs/files beneath those two directories.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Nathanael D. Noblet at 11/24/2010 - 13:45

On 11/24/2010 08:00 AM, Lennart Poettering wrote:


%ghost /var/run
%ghost /var/run/dspam
%ghost /var/lock
%ghost /var/lock/subsystem
%ghost /var/lock/subsystem/dspam


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/24/2010 - 15:30

No. /var/run and /var/lock and /var/lock/subsystem is not owned by
you. Don't add those three to the spec file!


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/23/2010 - 19:01

Added a comment about this now.

I guess extending the guidelines with a line or two about this is a good idea.

Sorry, it's not necessarily a great example.

The aging stuff is mostly useful for things like /tmp itself.


Hmm, it has been suggested that we should make it possible to create
these dirs in the .spec files by invoking the systemd-tmpfiles tool
directly from the scriptlets. I guess we should add a nice interface for
that. In the meantime it should be sufficient to simply place th right
"mkdir -p -m ..." in the scriptlet. Of course it would be desirable if
we have a single place where the dirs to create are encoded.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/24/2010 - 07:13

On 23/11/10 23:01, Lennart Poettering wrote:
Why not create the directories in the initscript/systemd equivalent?


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/24/2010 - 10:53

Because it's cumbersome and you need at least three invocations to get
things right, to do mkdir, chown and restorecon.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Toshio Kuratomi at 11/24/2010 - 07:02

On Wed, Nov 24, 2010 at 12:01:45AM +0100, Lennart Poettering wrote:

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Wouters at 11/29/2010 - 13:21

So with all of this, as a package maintainer, I will have to make sure the
dirs exist in the init scripts anyway. In which case one can wonder why
bothering with tmpfiles.d files (or %ghost). Just to cleanup a volatile directory in
case a package is uninstalled after start?


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Toshio Kuratomi at 11/30/2010 - 02:52

On Mon, Nov 29, 2010 at 12:21:08PM -0500, Paul Wouters wrote:
* after a reboot, the application is able to startup and write to a directory
in /var/run and/or /var/lock.
* The sysadmin would like to be able to see who owns the directories and
lock files in /var/run and/or /var/lock so rpm -qf /var/run/foo/ should
tell them that.

corner cases:
* After installation but before reboot, the application is able to startup
and write to a directory in /var/run and/or /var/lock
* After removal but before reboot, the directories that aren't needed are
cleaned up from /var/run and /var/lock

The first of these corner cases is much more important than the second of

So with all this, we know a few things:

1) The rpm metadata has to carry information about the directories (and
should for files as well) inside of /var/run and /var/lock. To me we should
just put the directories in per normal and %ghost any files (which is what
we should be doing already but probably aren't always).

2) The act of installing the rpm should create the necessary directories.
Alternately, the program (or as you say, the init script) can create the
necessary directories. Note that I don't believe that systemd gives you the
flexibility to do that sort of thing (there's no "script" in its init stuff)
so you'd need a wrapper script for the program itself or write a patch to
the program itself to achieve this where the program doesn't create the
directory already and if we don't do this from within the rpm payload.

3) We have to use tmpfiles.d to create the directories on reboot.

Of these, only %ghosting of files (not directories) is only about cleaning
up the directories. The rest can cause the service to fail to startup.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Wouters at 11/30/2010 - 15:25

All daemons should already be able to do that (meaning init scripts dealing
with non-existing directories)

Sure, and that's a difference. My daemons own their dirs, and it is suggested
to %ghost these (and any potential files, though files I'd say that since
they're in a very specific directory, a sysadmin already knows what daemon
they belong to)

Handled with the sam initscript code that should already exist.

This will also be covered already if the spec file has these directories
listed (ghosted or not). Daemons tend to remove all the files in these dirs already on stop, which is called before removal anyway.

I don't see too much value on ghosting files, but no objection either.


Why? If the initscripts do this already it should be fine. The only reason
I've heard so far is to do selinux context items, which I'm mostly
unfamiliar with (but would hope that most of the required permissions on
those are inherited from the parent directory policy?)

I would really like to avoid having THREE places to create directories
in /var/run and /var/lock, those being spec file, init scripts AND tmpfiles.d


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Tomasz Torcz at 11/30/2010 - 15:48

On Tue, Nov 30, 2010 at 02:25:50PM -0500, Paul Wouters wrote:

Scratch the initscript. This would mean initscript would need to
contain multiple
ExecStartPre=/sbin/mkdir --mode=777 /var/run/xx; /bin/chown x.x /var/run/xx; /sbin/restorecon /var/run/xx
lines, which look unwieldy.
So we are left with tmpfiles.d and spec file. Could the spec file be replaced
by tmpfilesd invocation in %post?

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Wouters at 11/30/2010 - 17:10

why not

mkdir -p /var/run/xx


[ ! -d /var/run/xx ] && mkdir -p /var/run/xx

Can't selinux pickup things without a restorecon? And what is the problem another (root)
process screwing over a pid or lock file? Can't SElinux lock that down from the /var/run

I'm still not convinced of tmpfiles.d. I'd like a package to be as system-start-system agnostic
as possible. I don't want to know if it is systemd or upstart or initng or whatnot.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/30/2010 - 17:17

On Tue, 30 Nov 2010 16:10:20 -0500 (EST)

/var/run is var_run_t in targeted policy, but hardly anything below
/var/run is - almost every subdir/file has its own context type.

Just creating a file/directory within /var/run using the initscript will
inherit the var_run_t, which in most cases is not what's needed, hence
the need for restorecon.

Having the daemon create the file/dir works better because there will
be a type transition defined in policy that results in the correct
context type being used.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Tom Lane at 11/30/2010 - 18:01

Paul Howarth <> writes:
That comment suggests you don't even understand the reason why those
subdirectories exist. It's this: the daemons do not, and should not,
run with the root privileges needed to create things directly in
/var/run. The point of a subdirectory is to be owned by the
lower-privilege account under which the particular daemon is running.
If the subdir has to be remade at runtime, that has to be done by the
root-privilege initscript, because /var/run is only writable by root.

I agree with Paul W that it's not obvious why selinux can't get this
right for us, instead of requiring an extra easy-to-forget step in the
initscripts. But on the whole I'd rather this responsibility weren't
getting dropped on the initscripts at all. It works perfectly well to
set up these dirs once via RPM --- why shouldn't we make the tmpfs
creation process responsible for cloning the directory structure from

regards, tom lane

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Joe Nall at 11/30/2010 - 20:45

I was nodding my head in agreement reading this paragraph, and then I
looked at my development box. Only avahi-daemon and hald follow this
pattern in my /var/run (which I'm sure is not a complete sample).


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Garry T. Williams at 11/30/2010 - 21:08

On Tuesday, November 30, 2010 19:45:11 Joe Nall wrote:
Well, the distinct types are there.

Plus, my /var/run has directories owned by:


Also, the group owners are different from root on:


I'm still nodding my head in agreement. :-)

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Daniel J Walsh at 11/30/2010 - 18:33

On 11/30/2010 05:01 PM, Tom Lane wrote:

SELinux can label a newly created directory in one of three ways.


Either it adopts the label of the parent directory, in this case
var_run_t. This is the most common what directories and files get
labeled, and has been stated is wrong.

A second way a directory can be labeled is on transition rules. We
write policy that says Process labeled a_t creating a directory in a
directory labeled b_t will get the label c_t.

Setroubleshoot for example has a rule that says

setroubleshootd_t creating directories in directories labeled var_run_t
will create the directory labeled setroubleshoot_var_run_t.

The problem is this only works for one combination, you can not have
another rule that says

a_t creating directories in b_t will create then as d_t.

The last way is to have the application have SELinux intelligence built
in. systemd-tmpfiles has this, udev has this, passwd has this, most
apps do not, and we do not want most apps to do this.

So if you create a directory in the postinstall of an rpm, the directory
will be created as var_run_t (rule 1), rpm has SELinux intelligence
built in, but since you did this in postinstall, rpm command does not
know you did it. You will have to run restorecon.

If you create this directory in an /etc/init.d/foobar, this will be
running as initrc_t, and will create the directory as var_run_t, and you
will have to do restorecon.

If you do this with systemd-tmpfiles or do not ghost it in rpm spec
file, the correct thing will happen.

As far as remembering to run restorecon in init scripts or postinstalls.
You have to run chcon and chown in the scripts often also, to make sure
the DAC LABEL is correct. SELinux has been around for eight years now,
none of this is rocket science.

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Frank Ch. Eigler at 12/01/2010 - 00:38

Daniel J Walsh < ... at redhat dot com> writes:

If the directories were listed in %files, as %ghost or whatever, rpm
could figure it out, no? Could we have a single early rpm-based
initscript that creates all /var/run & /var/lock subdirs owned by
installed packages, and selinux-labels them?

- FChE

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/30/2010 - 18:25

On Tue, 30 Nov 2010 17:01:16 -0500

Except for the cases where the daemon starts as root in order to do
things like bind to privileged ports, create subdir under /var/run for
its own purposes, write a pidfile to /var/run etc. and then drop
privileges like a good daemon should...


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/24/2010 - 10:58

I don't think this will be necessary since only a small subset of
services will need this treatment.

I have mentioned it a couple of times, but I will do so here again:
OpenSUSE and Ubuntu have been shipping their systems like this since
quite some time, as do we apparently with the stateless stuff. Most
software has already been fixed to properly create those subdirs on its
own. That's why adding tmpfiles drop-ins will be necessary only in

Well, because rpm has introduced %ghost for cases like this, and everybody
else uses it for that.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Toshio Kuratomi at 11/24/2010 - 17:39

On Wed, Nov 24, 2010 at 03:58:26PM +0100, Lennart Poettering wrote:
SUSe may have realized that too as they stopped %ghosting the /var/run/httpd
directory (but I can't see the bug that is referenced there ( 498490 ) so
I don't know for sure. OTOH, they didn't make the change across their
packageset as avahi still has the directory %ghosted.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By James Antill at 11/24/2010 - 18:07

On Wed, 2010-11-24 at 13:39 -0800, Toshio Kuratomi wrote:
It's also worth noting that %ghost tells rpm -V that it's ok if the
file/dir. is missing (or changes type) ... which we _don't_ want to

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Paul Howarth at 11/23/2010 - 17:19

On Tue, 23 Nov 2010 21:48:30 +0100
Lennart Poettering < ... at 0pointer dot de> wrote:
Given that the tmpfiles.d format doesn't mention SELinux contexts, I
assume that the files/directories will be set up to have whatever their
default context would be under the running policy, as restorecon would
set it?


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/23/2010 - 17:26

Yes, SELinux contexts are exclusively configured in the policy, we do
not spread that around in other ocnfiguration files.

The tmpfiles stuff includes an implicit restorecon, basically.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Daniel J Walsh at 11/23/2010 - 17:41

On 11/23/2010 04:26 PM, Lennart Poettering wrote:
/var/run/BLAH might have different labels in MLS policy versus Targeted.
And some of our partners ship their own policies.

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Doug Ledford at 11/23/2010 - 17:12

On 11/23/2010 03:48 PM, Lennart Poettering wrote:
Will the tmpfs mounts be available in the initramfs, or only on the
running system?

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/23/2010 - 17:32

Since /var/run is a subdir of /var which might be separate file system
it is difficult mounting /var/run before /var. That means that it won't
be available in the intird.

(Yes, one can do stuff with show-through mount hierachies, that would
allow replacing /var later on, but I am not a fan of such hackery.)

Also note that by now it's somewhat standard that code that needs to be
run as part of early boot creates a subdir in /dev, such as /dev/.udev
or /dev/.systemd. Not super-pretty, but I guess it's too late to
complain about that.

Ideally the FHS would have never specified that /var/run and /var/lock
would lie beneath /var, but I guess that's too late now.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Doug Ledford at 11/29/2010 - 17:43

On 11/23/2010 04:32 PM, Lennart Poettering wrote:
Hackery is in the eye of the beholder.

Those places all exist *because* no one took the time to create an
initrd managed writable /var/run or /var/lock. Using their existence to
justify continuing down a path that places files where they don't belong
is faulty logic. I took a *major* beating by the upstream mdadm
maintainer over the fact that we are putting files in /dev/ that don't
belong there. And he's right, we *shouldn't* be putting those files
there. Continuing a band aid hack because someone did it initially is
not the right course of action.

One failed specification isn't really a good reason to justify creating
a de facto second failed specification.

Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Till Maas at 11/23/2010 - 17:44

This is the first time I read about using a subdir in /dev. Is there
some inter-distro agreement about this? The last discussion I had with
someone from debian revealed that they use subdirs in /lib for stuff
that should be in /var according to the FHS but might be needed before
/var is mounted.


Re: Moving /var/run and /var/lock to tmpfs in Rawhide

By Lennart Poettering at 11/23/2010 - 17:57

Well it's not really inter-distro. It mostly inter-maintainers-of-
packages-that-are-involved-with-early-boot. I know that at least some
storage stuff also puts stuff there, as will most likely u-l-ng to place
meta information about mounts.

Yes, Debian has /lib/init/rw, and we shortly considered adopting that as
suggested default in systemd, but we decided not to do that since more
software was already using /dev/.foo/ than /lib/init/rw (which in fact
is used only by debian specific scripts afaik at this point), and we
didn't want to add yet another fs just for this purpose to the mix.

I talked to some Debian guys about this and they have ben thinking about
moving /lib/init/rw to /dev/.foobar/ too.

I think everybody agrees that /dev/.foobar/ isn't particularly
pretty. But given that /dev is some kind of tmpfs these days this should
be fine.

I don't think it is really necessary to make /dev/.foobar/ some kind of
standard however. As the number of services involved with early boot and
which need runtime files like this is very limited it should be
sufficient to simply come to an agreement with the various maintainers
involved, which is a small group.