It's a common complaint that it's too difficult to get updates to
critpath packages through the update system at the moment. We've been
looking into trying to make that easier without just dropping the
critpath requirements, and one thing we looked at was whether the
requirement for positive karma from proventesters was a net benefit.
Thankfully this is the kind of thing that we can actually generate
numbers for. Luke pulled some statistics which are available at
<a href="http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html" title="http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html">http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html</a> .
The relevant section here is the set of packages that have (a)
sufficient positive karma to be pushed, but (b) negative proventester
karma - that is, the packages where negative proventester karma
prevented a push.
Straight off, we can see that these amount to 1-2% of all critpath
updates. It's simply not common for proventester to make a difference to
the outcome. If we look at the individual packages, things get even more
interesting. Many of the updates receive a mixture of proventester
karma, so even with the negative the push would still go ahead. As far
as I can tell:
<a href="https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14" title="https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14">https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14</a>
<a href="https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14" title="https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14">https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14</a>
are the only two updates where the proventester karma requirement would
have made a difference, out of 1942 critpath updates that made it to
stable. That doesn't seem like a great hit rate.
So, assuming I'm not grossly misanalysing the data, it seems that we
could drop the proventester requirement from critical path updates with
a negligable change in the quality of the updates. Thoughts?
Comments
Re: Rethinking proventester and critpath
By Adam Williamson at 11/21/2011 - 14:32On Tue, 2011-11-01 at 13:59 +0000, Matthew Garrett wrote:
So, we discussed this at the QA meeting a couple of weeks back.
Initially the plan was for anyone who had concerns to reply to this
mail, but that doesn't seem to have happened, so I will try to summarize
the various responses from that meeting. You can check the full summary
and log of the meeting:
<a href="https://fedoraproject.org/wiki/QA/Meetings/20111107" title="https://fedoraproject.org/wiki/QA/Meetings/20111107">https://fedoraproject.org/wiki/QA/Meetings/20111107</a>
if you want to make sure I'm not misrepresenting anything.
adamw: 1-2% of all critpath updates (where proventester process is seen
to have 'had an impact') is a small number, but it is not 0. It only
takes *one* really bad update to cause big problems.
adamw: it's possible we could use proven tester feedback more
effectively: there have been cases where bad updates went out despite
negative feedback, because we currently don't give negative feedback -
even from proventesters - much significance at all. So there may have
been more cases where proventester input *may have been significant*
with a system where negative karma is more strongly considered. e.g.
updates could be blocked from being auto-pushed if there is any negative
feedback from a proven tester
red_alert (sandro mathys): critpath packages should have detailed test
plans, as currently proventesters often do not know exactly what they
need to verify with some critpath packages: "the process as its done
right now doesn't work (i.e. is no improvement to not having pts) but
having pts with a better process would surely add to the overall
quality"
tflink: we have to keep in mind the reason for the proposal -
maintainers frustrated that packages get stuck in updates-testing for
weeks - is a valid reason and a significant problem. "if we object to
getting rid of the proventester process, do we have any solutions to the
root of their complaints?"
brunowolff: how about keeping the PT karma requirement but adopting the
plan to allow critpath updates to go through without 'required' karma
after a two week wait
adamw: "i assume the stats made the assumption that the proventester
feedback on any update would still have been present but treated it no
different from regular feedback. so, that's not necessarily a safe
assumption: proventesters may feel a stronger 'responsibility to test',
and if you cancel the process, they might stop doing so. but that's hard
to gauge."
red_alert: proposes a meeting during FUDCon NA to try and come up with a
better pt process
That's about all the concrete thoughts / suggestions I can filter out of
the log.
Re: Rethinking proventester and critpath
By Nils Philippsen at 11/22/2011 - 09:06Hm. The list of (implicitly labeled) critpath packages seems to have
proliferated recently: a few days ago I submitted an update for
sane-backends and then found out that it's on critpath, probably by way
of critical-path-gnome -> control-center -> colord ->
sane-backends-libs. On the one hand it would probably be a good idea to
notify maintainers of such packages being implicitly pulled in (in this
case when BR: colord-devel was added to control-center). On the other
hand, sane-backends is a particularly nasty case and a detailed test
plan would probably start with this:
0. Have a lot of scanners handy, or at least models affected by the
changes.
Not even I get past that point in most cases: I have one USB scanner in
the office, and two rather ancient SCSI models at home. The only thing I
can meaningfully test is these models/their drivers and the
HW-independent parts of the package, and I don't expect more from the
testers. With the package being critpath now, I fear that updates for
sane-backends might never see the light of day, depending on the HW
affected, unless Bruno's proposal (two weeks of wait for critpath
without needed and no negative karma) or enough (proven)testers are
content with only cursory testing :-/.
Don't get me wrong: I'd really like sane-backends to get as much testing
as possible before turning updates loose on the unsuspecting public. I
just don't see how it can be done right at the moment.
Nils
Re: Rethinking proventester and critpath
By Adam Williamson at 11/22/2011 - 16:06On Tue, 2011-11-22 at 14:06 +0100, Nils Philippsen wrote:
This touches on a couple of issues. I know part of FESCo's response to
the proventesters question was to look at ways to slim down critpath.
It also hits the question of things which are critpath indirectly (via
dependencies). Part of the idea of having a test plan for all components
identified as critpath is to help identify those which are indirect
critpath packages, and make it clear what needs to be the case for those
packages to pass critpath validation - usually only that they don't
cause the actual critical path function to fail. In the sane_backends
case, for instance, in order to validate the critical path, you only
have to check the update doesn't somehow stop GNOME (and possibly
control-center) running. You don't have to make sure it doesn't break
scanning.
Providing a test plan is actually a good way to do this, if you take a
look at
<a href="https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation" title="https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation">https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_p...</a> and <a href="https://fedoraproject.org/wiki/QA:SOP_test_case_creation" title="https://fedoraproject.org/wiki/QA:SOP_test_case_creation">https://fedoraproject.org/wiki/QA:SOP_test_case_creation</a> : with the scheme explained there for categorizing test cases, you can both associate test cases with packages, and mark them as critical path or not.
So the 'intended' way to do things in this case would be to write a
bunch of test cases that actually test sane-backends, and put them in
the 'Category:Package_sane-backends_test_cases' category - but *not* to
put them in the 'Category:Critical_path_test_cases' category. You would
also add an existing test case for basic GNOME functionality - say,
<a href="https://fedoraproject.org/wiki/QA:Testcase_base_startup" title="https://fedoraproject.org/wiki/QA:Testcase_base_startup">https://fedoraproject.org/wiki/QA:Testcase_base_startup</a> - to the
'Category:Package_sane-backends_test_cases' category, and make sure it
was in the 'Category:Critical_path_test_cases' category too.
The result of that would be that, when a sane-backends update appeared,
the Bodhi page for it would list all the sane-backends test cases and
also the 'base_startup' test case, and identify the 'base_startup' test
case as being a critical path test case - thus indicating that the way
to really validate whether sane-backends *works* is to do the detailed,
specific tests, but the way to validate whether it breaks the critical
path is just to see if the system boots to a working desktop.
Then you'd need Bodhi 2.0 so you'd be able to provide feedback of "This
update doesn't break critical path" or "This update actually works for
scanning", rather than simply numeric feedback, as is the case now.
This may be a bit complex, I admit, and we haven't really completed the
'intended design' for many packages to show how it's supposed to work.
(That 'base_startup' test case should be associated with just about
every critpath package in existence, for instance, and it should be in
the critical path category too, which it currently isn't). But all the
bits are more or less there already, and just need hooking up. And we
need Bodhi 2.0 (yes, I know that's my catchphrase).
Re: Rethinking proventester and critpath
By Ken Dreyer at 11/22/2011 - 10:42On Tue, Nov 22, 2011 at 6:06 AM, Nils Philippsen < ... at redhat dot com> wrote:
Why not impose a self-restriction on the number of critpath packages?
Make a rule like "The ratio of proventesters to critpath packages must
be x or higher". When the critpath numbers start creeping up, drop a
few other packages from critpath. This will help focus testing
efforts, and allow it to expand along with the growth of the
community.
Re: Rethinking proventester and critpath
By Nils Philippsen at 11/22/2011 - 12:04On Tue, 2011-11-22 at 07:42 -0700, Ken Dreyer wrote:
But that wouldn't really help, would it? I assume packages are
explicitly labeled critpath for a reason: if they break, the system
might not boot up, or users might not be able to log in (or several
other reasons).
I think the current system is a bit too simplistic when it comes to the
packages upon which critpath components depend, however. For instance
(picking this example because it affects me), I don't think that logging
in should fail because of problems the scanner library may have. Or
because (drawing examples from a hat) the desktop background can't be
set, or there's an issue with the camera in your laptop. Yet, by way of
simply following package dependencies, we do as if that were the case.
Nils
Re: Rethinking proventester and critpath
By Adam Williamson at 11/22/2011 - 16:15Well, you'd think so, but reality frequently offends. ;)
There is, for instance, a bug in Fedora 16 which results in GNOME
failing to start if colord can't read a color profile in the user's home
directory due to it being incorrectly SELinux-labelled.
You might think this comes down to idiotic coding on the part of the
GNOME team, but actually it turns out to be rather more complex than
that. I'm too dumb to recap it accurately, but essentially, it's not
something that can actually be very easily fixed, though you'd think
it'd be easy to make sure the entirety of GNOME doesn't fall over
because of such a trivial sub-function, it actually isn't.
The general statement of this is 'it is sometimes the case that things
you'd never imagine could possibly break really critical functions, do
break really critical functions'.
Re: Rethinking proventester and critpath
By Bill Nottingham at 11/22/2011 - 12:00Ken Dreyer (<a href="mailto: ... at ktdreyer dot com"> ... at ktdreyer dot com</a>) said:
This then implies creation of a process that involves continual review of
the critpath package list, where it was designed to be able to be simply
computed without a lot of manual work.
Bill
Re: Rethinking proventester and critpath
By Bruno Wolff III at 11/21/2011 - 14:59On Mon, Nov 21, 2011 at 10:32:06 -0800,
Adam Williamson < ... at redhat dot com> wrote:
I don't remember which meeting I noted it at, but another idea is to
weight proventester feedback higher (maybe count as +2 or -2) in the
short term (until we have something more than a running count).
Re: Rethinking proventester and critpath
By =?UTF-8?B?IkrDs... at 11/01/2011 - 10:17On 11/01/2011 01:59 PM, Matthew Garrett wrote:
Agreed flag it as tried and tested stats dont lie...
JBG
Re: Rethinking proventester and critpath
By drago01 at 11/01/2011 - 10:11On Tue, Nov 1, 2011 at 2:59 PM, Matthew Garrett < ... at srcf dot ucam.org> wrote:
Well this looks like just a coincidence to me that the people finding
issues where proventesters in this cases.
I'd say go for it and I doubt there would be any quality change.