Architecture Diagram¶

[image: Architecture Diagram] [image]

This diagram contains a high-level approximation of the current Checkbox architecture. There are three main "pillars". On the left we have end products. Those are actual tools that certification and engineers are using. On the right we have the test market. This is a open market of tests vendors and suppliers. The tests are wrapped in containers known as providers. In the center we have three shared components. Those implement the bulk of the framework and user interfaces for test execution. Finally in the bottom-left corner there is a part of checkbox (a library) that is shared with HEXR for certain tasks. HEXR is a out-of-scope web application used by part of the certification process. Arrows imply communication with the shape of the arrow shows who calls who.

As mentioned before, in the center column there are three main components of shared code (shared by everyone using the end products that are discussed below). The shared code is composed of plainbox, checkbox-ng and checkbox-gui. Component responsibilities are discussed in more detail in the table below. Here we can see that checkbox-gui used DBus API exposed by checkbox-ng, which in turn uses checkbox-support (a helper library separated out so share some code with HEXR) and plainbox.

In the right hand side column there are various test providers. The checkbox project is producing and maintaining a number of providers (see the table below) but it is expected that our downstream users will also produce their own providers (specific to a customer or project). Eventually some providers may come from third parties that will adopt the format.

Lastly in the bottom-left corner, the shared library, this library contains many parsers of various file formats and output formats. Technically this library is a dependency of HEXR, checkbox-ng and of providers. As an added complexity the library needs to be called from python3 code and python2 code.

NOTE:

WARNING:

Component Descriptions¶

CheckboxNG 0.23 (unreleased)¶

CheckboxNG 0.22¶

CheckboxNG 0.3¶

CheckboxNG 0.2¶

CheckboxNG 0.1¶

brightness_test¶

This script tests the brightness of the systems backlight can be changed by using the kernel interfaces in /sys/class/backlight. There may be more than one interface to choose from, so the correct interface to use is selected by using the heuristic prescribed in https://www.kernel.org/doc/Documentation/ABI/stable/sysfs-class-backlight. The brightness is manipulated by updating the brightness file of the interface and the actual_brightness file is checked to see if the value was modified to the brightness selected.

Syntax¶

As checkbox-gui is a Qt application, settings must follow the INI-style rules of the QSettings <http://qt-project.org/doc/qt-5/QSettings.html> class.

Multiple-line values are supported but must be enclosed in doubles quotes and extra lines must start with one space, e.g:

[category]
key = "Hello


 World"

settings.value("category/key", i18n.tr("default_value"))

settings->value("category/key", app.tr("default_value"))

Conversely, checkbox-launcher-specific launchers must follow Python ConfigParser <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure> syntax.

Also, some settings only make sense for either GUI or CLI, and are thus not understood by the other. These are noted below.

Supported Settings¶

Legacy checkbox behavior control¶

In the past, Checkbox <http://plainbox.readthedocs.org/en/latest/glossary.html#term-checkbox>'s behavior was controlled by three mechanisms.

First, the functions of checkbox could be augmented by adding plugins. For example, the ability to submit to certification website was added by the checkbox-certification package using a plugin. The plugins included by checkbox-certification and which add new behavior to base checkbox were:

/usr/share/checkbox-certification/plugins/certify_message.py
/usr/share/checkbox-certification/plugins/submission_info.py
/usr/share/checkbox-certification/plugins/backup.py
/usr/share/checkbox-certification/plugins/certify_prompt.py
/usr/share/checkbox-certification/plugins/certify_report.py
/usr/share/checkbox-certification/plugins/certify_schemas.py

These added the way to prompt the user for submission-specific data, generate the xml report, and other functions.

Next, the behaviors of the plugins could be configured or controlled using configuration files, which are "cascaded". A config file can include others and those can in turn include others.

This is an example of a project-specific project-qt.ini main config file. It's the first file read when the project-specific client is launched. Some settings are abbreviated:

[DEFAULT]
includes = %(checkbox_oem_share)s/configs/checkbox-project-base-qt.ini %(checkbox_project_share)s/configs/checkbox-project-base.ini
[checkbox/plugins/environment_info]
repositories = deb http://.*\(archive\|security\).ubuntu.com/ubuntu precise-security
routers = multiple
server_iperf = 10.20.30.40
sources_list = /etc/apt/sources.list
wpa_n_psk = password
wpa_n_ssid = access-point
[checkbox/plugins/user_interface]
title = My project System Testing

Notice the includes line, this instructs it to load the config file for checkbox-project-base-qt and checkbox-project-base. Checkbox-project-base-qt loads the configs for checkbox-certification and checkbox-project. Settings are cascaded so the config options near the top override the ones near the bottom.

Finally, the "binary" used to invoke checkbox is a shell script that defines where to find the things checkbox needs to run: you can define a share directory, a specific data directory, point to a configuration file and define some environment variables that you may need during testing. Here's an example for checkbox-project-qt:

#!/bin/bash
export CHECKBOX_DATA=${CHECKBOX_DATA:-~/.checkbox}
export CHECKBOX_SHARE=${CHECKBOX_SHARE:-/usr/share/checkbox}
export CHECKBOX_OPTIONS=${CHECKBOX_OPTIONS:---log-level=debug --log=$CHECKBOX_DATA/checkbox-project.log}
export CHECKBOX_CERTIFICATION_SHARE=${CHECKBOX_CERTIFICATION_SHARE:-/usr/share/checkbox-certification}
export CHECKBOX_OEM_SHARE=${CHECKBOX_PROJECT_BASE_SHARE:-/usr/share/checkbox-project-base}
export CHECKBOX_PROJECT_SHARE=${CHECKBOX_PROJECT_SHARE:-/usr/share/checkbox-project}
# Convenience for defining the PYTHONPATH directory.
if [ "$CHECKBOX_SHARE" != "/usr/share/checkbox" ]; then


    export PYTHONPATH="$CHECKBOX_SHARE:$PYTHONPATH"
fi
python3 $CHECKBOX_SHARE/run "$@" $CHECKBOX_PROJECT_SHARE/configs/$(basename $0).ini

Here you can see that it defines some locations and an important part is the final python3 line, where it will locate and use the required .ini config file we saw earlier.

This hierarchical organization was very powerful but was also difficult to handle, and also had some limitations. Part of the work we did with checkbox was to integrate all the project-specific plugins into checkbox trunk, this way all the core code is in one place, and the project-specific variants only supply jobs, whitelists, data and configuration, without adding new behavior.

New plainbox behavior control¶

Unlike checkbox, plainbox's core is monolythic, and it has no concept of plugins. This makes it easier to understand and work with. The plainbox core has implementations for all the functions in the old checkbox packages, so no additions are necessary to use features such as submission to certification or report generation.

What we call plainbox is the library that implements all the functionality, as can be seen here.

Plainbox provides tools to help test developers write and package tests. These are delivered in "providers", which are entities designed to encapsulate test descriptions, custom scripts for testing, whitelists and assorted data. They were designed to allow teams to write and deliver their custom tests without worrying too much about the underlying plainbox code.

To get information on how to write tests and providers, see the Provider Tutorial <http://plainbox.readthedocs.org/en/latest/author/tutorial.html#tutorial>

However, when actually using these tests to verify a real system, we wanted to provide something easier and closer to the user experience of checkbox. We created two clients, checkbox-gui and checkbox-cli, which had some hardcoded behaviors, and we also started creating other clients which were based on these but were purpose specific. For instance, we had a version of checkbox for SRU testing, another for server certification, and so on.

But then we realized that a lot of the code was duplicated and the behaviors were common except for a few changes. So we came up with the concept of "launchers", which are somewhat similar to checkbox's configuration files and shell script launchers.

The idea is that checkbox-gui and checkbox-cli have some very basic behaviors, since they are the clients that are shipped by default with ubuntu. They can show all the available whitelists, show a predefined welcome message, and at the end will let the user see the html report and submit it to launchpad using their e-mail address, similar to the version of checkbox that shipped with Ubuntu.

Instead of using complicated command line switches, launchers allow you to configure some optional behaviors to customize your testing experience. A launcher contains settings, and is similar to a shell script, but the interpreter will be either checkbox-gui or checkbox-launcher.

Here are a few examples of what can be done with launchers.

As a surprise, checkbox-cli is itself a launcher:

#!/usr/bin/env checkbox-launcher
[welcome]
text = Welcome to System Testing!


    Checkbox provides tests to confirm that your system is working properly.


    Once you are finished running the tests, you can view a summary report for


    your system.


    Warning: Some tests could cause your system to freeze or become


    unresponsive. Please save all your work and close all other running


    applications before beginning the testing process.
[suite]
whitelist_filter = ^default$
whitelist_selection = ^default$
skip_whitelist_selection = True
[transport]
submit_to = launchpad

You can see here we customize a few options: it shows a welcome message, automatically selects the default whitelist, and will submit to launchpad when it's finished.

A graphical launcher example is canonical-certification-client.

#!/usr/bin/checkbox-gui
[welcome]
title = "System Certification"
text = "<p>Welcome to System Certification!</p><p></p><p>This application will
gather information from your system. Then you will be asked manual tests to
confirm that the system is working properly. Finally, you will be asked for
the Secure ID of the computer to submit the information to the certification
database.</p><p></p><p> To learn how to create or locate the Secure ID,
please see here: <a href=\"https://certification.canonical.com\">certification.canonical.com</a></p><p></p>"
[suite]
whitelist_filter = "^client-(cert|selftest).*"
[submission]
input_type = "regex"
input_placeholder = "Secure ID (15 or 18 characters)"
ok_btn_text = "Submit Results"
submit_to_hexr = "true"
[exporter]
xml_export_path = "/tmp/submission.xml"
[transport]
submit_to = "certification"

Graphical launchers are a bit more complicated, but essentially it's similar, what it allows is for you to define some parameters to customize your testing experience.

A very simple text-mode launcher is canonical-hw-collection which just runs the basic hardware information tests and uploads them to a hardware database:

[welcome]
title = Gathering hardware information
text = Gathering hardware information.  You may be prompted for your password.


       This process will take approximately 30 seconds and you will be provided


       with a URL through which you can confirm and register your hardware


       submission.
[suite]
whitelist_filter = ^hwsubmit$
whitelist_selection = ^hwsubmit$
skip_whitelist_selection = True
skip_test_selection = True
[submission]
# A bogus secure_id ensures we don't ask it
# It can always be overridden in the .conf file.
secure_id = 000
[transport]
submit_to = certification
submit_url = https://hardware-server.example.com/

FInally, canonical-driver-test-suite provides both a graphical and a text mode launcher, which are functionally equivalent:

#!/usr/bin/checkbox-gui
[welcome]
title = "Canonical Driver Test Suite"
text = "<p>Welcome to the Canonical Driver Test Suite.</p>


 <p></p>


 <p>This program contains automated and manual tests to help you discover


 issues that will arise when running your device drivers on Ubuntu.</p>


 <p></p>


 <p>This application will step the user through these tests in a


 predetermined order and automatically collect both system information as


 well as test results. It will also prompt the user for input when manual


 testing is required.</p>


 <p></p>


 <p>The run time for the tests is determined by which tests you decide to


 execute. The user will have the opportunity to customize the test run to


 accommodate the driver and the amount of time available for testing.</p>


 <p></p>


 <p>To begin, simply click the Continue button below and follow the onscreen


 instructions.</p><p></p>"
[suite]
whitelist_filter = "^ihv-.*"
[submission]
ok_btn_text = "Save Results"
input_type = "none"
[exporter]
xml_export_path = ""
[transport]
submit_to = "local"

Text mode:

#!/usr/bin/env checkbox-launcher
[welcome]
text = Welcome to the Canonical Driver Test Suite


    This program contains automated and manual tests to help you discover


    issues that will arise when running your device drivers on Ubuntu.


    This application will step the user through these tests in a


    predetermined order and automatically collect both system information as


    well as test results. It will also prompt the user for input when manual


    testing is required.


    The run time for the tests is determined by which tests you decide to


    execute. The user will have the opportunity to customize the test run to


    accommodate the driver and the amount of time available for testing.


    To begin, simply click the Continue button below and follow the onscreen


    instructions.
[suite]
# Whitelist(s) displayed in the suite selection screen
whitelist_filter = ^ihv-.*
# Whitelist_selection is mandatory so we set it to a bogus value so
# no whitelists are preselected.
whitelist_selection = bogus

Overview¶

Currently the process runs on a bi-weekly cadence, with a new release of Checkbox every two weeks. This covers ten working days, and the tasks carried out on each day or group of days is described below:

Launchpad Branches¶

The release process requires separate branches in Launchpad containing a semi-frozen version of the code that was in trunk on day 5 of the process. This is so that development can continue on trunk without jeopardising the stability of the to-be released version of Checkbox. The relationship between all branches involved in the process is as shown below:

Auditing milestoned bugs¶

Prior to creating the release candidate the release manager should review the list of bugs milestoned for the next release of Checkbox. They should visit checkbox milestones <https://launchpad.net/checkbox/+milestonesmilestones> and locate the milestone dated with the release date.

Cutting the release¶

In order to cut the release, we have to merge the changes from trunk into the release branch, commit them with a suitable message and update the changelog in trunk so that future changes go under the correct version. For each combination of branches shown above, do the following (the example uses lp:checkbox and lp:checkbox/release):

bzr branch lp:checkbox/release checkbox-release
bzr branch lp:checkbox checkbox-trunk
cd checkbox-release
current_stable=`head -n1 $(find . -name 'changelog') | grep -oP '(?<=\().*(?=\))'`
bzr merge lp:checkbox

at this point if no changes (other than one to debian/changelog) get merged in then we do not perform a release of the package in question. In practice this often happens with checkbox-certification but never with checkbox:

bzr commit -m "Merged in changes from rev$(bzr revno -r tag:$current_stable lp:checkbox) to rev$(bzr revno lp:checkbox) from lp:checkbox"
bzr push lp:checkbox/release
cd `find . -name 'debian'`; cd ..
bzr tag `head -n1 debian/changelog | grep -oP '(?<=\().*(?=\))'`
dch -r (save modified changelog)
dch -i -U 'Incremented changelog'
debcommit
bzr push lp:checkbox

The last step in the process is to perform a build of the packages in the ppa:checkbox-dev/testing PPA. To do this we need to go to the recipe pages for the checkbox and/or checkbox-certification release branches.

The Build Now option should be available on the page. Click it to start a build.

Copying Firmware Test Suite to the Testing PPA¶

The Firmware Test Suite tool is a test tool for system firmware that is naturally heavily utilised by Checkbox. To make sure the latest version which contains fixes and new tests/features needed by Checkbox is available and also doesn't break anything in Checkbox, we need to release it alongside Checkbox. After cutting the release if the Firmware Testing team have notified that a new version is available and that this version should be used for certification, we need to copy it to the Testing PPA. To do this we need to go to the Copy packages view of the Firmware Test Suite (Stable) PPA <https://launchpad.net/~firmware-testing-team/+archive/ppa-fwts-stable/+copy-packages> and select the 'fwts' packages for all releases back to Precise. We need to set the 'Destination PPA' as 'Checkbox Release Testing [~checkbox-dev/testing]' and the 'Copy options' field to 'Copy existing binaries', then click 'Copy packages'. This step then needs to be repeated but set the 'Destination PPA' field to 'PPA for Checkbox Developers [~checkbox-dev/ppa]'.

Next Release of Checkbox e-mail¶

So that everyone has the opportunity to perform whatever testing is required in a timely manner, after the PPA builds have been completed an email should be sent to the following mailing lists:

The content is typically something like this:

Subject: Next Release of Checkbox (18/11/2013)
Hi,
The next release of Checkbox is available in the
https://code.launchpad.net/~checkbox-dev/+archive/testing PPA.
Please test it at your convenience. Checkbox is based on revision 2484 of
lp:checkbox and Checkbox Certification is based on revision 586 of
lp:checkbox-certification.
Thanks,

If one or the other of Checkbox and Checkbox Certification have not been updated then there is no need to mention that package

Testing the release¶

Now that the release has been cut, testing should take place prior to the release meeting. From the point of view of the certification team, what needs to be tested is checkbox-certification-client and checkbox-certification-server which form the basis for CE QAs OEM specific versions of Checkbox. Checkbox certification server is tested in the CI loop Checkbox certification client needs to be tested manually.

Release Meeting¶

On the Thursday before the release is made, a meeting is held between a representative of the Certification team and a representative of the Commercial Engineering QA team. The meeting is held at 7:30 UTC as shown in this calendar invite <https://www.google.com/calendar/hosted/canonical.com/event?action=TEMPLATE&tmeid=Y3QxcWVla3ViMTRvMXByOHZlOTFvc283Y2NfMjAxMzA4MjlUMDczMDAwWiBicmVuZGFuLmRvbmVnYW5AY2Fub25pY2FsLmNvbQ&tmsrc=brendan.donegan%40canonical.com>. An agenda for the meeting is included in the invite.

Publishing the release¶

To publish the release we simply need to copy a number of packages from the Checkbox Release Testing PPA <https://launchpad.net/~checkbox-dev/+archive/testing> to the Hardware Certification Public PPA <https://launchpad.net/~hardware-certification/+archive/public>. To do this we go to the Copy packages view of the Checkbox Release Testing PPA <https://launchpad.net/~checkbox-dev/+archive/testing/+copy-packages> and select all versions of the following list of packages: checkbox, checkbox-certification, fwts. Make sure that the 'Destination PPA' field is set to 'Public PPA for Hardware Certification [~hardware-certification/public]' and that the 'Copy options' field is set to 'Copy existing binaries', then click 'Copy Packages'.

After that is done an announcement email should be sent to commercial-engineering@lists.canonical.com <commercial-engineering@lists.canonical.com>. A template for the announcement in included below:

Hi,
A new release of checkbox has been uploaded to the Hardware
Certification Public PPA
(https://launchpad.net/~hardware-certification/+archive/public). The
release is based on revision 2294 of lp:checkbox
Thanks,

Please attach the most recent part of the changelog as release notes