Xubuntu

Chapter 5. Package and ISO Testing

Table of Contents

ISOs
Exploratory Testing
Using Development PPAs
Static Testing
Milestone Testing
Daily Testing
Upgrade Testing
Package Testing

ISOs

In general links for specific isos for daily or milestone use can be found at the ISO Tracker A link exists for the current development daily iso. This can be useful, especially when setting up methods to download your testing iso, such as with zsync

[Note]

You can find the last known boot and installation status of our ISOs at QA Status of the development status tracker

Exploratory Testing

In addition to testing the images and software with predefined testing actions (static testing), community members can take part in exploratory testing as well.

In essence, exploratory testing means running the development release and doing your daily tasks with the system and finally, filing bugs when you find them. This allows Xubuntu to get a much larger spread of testing than is possible with predefined tests.

A useful way to deal with this is dual (or multiple) boots. By dual (or multiple) booting you can ensure you are able to access at all times a working version while testing the development system. In this way you can do as much work in the development release as is practical – all the while watching for bugs and regressions in the development release.

Issues which the development team are keen to address include, in addition to normally reported issues, usability bugs, missing icons, inconsistent functionality.

Using Development PPAs

We have 3 PPAs which we use regularly to test packages. These are:

  • Shimmer Themes for daily builds of the Shimmer Project’s projects

  • Xubuntu Staging for packages and package versions that are being prepared for inclusion in Xubuntu

  • Xubuntu daily builds for daily Git and Bzr builds for packages related to Xubuntu and/or Xfce4

[Note]

At times the development team use other PPAs to test specific packages. Details of these will be made known to testers when appropriate

Installing these three PPAs means that you will be using and testing packages that developers are currently working with, this means that regressions found by you will not be present once they are released.

Removing an in use PPA. Using ppa-purge causes apt to disable a PPA source list and then change affected packages back to the default versions.

  • Run sudo apt-get install ppa-purge

  • Run sudo ppa-purge ppa:ppaname

Reporting bugs with PPAs can be problematic, please see further information on reporting bugs with PPAs in Reporting bugs with PPAs.

Static Testing

Testing the development versions of Xubuntu and packages in it helps keep up the every day quality. If you have spare hardware resources or are able to run a virtualized testing environment, you can help.

  • Check the devel mailing list close to the start of the cycle, where the decision as to which milestones we will participate in will be discussed.

  • While milestone testing is in progress, please watch for rebuilds. A rebuilt iso will have .something appended to the iso name (eg. "".1"). These will take place either when we have reason to rebuild, eg following a bug fix landing, or more likely when the Ubuntu Release Team have cause to do so.

Milestone Testing

Stand alone image testing for milestones is useful in the 2 days before the release. To see how these milestones are scheduled, refer to the Ubuntu release schedule.

  • Live Session: Boot with the image and ensure that basic application testing (open, close, saving etc.) passes.

  • Installation tests required pass. These test only the installation.

Daily Testing

The importance of daily image testing lies in the main in knowing that boot or installation regressions haven’t appeared. Where it appears they have, if you are able to boot with a different flavours image, this can help prove a global or local to Xubuntu issue. If unsure contact the Xubuntu team, preferrably on IRC.

  • Live Session: Boot with the image and ensure that basic application testing (open, close, saving etc.) passes.

  • Installation tests required pass. These test only the installation.

Upgrade Testing

About half way through any development cycle (around the beta milestones) we need to test upgrades from the previous release to the new development version. When the development version is intended to be the next LTS release, we need to test both upgrade paths: LTS to LTS and regular to LTS.

The ISO tracker has a section for upgrade tests which does not change daily, but runs from Friday to Friday. This enables us to test upgrades for a week rather than a day. During these tests it is preferred that the image used for testing is the one built on the first Friday, except where bugs are reported and fixes landed.

  • Upgrade: tested using the update-manager

  • Upgrade (image): tested using the daily image

Package Testing

For releases where we are making use of the Package Tracker, calls for package testing will be made to the xubuntu-devel mailing list as required. This could be a call from QA or Developers.

A schedule of planned tests will be mailed to the Xubuntu Devel mailing list close to the cycle start. Reminders of upcoming package test requirements are mailed to the Xubuntu devel mailing list as required. Specific developer testing requirement are mailed as they are available, often these packages will be those found on one of our PPAs.