Certify this! The automated testing of Ubuntu SRUs

Testing Stable Release Updates (SRU) for Ubuntu in the certification lab is a de-risking move that will enable the Ubuntu Platform team to release SRUs frequently (bi-weekly). This week was the first time that we did it! but there is still areas for improvements.

For this cycle, we agreed that the test will consists on installing that latest (point) release image and upgrading to the proposed kernel. Then after rebooting, the test will make sure that system still has internet access so it could be recovered via the Update Manager.

The proposed Kernels are 2.6.35-23.40 for Maverick (Ubuntu 10.10) and 2.6.32-26.47 Lucid (Ubuntu 10.04 LTS). The aim is to test the SRU for all the systems that we have certified for the release (in average that is around 100 systems). We had done some testing the week before that indicated that 50 systems was a good target for the first real attempt.

The test would be perform by running a set of automated scripts after installation and ensuring that results were reported back to a remote server, hence ensuring internet connection was still functional (regardless of the pass/fail rate of the actual scripts).

It took Mr Marc Tardiff just under a day to tested each release and achieve the target. He recorded the results in the two tracking bug (Maverick &  Lucid)

So, Should we congratulate ourselves? Yes. We’ve achieved what we set out to do … but we shouldn’t be complacent. The target of 100 systems in 1 hour is still far from our grasp.

The main challenge is using an infrastructure designed to certify individual systems remotely, rather than act as an automated test farm.  This leads to reliability problems that we will need to overcome in the coming weeks.

For the next SRU  round , we will targeting 75 systems. The time will still be within 2 lab days.