Quality Metrics

Maintaining the current high quality levels of the platform assets is a big priority for both our members and ourselves.

In the past, Symbian Ltd. did a good job at understanding how the software maturity was evolving and when the platform would be ready for releasing. Moving to open source has changed the sources of information available to us and their reliability. So, can we still measure the quality of our platform?

To answer this question, the Release Council launched a working group, back in June, that has identified a set of initial metrics for the job.

We are taking an Agile approach by implementing quality metrics in iterations. The first iteration will be a small set that would deliver to us substantial value, it will enable us to learn from the experience and then apply improvements to the next iteration.

Here is my take on why the proposed metrics are the right ones to start with:

Turnaround times, Inflow, Outflow and Open Bugs

One source of information available to the Symbian Foundation is Bugzilla. The proposed metrics not only will allow us to get a view on the quality of each package but also on the health of the community around it. By using these metrics we will try to answer questions such as:

• How diverse is the community contributing to the package?
• Is the package actively being tested/used by other members beyond the package owner?
• How active is the community around the package? Do they care enough to ensure that critical bugs are turned around quickly?

Implementing these metrics becomes easier thanks to open source tools such as bugzilla metrics!

However, to have a decent stab at answering these questions, we need to know more about the package in question. In particular, how much is it changing…

Code Size and Churn

To provide context to the bug metrics, it is key to understand the size and speed of change of the software asset. While size can act as a normalising factor when comparing packages, the speed and volume of change can eventually lead us to predicting latent defects in the asset.

Stability through measuring Code Coverage, Build Success Rate and Unapproved BC breaks

When offering a development platform it is important that the community can understand the stability of the platform. This will be measured during the first iteration by coverage of the package automated testing, the success rate of the platform being built by the Symbian Foundation team and the introduction of unapproved binary compatibility breaks.

Next Steps

We are now working on reviewing the detailed definition of these metrics and implementing them in the following months. If you want to help, join the discussion!