When looking back nowadays to the work done on diversity, I’ve realized that it has been quite a trip! My first approach to the topic was in an informal meeting with Nithya Ruff, currently at Comcast. She mentioned that the OpenStack Summit in Tokyo reached (as far as I remember!) 13% of women attending the Summit. And this was a great number if compared to previous summits as the percentage kept growing. But she also mentioned that they received a tweet asking about the current number of technical contributions. Then this is where we decided to have a look at that issue: have numbers, and try to produce some of them from a quantitative point of view.
Since last OpenStack Summit in Tokyo we were wondering at Bitergia if we could mix our knowledge on Software Development Metrics and some research we had done about gender/diversity contributions in Debian long time ago. Daniel started working on it, and he submitted a talk for OpenStack Summit in Austin that got accepted!
Updated: Vídeo and slides already available..
Public quarterly reports are used for understanding the performance of companies. And so, quarterly reports done by Bitergia fill the gap of understanding the performance of open source communities. This type of analysis focuses on those that are still interested in metrics, but do not have the time to play with the dashboards. This indeed provides a full overview of the current quarter, but adds a comparison with the previous quarters. This allows to have some extra context about where the community is heading.
[This post is based on the executive summary of the 2015-Q5 OpenStack Community Activity Report, sponsored by the OpenStack Foundation]
The October-Devember 2015 penStack Community Activity Report shows a stable growth of the OpenStack Community. As new repositories and teams keep being added, the number of projects keeps growing. On the other hand it is worth mentioning the decrease in activity during the latest quarters in project teams such as Nova, or stabilization of some others, such as Horizon or Cinder. This is a clear signal of the maturity reached by the some of the project teams in the OpenStack Foundation.
Active Core Reviewers reach a new peak
Although Git activity (changesets merged in the code base) does not show a large increase of activity, if compared to Gerrit, the development effort in the project keeps increasing. This last quarter of 2015 the number of Active Core Reviewers reached a new record of 449 different developers.
Time to merge keeps decreasing
During the third quarter of 2015, a small increase in time to merge seemed to signal a change in trend. However, this last quarter of 2015 keeps the previous decreasing trend. During this period the median time to merge a changeset into master decreased from 2.91 days down to 2.38 days.
Efficiency closing tickets decreases
It is noticeable the decrease of the relative number of tickets closed in OpenStack projects (also known as the efficiency of the ticket closing process). Previous quarters topped at about 60% of closed tickets (with respect to tickets opened during the same period), while this quarter shows a much lower 44%. This could be seen as a poor performance indicator of the project teams.
However, the efficiency closing changesets in the review system (number of changesets merged or abandoned with respect to number of new changesets being proposed for review) remains stable at around 80%.
In a large free, open source software development community, not all developers are equal. Some are more experienced, some are better known, some know better how to adhere to the uses and customs of the project, some write code that is more easily accepted by others. One of the areas where these differences are more noticeable is the code review process. It is not equally easy for all developers to push their patches through code review. Even when the process itself tries to be fair and unbiased, well, “all developers are equal, but some are more equal than others”.
Fortunately, we have plenty of data in the code review system. We can use analytics on that data to learn about how difficult it is for developers to get their patches accepted. We will use a OpenStack code review dashboard, composed with data from the OpenStack Gerrit instance to illustrate the process.
As a part of our tests with Kibana and Elasticserch as frontends for our MetricsGrimoire databases, we’ve set up a dashboard for understanding the code review process in OpenStack (be sure of visiting it with a large screen and a reasonable CPU, otherwise your experience may be a bit frustrating).
This dashboard includes information about all review processes (changesets) in OpenStack, using information obtained from their Gerrit instance. For each review, we have information such as the submitter (owner), the time it was first uploaded and accepted or abandoned, the number of patchsets (iterations) needed until it was accepted, and the time until it was merged or abandoned. With all of them we have prepared an active visualization that allows both to understand the big picture and to drill down looking for the details. Follow on reading to learn about some of these details.
[Note: this is our second post about our dashboards based on Kibana. If you’re interested, have a look at the first one, about OpenStack code contributions.]
We at Bitergia are busy testing new stacks for analyzing and visualizing the software development data we collect. Some our latest tests involve using Kibana for visualization. In this case, we have prepared a dashboard showing the latest contribution data for OpenStack.
One of the nice things that these new dashboards allow is the level of filtering and drill down which is possible. For example, in the above dashboard, it is possible to click on any sector on a pie chart, on any entry of a table, on any bar in a bar chart, and the corresponding filter will act. This allows for obtaining specialized dashboards very easily, such as this one with the contributions by RedHat (produced by clicking on RedHat in the list of of top organizations, or the contributions to Liberty, the latest release cycle of OpenStack, by selecting the corresponding period (last bar) in the “OpenStack ten top organizations by release” chart.
If you’re interested in learning about some tips and tricks about what can be done with these dashboards, follow on reading…
- There are 25,268 commits in total merged into master thanks to the work of 1,873 different developers.
- In order to have that code into master, it was necessary the effort of 2,239 people that submitted at least one patchset to Gerrit. That means that 83% of them are actual contributors of Liberty release.
- In terms of community, Launchpad activity shows that 9,919 people helped participating in the bug tracking process, opening, commenting and closing tickets.
- The mailing lists are a busy channel of communication with 1,742 participants, but IRC seems to be the preferred channel with more than 6,000 detected nicknames.
- Ask.openstack.org is also an interesting communication channel where there have been 1,386 people participating and around 2,200 different questions.
As a disclaimer, this post will not focus on organizations participating in the OpenStack development, but in the software development process. Organizations information can be easily retrieved in the Activity Board. We believe that process in the OpenStack community is important and even more when we are talking about a team of more than 2,000 different contributors.
Efficiency of the community closing changesets
As mentioned, there have been more than 2,200 people in Liberty release that aimed at contributing with a patchset in the OpenStack community. And only a subset of those pieces of code were eligible to be part of the project and merged into master. However, not all of the people that were part of the that set were even reviewed. Each quarter, the community leaves around a 20% of the changesets population open. This can be observed in the following chart. The y-axis represents the percentage of changesets that were closed (abandoned or merged) per quarter (x-axis).
- Users community keeps growing. With an increase of more than a 300% in the total number of questions posted in ask.openstack.org, this is the unique communnication channels with such measured increase. Other channels such as IRCor mailing lists also grow but slower. Although this is the activity measured for the last year, the last quarter shows a significant activity drop.
- Active Core Reviewers are increasing. Although mid-release cycle analysis usually show drops of activity in most of the areas, the total number of active core reviewers has reached a new peak. Up to 339 core reviewers participated in the review process. This shows an increase of around 9% if compared to the previous quarter analysis.
- Process keeps stable. The time to merge patches for the main projects show similar numbers than in previous quarter. However, Nova seems to be the project out of the common time to review with up to 10 days. On the other hand, Glance is more in line with the rest of the projects if compared to previous quarter. In any case, Glance shows the second highest meadian time to review with up to 7 days.
- IRC activity recovers old activity. Due to unknown issues, the total logs analyzed in the last two previous quarters indicated a huge drop of activity. However, last analysis on this data source shows activity in line with previous quarters. Although this is still an unknown issue, the IRC activity has recovered expected activity.
Project teams are also covered in this quarterly report. This quarterly report follows the Governance file for projects, but ignores specs files. Previous versions of the quarterly report such as 2015-Q1 or 2014-Q4 divided projects into the integrated project, specs, clients and others, with specific sections.