In what the authors are claiming is the largest-scale study of gender bias to date, researchers in the US have found that code written by female programmers is rated more highly than code written by men.

But this higher rating – based on code acceptance from other coders – is lost when female programmers publicly identify their gender online, with acceptance of their contributions then falling below the acceptance level of code written by men.

The findings suggest that female programmers may be better at what they do than their male counterparts, but that attitudes within the software community might be making it harder for them to have their contributions recognised and accepted – unless they're already known by collaborators, or elect to hide their gender, that is.

To examine the prevalence of gender bias within the world of open source programming, researchers from California Polytechnic State University and North Carolina State University analysed user behaviour on the massive code repository, GitHub. The community consists of some 10 million users, and the gender is apparent in 1.4 million of these profiles.

"Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless," the authors write.

Investigating what role gender plays in terms of how code and its authorship is perceived on the platform, the researchers looked at 'pull requests' between members. Pull requests occur when programmers suggest new code contributions to projects maintained by others. If the pull request is accepted by the project owner, the new code is then merged with the project.

What the researchers found, to their surprise, was that pull requests made by women were accepted at a higher rate (78.6 percent) than those made by men (74.6 percent). While they don't fully understand why this is so, the team suggests it's not always the case.

If collaborators receive a pull request from a female programmer whom they don't know and whose gender is not identifiable, the acceptance rate drops to 71.8 percent. But if the pull request is received from an unknown woman whose gender is made public, the acceptance rate drops significantly to just 62.5 percent.

The research, available online, has not yet been peer-reviewed, but it seems to reveal a serious trend that the study authors say needs to be examined further.

"[I]t's imperative that we use big data to better understand the interaction between genders," they write. "While our big data study does not definitely prove that differences between gendered interactions are caused by bias among individuals, the trends observed in this paper are troubling. The frequent refrain that open source is a pure meritocracy must be reexamined."

The findings could be particularly important in light of how computer science is growing in popularity and relevance, with US states now considering proposals to include computer coding classes alongside foreign language study options in schools.

Officials in Florida, Kentucky, Georgia, New Mexico, Oregon, and Washington are all contemplating such changes, with the thinking being that children's futures could stand to benefit from greater exposure to programming languages such as Python and JavaScript, in addition to traditional foreign language choices.

The move would complement President Obama's new US$4.2 billion "Computer Science For All" educational program, although some commentators are concerned about the effect the substitution could have on spoken language study.

Update: Since this paper was made available online, it has received a number of criticisms based on the fact that the data don't really back up some of the interpretations offered by the authors.

One of the more pertinent criticisms is that when all coders – either male or female – disclose their gender to programmers that they don't know, their pull requests are accepted less. In other words, both men and women can be seemingly disadvantaged by revealing their gender to others, not just women.

The authors have been accused of glossing over this in their abstract, opting to instead highlight a narrative of bias against women instead.

According to science blog Sauropod Vertebra Picture of the Week, the repercussions of this narrative and its subsequent reporting by the media don't do any favours for female coders: "[N]o-one comes out of this as winners… most of all, not the women who will now be discouraged from contributing to open-source projects."

Here's hoping the authors take note of some of these public comments, which may help them revise their paper for publication in a journal. This is why peer-review is so important – it helps steer study authors towards the best presentation of their research and findings, including helping them to avoid coming up with incorrect or misleading conclusions.

In the meantime, you can read the full paper online here to take a closer look at their results for yourself. And for a more detailed analysis of the paper's shortcomings, check out this post at Sauropod Vertebra Picture of the Week.