NLBSE 2022

Tool Competition

Introduction

NLP-based approaches and tools have been proposed to improve the efficiency of software engineers, processes, and products, by automatically processing natural language artifacts (issues, emails, commits, etc.).

We believe that the availability of accurate tools is becoming increasingly necessary to improve Software Engineering (SE) processes. One important process is issue management and prioritization where developers have to understand, classify, prioritize, assign, etc. incoming issues reported by end-users and developers.

This year, we are pleased to announce the first edition of the NLBSE’22 tool competition on issue report classification, an important task in issue management and prioritization.

For the competition, we provide a dataset encompassing more than 800k labeled issue reports (as bugs, enhancements, and questions) extracted from real open-source projects. You are invited to leverage this dataset for evaluating your classification approaches and compare the achieved results against a proposed baseline approach (based on FastText).

Competition overview

We created a Colab notebook with detailed information about the competition (provided data, baseline approach, paper submission, paper format, etc.).

If you want to participate, you must:

• Train and tune a multi-label multi-class classifier using the provided training set. The classifier should assign one label to an issue.
• Evaluate your classifier on the provided test set
• Write a paper (4 pages max.) describing:
• The architecture and details of the classifier
• The procedure used to pre-process the data
• The procedure used to tune the classifier on the training set
• The results of your classifier on the test set
• Additional info.: provide a link to your code/tool with proper documentation on how to run it
• Submit the paper by emailing the tool competition organizers (see below)

Submissions will be evaluated and accepted based on correctness and reproducibility, defined by the following criteria:

• Clarity and detail of the paper content
• Availability of the code/tool, released as open-source
• Correct training/tuning/evaluation of your code/tool on the provided data
• Clarity of the code documentation

The accepted submissions will be published at the workshop proceedings.

The submissions will be ranked based on the F1 score achieved by the proposed classifiers on the test set, as indicated in the papers.

The submission with the highest F1 score will be the winner of the competition.

How to participate?

Email your paper to Oscar Chaparro (oscarch@wm.edu) and Rafael Kallis (rk@rafaelkallis.com) by the submission deadline.