Pull Requests

The files of pull requests and reports generated by Codegex is available here.

Diversity

To evaluate the real capability of Codegex in generating review comments, we build a crawler to get the 10977 most recently opened PRs in GitHub. We select the PRs that have at least one code change in Java files (because our tool only analyzes Java files). This results in 4256 PRs from 2769 different GitHub projects. As shown in the table in supplementary material, the evaluated PRs are quite diverse as they involve patches that modify 0–25267 lines of code, spanning across 1 – 30 different Java files.

Code Review and Response

In our experiments, Codegex produces 372 code review comments for 4256 PRs so it can be time-consuming to manually check for their correctness. Hence, we rely on the real developer feedbacks to assess the quality of the generated comments. In total, we received 116 feedbacks from developers with about only 10% false positives which match well with the expectation in prior study. Among the 116 feedbacks that we received from developers, 78.45% of them are positive.