MSR 2025 - Mining Challenge

Call for Mining Challenge Papers
Call for Mining Challenge Proposals

Call for Mining Challenge Papers

Using package managers is a simple and common method for reusing code through project dependencies. However, these direct dependencies can themselves rely on additional packages, resulting in indirect dependencies. It may then become complex to get a grasp of the whole set of dependencies of a project. Beyond individual projects, a deep understanding of how software ecosystems work and evolve is also a critical prerequisite for achieving sustained success in software development.

This year’s mining challenge focuses on dependencies and dependency ecosystem analysis using the Goblin framework that has been presented at the previous edition of the MSR conference. Goblin is composed of a Neo4J Maven Central dependency graph and a tool called Weaver for on-demand metric weaving into dependency graphs. As a whole, Goblin is a customizable framework for ecosystem and dependency analysis.

Challenge

The analysis of a software ecosystem graph presents numerous research opportunities, allowing for the investigation of various questions in areas such as structural analysis, community detection, dependency optimization, and risk assessment within the Maven Central ecosystem.

The following suggested questions outline potential research inquiries: Questions in groups 1 to 5 can be addressed with the dependency graph database alone, questions in group 6 require the additional use of the Weaver, and questions in group 7 illustrate examples of inquiries that would require an extension of the Weaver.

Ecosystem evolution
i. What are the patterns in the growth of the Maven Central graph across different time periods?
ii. Do libraries tend to use more dependencies than in the past?
iii. Is the rhythm of library releases higher than in the past, and how has this rhythm evolved over time?
iv. Does the emergence of project management methods (e.g., agile methods) have any impact on the release rhythm of libraries?
v. To what extent does the ecosystem contain unmaintained libraries?
vi. How do projects with unmaintained dependencies cope with the challenges they face?
Clustering
i. Can we deduce different clusters from Maven Central’s comprehensive dependency graph? How do these clusters interact with one another?
ii. Can dependency-based clustering reveal domain-specific groupings, and how well do they align with known categorizations of projects?
iii. How can clustering be used to identify high-risk clusters in the Maven Central ecosystem?
iv. Which artifacts serve as the most crucial dependencies for the ecosystem (i.e., most depended upon)?
v. How do these central nodes affect the overall health and stability of the ecosystem?
Dependency update
i. How often do projects update their dependencies, and what factors influence this frequency (e.g., project size, popularity, type)?
ii. Whenever an artifact releases a new version, how do its dependents react?
iii. How does the removal or failure of certain projects affect the overall network (e.g., log4j Vulnerability)?
iv. How do major versus minor dependency updates differ in frequency and impact?
v. Do projects tend to avoid major updates due to the potential for breaking changes?
Trends
i. How has the adoption of new frameworks (e.g., Spring Boot, Microservices) changed the dependency structures in Maven Central?
ii. What impact do modern dependency management tools (e.g., Dependabot) have on the ecosystem?
iii. How does the adoption of newer Java versions influence dependency graphs?
iv. Does an artifact’s number of dependents correlate with other popularity metrics such as GitHub stars?
Graph theory
i. How do metrics such as degree distribution, clustering coefficient, and average path length characterize the dependency graph?
ii. Is the graph scale-free, small-world, or does it exhibit other known graph structures?
iii. Are certain types of projects more likely to be central (hubs) or peripheral (leaves) in the graph structure?
iv. Is the graph made up of connected components with no relationship between them?
v. How do shortest path lengths between projects vary, and what does this tell us about the overall connectivity of the ecosystem?
Vulnerability
i. How do vulnerabilities propagate through the dependency network, and which projects are most affected?
ii. What proportion of releases have vulnerabilities? What is the proportion of releases directly and transitively impacted?
iii. What is the average time taken to patch a vulnerability in a dependency?
iv. How do users of an artifact react to the discovery of a vulnerability in that artifact?
Licensing and Compliance
i. Are there dominant license types, and how do they influence the usage and distribution of projects?
ii. How does the choice of licenses affect the artifact graph structure?
iii. What percentage of projects have conflicting licenses within their dependency trees?

Description of the Challenge Dataset and Tooling

The Goblin framework (see figure below) is organized around a Neo4J database of the whole Maven Central dependency graph. This database can be created and updated incrementally using Goblin Miner. The database can be queried directly using Cypher (the Neo4j query language) or through the Goblin Weaver tool. More generally, Goblin Weaver comes with an API for the on-demand weaving of user-programmed metrics of interest (added values) into the dependency graph. Several such metrics are already available: CVEs, popularity, freshness and release rhythm. As a whole, Goblin aims to be a customizable framework for working on software dependencies at the ecosystem level.

The dependency graph database is composed of two node types (for libraries and for their releases) and two edge types (from releases to their dependencies and from libraries to their releases). The nodes for libraries (type Artifact) contain the Maven id (g.a) information. The nodes for releases (type Release) contain the Maven id (g.a.v), the release timestamp, and the version information. The edges for dependencies (type dependency) are from Release nodes to Artifact nodes and contain target version (which can be a range) and scope (compile, test, etc). The edges for versioning (type relationship_AR) edges are from Artifact nodes to Release nodes.

The latest version of our dataset, dated August 30th, 2024, contains 15,117,217 nodes (658,078 libraries and 14,459,139 releases) and 134,119,545 edges (119,660,406 dependencies and 14,459,139 versioning edges).

We also provide a second version of this dataset enriched with the Weaver metrics, which has the effect of creating new “AddedValue” nodes in the database containing the metrics (CVE (dated September 4, 2024), freshness, popularity and speed).

The Goblin Miner allows you to update the dependency graph database or recreate it from scratch. The Miner Java source code is available.

The Goblin Weaver REST API is available as an alternative for direct access to the database using the Cypher language and for on-demand enrichment of the dependency graph with new information. A memoization principle is available to avoid re-computing enrichments, as soon as the base graph is not re-computed or incremented. For this, new kinds of nodes (type AddedValue) and edges (type addedValues from an Artifact or Release node to an AddedValue node) are used in the graph database. One should be careful, as the graph is large, calculating metrics (especially aggregate ones) for the whole graph can be time-consuming. The Weaver Java source code is available.

A tutorial is available online on this GitHub repository. The issue tracking system will also enable one to ask questions on the datasets and tooling.

How to Participate in the Challenge

First, familiarize yourself with the Goblin framework:

The Maven Central Neo4j dataset is available on this Zenodo archive.
To import this dump into Neo4J, please use a dbs version 4.x.
The Weaver project is available in our GitHub repository.
This paper https://doi.org/10.1145/3643991.3644879 gives a more in-depth presentation of how the Goblin framework works.
A project for simply setting up a Neo4j database and Weaver API using Docker is available here: https://github.com/Goblin-Ecosystem/Neo4jWeaverDocker

Use the dataset to answer your research questions, and report your findings in a four-page challenge paper that you submit to our challenge. If your paper is accepted, present your results at MSR 2025 in Ottawa, Canada!

Submission

A challenge paper should describe the results of your work by providing an introduction to the problem you address and why it is worth studying, the version of the dataset you used, the approach and tools you used, your results and their implications, and conclusions. Make sure your report highlights the contributions and the importance of your work. See also our open science policy regarding the publication of software and additional data you used for the challenge.

To ensure clarity and consistency in research submissions:

When detailing methodologies or presenting findings, authors should specify which snapshot/version of the Goblin dataset (and the Weaver version if used) was utilized.
Given the continuous updates to the dataset, authors are reminded to be precise in their dataset references. This will help maintain transparency and ensure consistent replication of results.

Submissions must conform to the IEEE formatting instructions IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options).

Submissions to the Challenge Track can be made via the submission site by the submission deadline. We encourage authors to upload their paper info early (the PDF can be submitted later) to properly enter conflicts for anonymous reviewing. All submissions must adhere to the following requirements:

Submissions must not exceed the page limit (4 pages plus 1 additional page of references). The page limit is strict, and it will not be possible to purchase additional pages at any point in the process (including after acceptance).
Submissions must strictly conform to the IEEE conference proceedings formatting instructions specified above. Alterations of spacing, font size, and other changes that deviate from the instructions may result in desk rejection without further review.
Submissions must not reveal the authors’ identities. The authors must make every effort to honor the double-anonymous review process. In particular, the authors’ names must be omitted from the submission and references to their prior work should be in the third person. Further advice, guidance, and explanation about the double-anonymous review process can be found in the Q&A page from ICSE.
Submissions should consider the ethical implications of the research conducted within a separate section before the conclusion.
The official publication date is the date the proceedings are made available in the ACM or IEEE Digital Libraries. This date may be up to two weeks prior to the first day of ICSE 2025. The official publication date affects the deadline for any patent filings related to published work.
Purchases of additional pages in the proceedings are not allowed.

Any submission that does not comply with these requirements is likely to be desk rejected by the PC Chairs without further review. In addition, by submitting to the MSR Challenge Track, the authors acknowledge that they are aware of and agree to be bound by the following policies:

The ACM Policy and Procedures on Plagiarism and the IEEE Plagiarism FAQ. In particular, papers submitted to MSR 2025 must not have been published elsewhere and must not be under review or submitted for review elsewhere whilst under consideration for MSR 2025. Contravention of this concurrent submission policy will be deemed a serious breach of scientific ethics, and appropriate action will be taken in all such cases (including immediate rejection and reporting of the incident to ACM/IEEE). To check for double submission and plagiarism issues, the chairs reserve the right to (1) share the list of submissions with the PC Chairs of other conferences with overlapping review periods and (2) use external plagiarism detection software, under contract to the ACM or IEEE, to detect violations of these policies.
The authorship policy of the ACM and the authorship policy of the IEEE.

Upon notification of acceptance, all authors of accepted papers will be asked to fill a copyright form and will receive further instructions for preparing the camera-ready version of their papers. At least one author of each paper is expected to register and present the paper at the MSR 2025 conference. All accepted contributions will be published in the electronic proceedings of the conference.

This year’s mining challenge and the data can be cited as:

@inproceedings{
    title={Navigating and Exploring Software Dependency Graphs using Goblin},
    author={Jaime, Damien and El Haddad, Joyce and Poizat, Pascal},
    year={2025},
    booktitle={Proceedings of the International Conference on Mining Software Repositories (MSR 2025)},
}

A preprint will be available online.

Submission Site

Papers must be submitted through HotCRP: https://msr2025-challenge.hotcrp.com/

Important Dates

AoE: Anywhere on Earth

Abstract Deadline: Dec 3, 2024 AoE
Paper Deadline: Dec 6, 2024 AoE
Author Notification: Jan 12, 2025 AoE
Camera Ready Deadline: Feb 5, 2025 AoE

Open Science Policy

Openness in science is key to fostering progress via transparency, reproducibility and replicability. Our steering principle is that all research output should be accessible to the public and that empirical studies should be reproducible. In particular, we actively support the adoption of open data and open source principles. To increase reproducibility and replicability, we encourage all contributing authors to disclose:

the source code of the software they used to retrieve and analyze the data
the (anonymized and curated) empirical data they retrieved in addition to the Goblin Maven dataset
a document with instructions for other researchers describing how to reproduce or replicate the results

Already upon submission, authors can privately share their anonymized data and software on archives such as Zenodo or Figshare (tutorial available here). Zenodo accepts up to 50GB per dataset (more upon request). There is no need to use Dropbox or Google Drive. After acceptance, data and software should be made public so that they receive a DOI and become citable. Zenodo and Figshare accounts can easily be linked with GitHub repositories to automatically archive software releases. In the unlikely case that authors need to upload terabytes of data, Archive.org may be used.

We recognise that anonymizing artifacts such as source code is more difficult than preserving anonymity in a paper. We ask authors to take a best effort approach to not reveal their identities. We will also ask reviewers to avoid trying to identify authors by looking at commit histories and other such information that is not easily anonymized. Authors wanting to share GitHub repositories may want to look into using https://anonymous.4open.science/ which is an open source tool that helps you to quickly double-blind your repository.

We encourage authors to self-archive pre- and post prints of their papers in open, preserved repositories such as arXiv.org. This is legal and allowed by all major publishers including ACM and IEEE and it lets anybody in the world reach your paper. Note that you are usually not allowed to self-archive the PDF of the published article (that is, the publisher proof or the Digital Library version). Please note that the success of the open science initiative depends on the willingness (and possibilities) of authors to disclose their data and that all submissions will undergo the same review process independent of whether or not they disclose their analysis code or data. We encourage authors who cannot disclose industrial or otherwise non-public data, for instance due to non-disclosure agreements, to provide an explicit (short) statement in the paper.

Best Mining Challenge Paper Award

As mentioned above, all submissions will undergo the same review process independently of whether or not they disclose their analysis code or data. However, only accepted papers for which code and data are available on preserved archives, as described in the open science policy, will be considered by the program committee for the best mining challenge paper award.

Best Student Presentation Award

Like in the previous years, there will be a public voting during the conference to select the best mining challenge presentation. This award often goes to authors of compelling work who present an engaging story to the audience. Only students can compete for this award.

Call for Mining Challenge Proposals

The International Conference on Mining Software Repositories (MSR) has hosted a mining challenge since 2006. With this challenge, we call upon everyone interested to apply their tools to a common dataset. The challenge is for researchers and practitioners to bravely use their mining tools and approaches on a dare.

One of the secret ingredients behind the success of the International Conference on Mining Software Repositories (MSR) is its annual Mining Challenge, in which MSR participants can showcase their techniques, tools, and creativity on a common data set. In true MSR fashion, this data set is a real data set contributed by researchers in the community, solicited through an open call. There are many benefits of sharing a data set for the MSR Mining Challenge. The selected challenge proposal explaining the data set will appear in the MSR 2025 proceedings, and the challenge papers using the data set will be required to cite the challenge proposal or an existing paper of the researchers about the selected data set. Furthermore, the authors of the data set will join the MSR 2025 organizing committee as Mining Challenge (co-)chair(s), who will manage the reviewing process (e.g., recruiting a Challenge PC, managing submissions, and reviewing assignments). Finally, it is not uncommon for challenge data sets to feature in MSR and other publications well after the edition of the conference in which they appear!

If you would like to submit your data set for consideration for the 2025 MSR Mining Challenge, prepare a short proposal (1-2 pages plus appendices, if needed) containing the following information:

Title of data set.
High-level overview:
- Short description, including what types of artifacts the data set contains.
- Summary statistics (how many artifacts of different types).
Internal structure:
- How are the data structured and organized?
- (Link to) Schema, if applicable
How to access:
- How can the data set be obtained?
- What are recommended ways to access it? Include examples of specific tools, shell commands, etc, if applicable.
- What skills, infrastructure, and/or credentials would challenge participants need to effectively work with the data set?
What kinds of research questions do you expect challenge participants could answer?
A link to a (sub)sample of the data for the organizing committee to pursue (e.g., via GitHub, Zenodo, Figshare).

Submissions must conform to the IEEE conference proceedings template, specified in the IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options). Submit your proposal here.

The first task of the authors of the selected proposal will be to prepare the Call for Challenge Papers, which outlines the expected content and structure of submissions, as well as the technical details of how to access and analyze the data set. This call will be published on the MSR website on September 2nd. By making the challenge data set available by late summer, we hope that many students will be able to use the challenge data set for their graduate class projects in the Fall semester.

Important dates: