MSR 2025
Mon 28 - Tue 29 April 2025 Ottawa, Ontario, Canada
co-located with ICSE 2025
Mon 28 Apr 2025 16:20 - 16:30 at 215 - Software evolution and analysis Chair(s): Mauricio Verano Merino

Computational notebooks are the de facto platforms for exploratory data science, offering an interactive programming environment where users can create, modify, and execute code cells in any sequence. However, this flexibility often introduces code quality issues, with prior studies showing that approximately 76% of public notebooks are non-executable, raising significant concerns about reusability. We argue that the traditional notion of executability—requiring a notebook to run fully and without error—is overly rigid, misclassifying many notebooks and overestimating their non-executability.

This paper investigates pathological executability issues in public notebooks under varying notions and degrees of executability. Notebooks, by construction, are incrementally and interactively executed, where each cell execution advances logic toward the notebook’s goal. Even partially improving executability can improve code comprehension and offer a pathway for dynamic analyses. With this insight, we first categorize notebooks into potentially restorable and pathological non-executable notebooks and then measure how removing misconfiguration and superficial execution issues in notebooks can improve their executability (i.e., additional cells executed without error). For instance, we use a Large-Language Model (LLM) to generate synthetic input data to restore non-executable notebooks with “FileNotFound” errors. In a dataset of 42,546 popular public notebooks, containing 34,659 non-executable notebooks, only 21.3% are truly pathologically non-executable. For restorable notebooks, LLM-based methods fully restore 5.4% of previously non-executable notebooks. Among the partially restored, it improves the notebook’s executability by 42.7% and 28% by installing the correct modules and generating synthetic data. These findings challenge prior assumptions, suggesting that notebooks have higher executability than previously reported, many of which offer valuable partial execution, and that their executability should be evaluated within the interactive notebook paradigm rather than through traditional software executability standards.

Mon 28 Apr

Displayed time zone: Eastern Time (US & Canada) change

16:00 - 17:30
Software evolution and analysisData and Tool Showcase Track / Technical Papers / Industry Track at 215
Chair(s): Mauricio Verano Merino Vrije Universiteit Amsterdam
16:00
10m
Talk
50 Years of Programming Language Evolution through the Software Heritage looking glass
Technical Papers
Adèle Desmazières Sorbonne Unversité, Roberto Di Cosmo Inria, France / University of Paris Diderot, France, Valentin Lorentz Inria Foundation
16:10
10m
Talk
It Works (only) on My Machine: A Study on Reproducibility Smells in Ansible Scripts
Technical Papers
Ghazal Sobhani Dalhousie University, Israat Haque Dalhousie University, Tushar Sharma Dalhousie University
Pre-print
16:20
10m
Talk
Are the Majority of Public Computational Notebooks Pathologically Non-Executable?
Technical Papers
Waris Gill Virginia Tech, Muhammad Ali Gulzar Virginia Tech, Tien Nguyen Virginia Tech
Pre-print
16:30
10m
Talk
Understanding Test Deletion in Java Applications
Technical Papers
Suraj Bhatta North Dakota State University, Frank Kendemah North Dakota State University, Ajay Jha North Dakota State University
Pre-print
16:40
10m
Talk
A Public Benchmark of REST APIs
Technical Papers
Alix Decrop University of Namur, Sara Eraso University of Valle, Xavier Devroey University of Namur, Gilles Perrouin Fonds de la Recherche Scientifique - FNRS & University of Namur
Pre-print
16:50
5m
Talk
What Do Contribution Guidelines Say About Software Testing?
Technical Papers
Pre-print
16:55
5m
Talk
Measuring InnerSource Value
Industry Track
17:00
5m
Talk
CoUpJava: A Dataset of Code Upgrade Histories in Open-Source Java Repositories
Data and Tool Showcase Track
Kaihang Jiang University of Waterloo, Bihui Jin University of Waterloo, Pengyu Nie University of Waterloo
17:05
5m
Talk
EvoChain: A Framework for Tracking and Visualizing Smart Contract Evolution
Data and Tool Showcase Track
Ilham Qasse Reykjavik University, Mohammad Hamdaqa Polytechnique Montréal, Björn Þór Jónsson Reykjavik University
17:10
5m
Talk
CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance
Data and Tool Showcase Track
Kunal Suresh Pai UC Davis, Prem Devanbu University of California at Davis, Toufique Ahmed IBM Research
Pre-print
17:15
5m
Talk
RefExpo: Unveiling Software Project Structures through Advanced Dependency Graph Extraction
Data and Tool Showcase Track
Vahid Haratian Bilkent Univeristy, Pouria Derakhshanfar JetBrains Research, Vladimir Kovalenko JetBrains Research, Eray Tüzün Bilkent University
17:20
5m
Talk
HyperAST: Incrementally Mining Large Source Code Repositories
Data and Tool Showcase Track
Quentin Le Dilavrec TU Delft, Netherlands, Andy Zaidman Delft University of Technology
Pre-print