Greetings! Welcome to the IAP Newsletter with recent and upcoming research publications, news and events. Research includes applications and infrastructure for AI and machine learning, hardware acceleration, operating systems, networking, security, storage and data management - all in the context of distributed systems.

Join our new Series of webinars featuring topics in AI presented by prominent experts in machine learning.

Pre-registration is required. Please see the abstract, bio and registration page.

Abstract:  Machine learning applications have sparked the development of specialized software frameworksand hardware accelerators. Yet, in today’s machine learning ecosystem, one important part of the system stack has received far less attention and specialization for ML: how we store and preprocess training data. This talk will describe the key challenges for implementing high-performance ML input data processing pipelines. We analyze millions of ML jobs running in Google's fleet and find that input pipeline performance significantly impacts end-to-end training performance and resource consumption. Our study shows that ingesting and preprocessing data on-the-fly during training consumes 30% of end-to-end training time, on average. Our characterization of input data pipelines motivates several systems research directions, such as disaggregating input data processing from model training and caching commonly reoccurring input data computation subgraphs. We present the multi-tenant input data processing service that we are building at ETH Zurich, in collaboration with Google,  to improve ML training performance and resource usage.

​Bio: Ana Klimovic is an Assistant Professor in the Systems Group of the Computer Science Department at ETH Zurich. Her research interests span operating systems, computer architecture, and their intersection with machine learning. Ana's work focuses on computer system design for large-scale applications such as cloud computing services, data analytics, and machine learning. Before joining ETH in August 2020, Ana was a Research Scientist at Google Brain and completed her Ph.D. in Electrical Engineering at Stanford University in 2019. Her dissertation research was on the design and implementation of fast, elastic storage for cloud computing.

Below, Ana receives the Best Poster Award at the 2018 Stanford-UCSC Workshop.

Optimizing AI Systems with Optical Technologies

Pre-registration is required. Please see the abstract, bio and registration page.

Abstract: Our society is rapidly becoming reliant on deep neural networks (DNNs). New datasets and models are invented frequently, increasing the memory and computational requirements for training. The explosive growth has created an urgent demand for efficient distributed DNN training systems. In this talk, I will discuss the challenges and opportunities for building next-generation DNN training clusters. In particular, I will propose optical network interconnects as a key enabler for building high-bandwidth ML training clusters with strong scaling properties. Our design enables accelerating the training time of popular DNN models using reconfigurable topologies by partitioning the training job across GPUs with hybrid data and model parallelism while ensuring the communication pattern can be supported efficiently on an optical interconnect. Our results show that compared to similar-cost interconnects, we can improve the training iteration time by up to 5x.

Bio: Manya Ghobadi is an assistant professor at the EECS department at MIT. Before MIT, she was a researcher at Microsoft Research and a software engineer at Google Platforms. Manya is a computer systems researcher with a networking focus and has worked on a broad set of topics, including data center networking, optical networks, transport protocols, and network measurement. Her work has won the best dataset award and best paper award at the ACM Internet Measurement Conference (IMC) as well as Google research excellent paper award.

MIT and UNIVERSITY of WASHINGTON WORKSHOP ON THE FUTURE OF CLOUD COMPUTING

AI IMPLEMENTATIONS & APPLICATIONS: ML ARCHITECTURES, SYSTEMS & PROGRAMMING ENVIRONMENTS

Pre-registration is required.

The AI Workshop is being co-organized by Prof. Arvind Krishnamurthy, UW (photo above), Prof. Manya Ghobadi and Prof. Mohammad Alizadeh, MIT CSAIL (photos below).

Apache TVM and other leading machine learning solutions were (and are being) developed at UW with widespread support from the community.

Hear from faculty and students from interdisciplinary research groups at UW and MIT, such as SAMPL (System, Architecture, Machine learning, and Programming languages), SYSLab (Computer Systems Lab), MODE (Machine learning, Optimization, Distributed systems, and Estadística), and PLSE (Programming Languages and Software Engineering), and SAMPA.

HIPEAC 2021 - European Network on High Performance and Embedded Architecture and Compilation - January 17-19, 2021

Exploiting Parallelism Opportunities with Deep Learning Frameworks, Yu Emma Wang, Carole-Jean Wu, Xiaodong Wang, Kim Hazelwood, and David Brooks

SMAUG: End-to-End Full-Stack Simulation Infrastructure for Deep Learning Workloads, Sam (Likun) Xi, Yuan Yao, Kshitij Bhardwaj, Paul Whatmough, Gu-Yeon Wei, and David Brooks

Refresh Triggered Computation: Improving the Energy Efficiency of Convolutional Neural Network Accelerators, Syed M. A. H. Jafri, Hasan Hassan, Ahmed Hemani, and Onur Mutlu

HPCA-27 - The 27th IEEE International Symposium on High-Performance Computer Architecture, Seoul, South Korea - March 1-3, 2021

Christina Giannoula (National Technical University of Athens / ETH Zürich); Nandita Vijaykumar (University of Toronto / ETH Zürich); Nikela Papadopoulou, Vasileios Karakostas (National Technical University of Athens); Ivan Fernandez (University of Malaga / ETH Zürich); Juan Gómez-Luna, Lois Orosa (ETH Zürich); Nectarios Koziris, Georgios Goumas (National Technical University of Athens); Onur Mutlu (ETH Zürich)

ASPLOS 2021 - The 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems - April 19-23, 2021

Yanqi Zhang, Weizhe Hua, Zhuangzhuang Zhou, Edward Suh, Christina Delimitrou (Cornell University)

Yu Gan, Mingyu Liang (Cornell University); Sundar Dev, David Lo (Google); Christina Delimitrou (Cornell University)

Guowei Zhang, Nithya Attaluri (MIT); Joel Emer (MIT & NVIDIA); Daniel Sanchez (MIT)

Irina Calciu (VMware Research); M. Talha Imran (The Pennsylvania State University); Ivan Puddu (ETH Zurich); Sanidhya Kashyap (Georgia Institute of Technology); Hasan Al Maruf (University of Michigan); Onur Mutlu (ETH Zurich); Aasheesh Kolli (The Pennsylvania State University and Google)

Joshua Landgraf, Tiffany Yang, Will Lin (UT Austin); Christopher J. Rossbach (UT Austin and VMware Research and Katana Graph); Eric Schkufza (Amazon)

Mark Wilkening, Udit Gupta, Samuel Hsia (Harvard University); Caroline Trippel (Facebook); Carole-Jean Wu (Facebook/ASU); David Brooks, Gu-Yeon Wei (Harvard University)

Call for papers is due on March 5.