PyMM: Heterogeneous Memory Programming for Python Data Science

Daniel Waddington, Moshik Hershcovitch, Clem Dickey

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

While persistent memory (PMEM) is a promising technology, leveraging it with legacy applications is non-trivial. This is primarily because legacy applications assume all memory is volatile and there is no notion of crash-consistency or state recovery. As new types of persistent and intelligent memory emerge, propelled by the CXL standard, the problem of integration and adoption remains. In this paper we present PyMM, a framework for heterogeneous memory management in Python. It provides a means to abstract upon different memory types and their underlying traits (e.g., persistence, near/far). PyMM focuses on ease-of-use and employs an approach of sub-classing existing heavily-used types such as NumPy ndarray and PyTorch tensors. By doing so, PyMM allows new memory adoption with only minor modification to the application.

Original languageEnglish
Title of host publicationPLOS 2021 - Proceedings of the 2021 11th Workshop on Programming Languages and Operating Systems
PublisherAssociation for Computing Machinery, Inc
Pages31-37
Number of pages7
ISBN (Electronic)9781450387071
DOIs
StatePublished - 25 Oct 2021
Externally publishedYes
Event11th Workshop on Programming Languages and Operating Systems, PLOS 2021 - Virtual, Online, Germany
Duration: 25 Oct 2021 → …

Publication series

NamePLOS 2021 - Proceedings of the 2021 11th Workshop on Programming Languages and Operating Systems

Conference

Conference11th Workshop on Programming Languages and Operating Systems, PLOS 2021
Country/TerritoryGermany
CityVirtual, Online
Period25/10/21 → …

Fingerprint

Dive into the research topics of 'PyMM: Heterogeneous Memory Programming for Python Data Science'. Together they form a unique fingerprint.

Cite this