IoCTopus: Outsmarting nonuniform DMA

Igor Smolyar, Alex Markuze, Boris Pismenny, Haggai Eran, Gerd Zellweger, Austin Bolen, Liran Liss, Adam Morrison, Dan Tsafrir

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In a multi-CPU server, memory modules are local to the CPU to which they are connected, forming a nonuniform memory access (NUMA) architecture. Because non-local accesses are slower than local accesses, the NUMA architecture might degrade application performance. Similar slowdowns occur when an I/O device issues nonuniform DMA (NUDMA) operations, as the device is connected to memory via a single CPU. NUDMA effects therefore degrade application performance similarly to NUMA effects. We observe that the similarity is not inherent but rather a product of disregarding the intrinsic differences between I/O and CPU memory accesses. Whereas NUMA effects are inevitable, we show that NUDMA effects can and should be eliminated. We present IOctopus, a device architecture that makes NUDMA impossible by unifying multiple physical PCIe functions-one per CPU-in manner that makes them appear as one, both to the system software and externally to the server. IOctopus requires only a modest change to the device driver and firmware. We implement it on existing hardware and demonstrate that it improves throughput and latency by as much as 2.7× and 1.28×, respectively, while ridding developers from the need to combat (what appeared to be) an unavoidable type of overhead.

Original languageEnglish
Title of host publicationASPLOS 2020 - 25th International Conference on Architectural Support for Programming Languages and Operating Systems
PublisherAssociation for Computing Machinery
Pages101-115
Number of pages15
ISBN (Electronic)9781450371025
DOIs
StatePublished - 9 Mar 2020
Event25th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2020 - Lausanne, Switzerland
Duration: 16 Mar 202020 Mar 2020

Publication series

NameInternational Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS

Conference

Conference25th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2020
Country/TerritorySwitzerland
CityLausanne
Period16/03/2020/03/20

Keywords

  • Bifurcation
  • DDIO
  • NUDMA
  • NUMA
  • OS I/O
  • PCIe

Fingerprint

Dive into the research topics of 'IoCTopus: Outsmarting nonuniform DMA'. Together they form a unique fingerprint.

Cite this