De-anonymizing text by fingerprinting language generation

Zhen Sun, Roei Schuster, Vitaly Shmatikov

    Research output: Contribution to journalConference articlepeer-review


    Components of machine learning systems are not (yet) perceived as security hotspots. Secure coding practices, such as ensuring that no execution paths depend on confidential inputs, have not yet been adopted by ML developers. We initiate the study of code security of ML systems by investigating how nucleus sampling—a popular approach for generating text, used for applications such as auto-completion—unwittingly leaks texts typed by users. Our main result is that the series of nucleus sizes for many natural English word sequences is a unique fingerprint. We then show how an attacker can infer typed text by measuring these fingerprints via a suitable side channel (e.g., cache access times), explain how this attack could help de-anonymize anonymous texts, and discuss defenses.

    Original languageEnglish
    JournalAdvances in Neural Information Processing Systems
    StatePublished - 2020
    Event34th Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online
    Duration: 6 Dec 202012 Dec 2020


    Dive into the research topics of 'De-anonymizing text by fingerprinting language generation'. Together they form a unique fingerprint.

    Cite this