Dawn Song

Life and career

The photograph that made Dawn Song's research legible to the general public shows an ordinary octagonal stop sign with a few strips of black and white tape stuck to it. To a human driver it is a stop sign with graffiti. To a well-trained deep network classifier, under a range of viewing distances and angles, it is a speed limit sign. That image — from work she and her collaborators published in 2018 — did more to puncture confidence in deployed computer vision than any amount of benchmark analysis, and it is characteristic of how she works: find the assumption everyone is making, then break it in public.

Song was born in China and studied at Tsinghua University before coming to the United States for graduate work, taking a master's degree at Carnegie Mellon and a doctorate at UC Berkeley in the early 2000s. Her training was in computer security, not machine learning, and that ordering matters more than almost anything else about her career. Security researchers are professionally trained to reason about an adversary who knows your defense, who has read your paper, and who is optimizing against you — an epistemology fundamentally different from the machine learning tradition of measuring average-case accuracy on a held-out set drawn from the same distribution as training.

She began her faculty career at Carnegie Mellon before returning to Berkeley, where she has been ever since in the EECS department. Her early work spanned the classic security agenda — network protocols, binary program analysis, malware, automatic vulnerability discovery — and produced a series of results that are still cited as foundational, including work showing that encrypted interactive sessions leak substantial information through packet timing alone, and work on automatically generating exploits from released patches, which reframed how the industry thinks about patch disclosure windows.

She was named a MacArthur Fellow in 2010. As deep learning became infrastructure over the following decade, she redirected the same adversarial instincts at it, becoming one of the central figures in adversarial machine learning and in the privacy analysis of trained models. She has also built institutions and companies, founding a startup in privacy-preserving computation and directing Berkeley centers focused on decentralized and responsible AI, and in recent years she has contributed to the evaluation infrastructure the field uses to measure large language model capability and behavior.

Key contributions

**Physical-world adversarial examples.** The theoretical result that neural networks are vulnerable to tiny worst-case input perturbations was established by others; the question that mattered for deployed systems was whether such attacks survive contact with reality — printing, weather, distance, viewing angle, camera sensor noise. Song's group answered it. Their robust physical-world attack framework optimizes a perturbation not against a single image but against a *distribution* of imaging conditions, and constrains the perturbation to a physically realizable form (printable stickers in a bounded region) rather than an arbitrary pixel-space delta. The stop sign result showed that a production-grade classifier could be reliably fooled from a moving vehicle. This closed the gap between a theoretical curiosity and a safety concern.

**Transferability and black-box attacks.** With collaborators she studied how adversarial examples crafted against one model transfer to another with a different architecture trained on different data. Transferability is what makes black-box attacks practical: an attacker who cannot query your model can attack a surrogate and carry the perturbation over. Her group's work on generating transferable examples, particularly for targeted attacks, established both that this works and roughly how well.

**Honest evaluation against adaptive adversaries.** A methodological contribution that may outlast the specific attacks. A recurring failure mode in the defense literature is *gradient obfuscation*: a defense that makes gradients uninformative appears robust against standard attacks while providing no actual security, since an adaptive adversary who understands the defense can route around it. Song has been among the researchers insisting that a defense must be evaluated against an attacker who knows it, and a great many published defenses have not survived that standard.

**Memorization and privacy of trained models.** With Nicholas Carlini and others, Song helped establish that neural language models memorize and can be induced to emit rare training sequences verbatim. The methodology — inserting canary sequences of known format into training data and then measuring their extractability via the model's own likelihoods — gives a quantitative measure of unintended memorization rather than an anecdote. The follow-on work demonstrating extraction of genuine training data from large deployed language models is one of the most consequential privacy results of the deep learning era, and it directly motivated the differentially private and deduplicated training pipelines now common.

**Privacy-preserving computation.** Her earlier work on practical techniques for searching over encrypted data is a foundational reference in that literature — allowing a server to answer keyword queries over ciphertext it cannot read. This thread runs through her later work on federated and differentially private learning, and on systems that let mutually distrustful parties train a shared model without pooling raw data.

**Program analysis and synthesis.** A quieter but substantial line: dynamic taint analysis for tracking untrusted data through a running binary, automatic signature generation for polymorphic worms, and later neural approaches to program synthesis and code understanding.

In battle

Song's card is bimodal in the way most security researchers' cards are: mean 26.2, median 18, with 58 problems at or below 20, but eight dominant problems and a peak of 97. She is close to unbeatable inside the security-and-privacy band and close to useless outside it.

**P239, "The sticker that erases the stop sign"** (97) is her own published result, down to the physical realizability constraint and the demand that the attack survive real viewing conditions. The problem's caution that obscured gradients have already embarrassed a round of published defenses is a direct statement of the evaluation standard she helped enforce. **P152, "The pixel you cannot see"** (90) is the imperceptible-perturbation attack in its canonical form.

Her privacy cluster is equally strong. **P294, "The model memorized the secrets"** (95) is the canary-extraction methodology she co-developed. **P290, "Anonymous, except for everything about you"** (88) is re-identification from ostensibly anonymized records, **P293, "Train on the phones, keep the secrets"** (88) is federated learning, **P098, "The statistic that can't betray you"** (85) is differential privacy in its formal statement, and **P291, "The census under a privacy budget"** (80) is its most consequential deployment. **P258, "The inbox under siege"** (87) rewards her long history in adversarial systems where a motivated opponent adapts to every filter.

Her `fairness` category average of 52.7 across fifteen problems is the top of her card, but read it carefully — in this game's taxonomy that band absorbs privacy and accountability problems, and it is privacy that is carrying her, not fairness proper.

Now the losses, which are stark and unusually coherent. Song scores near zero on classical statistics of every kind. **P118, "The factor you can't keep changing"** (4) — split-plot design with whole-plot and subplot error strata — is her floor, and her `experimental-design` average of 5.5 is the lowest category on her card. **P115, "Where to place the measurements"** (7) is optimal design, **P140, "The ruler that lies a little"** (7) is measurement error and attenuation, **P126, "When the bell curve won't hold"** (5) is nonparametric inference, and **P184, "Fill in the hidden variables"** (6) is the EM algorithm. Her `testing` average of 12.2 across sixteen problems and `regression` at 12.5 tell the same story: this is a researcher whose entire toolkit assumes an adversary, and classical statistics assumes noise instead.

She is also weak in reinforcement learning and games — **P201, "The dice make it learnable"** (5), `games` at 9.0 — and, more surprisingly, only middling in `systems` (20.7 across eighteen problems) despite her security background. The reason is that the game's systems problems are largely about distributed architecture, reliability, and scale, which is adjacent to but distinct from the attack-and-analysis tradition she comes from.

The practical rule: play Song whenever a problem mentions an adversary, a leak, an extraction, or a privacy budget. Do not play her when the problem is about estimating something carefully from limited data — she brings the wrong century of statistics to that fight.