“Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting,” a paper by Somesh Jha and his colleagues, will be presented at the 31st IEEE Computer Security Foundations Symposium July 9-12 at Oxford and has been chosen as a distinguished paper at that conference. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, the authors illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. They find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, their formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, they explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.
Founded in 1988, Computer Security Foundations Symposium (CSF) is an annual conference for researchers in computer security to examine current theories of security, the formal models that provide a context for those theories, and techniques for verifying security. Over the past two decades, many seminal papers and techniques have been presented first at CSF. Jha’s collaborators on the paper are Samuel Yeom, PhD student at Carnegie Mellon University; Irene Giacomelli, post-doctoral researcher at ISI Foundation; and Matt Fredrikson, Assistant Professor at Carnegie Mellon University.