Adaptive Data Analysis for Growing Data
Reuse of data in adaptive workflows poses challenges regarding overfitting and the statistical validity of results. Previous work has demonstrated that interacting with data via differentially private algorithms can mitigate overfitting, achieving worst-case generalization guarantees with asymptotic...
Saved in:
Main Authors | , |
---|---|
Format | Journal Article |
Language | English |
Published |
22.05.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Reuse of data in adaptive workflows poses challenges regarding overfitting
and the statistical validity of results. Previous work has demonstrated that
interacting with data via differentially private algorithms can mitigate
overfitting, achieving worst-case generalization guarantees with asymptotically
optimal data requirements. However, such past work assumes data is static and
cannot accommodate situations where data grows over time. In this paper we
address this gap, presenting the first generalization bounds for adaptive
analysis in the dynamic data setting. We allow the analyst to adaptively
schedule their queries conditioned on the current size of the data, in addition
to previous queries and responses. We also incorporate time-varying empirical
accuracy bounds and mechanisms, allowing for tighter guarantees as data
accumulates. In a batched query setting, the asymptotic data requirements of
our bound grows with the square-root of the number of adaptive queries,
matching prior works' improvement over data splitting for the static setting.
We instantiate our bound for statistical queries with the clipped Gaussian
mechanism, where it empirically outperforms baselines composed from static
bounds. |
---|---|
DOI: | 10.48550/arxiv.2405.13375 |