AutoML Two-Sample Test

Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require spec...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Kübler, Jonas M, Stimper, Vincent, Buchholz, Simon, Muandet, Krikamol, Schölkopf, Bernhard
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 15.01.2023
Subjects
Online AccessGet full text

Cover

Loading…
Abstract Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Without any user input about the problems at hand, and using the same method for all our experiments, our AutoML two-sample test achieves competitive performance on a diverse distribution shift benchmark as well as on challenging two-sample testing problems. We provide an implementation of the AutoML two-sample test in the Python package autotst.
AbstractList Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Without any user input about the problems at hand, and using the same method for all our experiments, our AutoML two-sample test achieves competitive performance on a diverse distribution shift benchmark as well as on challenging two-sample testing problems. We provide an implementation of the AutoML two-sample test in the Python package autotst.
Author Buchholz, Simon
Kübler, Jonas M
Muandet, Krikamol
Stimper, Vincent
Schölkopf, Bernhard
Author_xml – sequence: 1
  givenname: Jonas
  surname: Kübler
  middlename: M
  fullname: Kübler, Jonas M
– sequence: 2
  givenname: Vincent
  surname: Stimper
  fullname: Stimper, Vincent
– sequence: 3
  givenname: Simon
  surname: Buchholz
  fullname: Buchholz, Simon
– sequence: 4
  givenname: Krikamol
  surname: Muandet
  fullname: Muandet, Krikamol
– sequence: 5
  givenname: Bernhard
  surname: Schölkopf
  fullname: Schölkopf, Bernhard
BookMark eNrjYmDJy89LZWLgNDI2NtS1MDEy4mDgLS7OMjAwMDIzNzI1NeZkEHMsLcn39VEIKc_XDU7MLchJVQhJLS7hYWBNS8wpTuWF0twMym6uIc4eugVF-YWlQAXxWfmlRXlAqXigURam5paGJqbGxKkCAMkEKqA
ContentType Paper
Copyright 2023. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2023. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID 8FE
8FG
ABJCF
ABUWG
AFKRA
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
HCIFZ
L6V
M7S
PIMPY
PQEST
PQQKQ
PQUKI
PRINS
PTHSS
DatabaseName ProQuest SciTech Collection
ProQuest Technology Collection
Materials Science & Engineering Collection
ProQuest Central (Alumni)
ProQuest Central
ProQuest Central Essentials
ProQuest Central
Technology Collection
ProQuest One Community College
ProQuest Central Korea
SciTech Premium Collection
ProQuest Engineering Collection
Engineering Database
Publicly Available Content Database
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Academic
ProQuest One Academic UKI Edition
ProQuest Central China
Engineering Collection
DatabaseTitle Publicly Available Content Database
Engineering Database
Technology Collection
ProQuest Central Essentials
ProQuest One Academic Eastern Edition
ProQuest Central (Alumni Edition)
SciTech Premium Collection
ProQuest One Community College
ProQuest Technology Collection
ProQuest SciTech Collection
ProQuest Central China
ProQuest Central
ProQuest Engineering Collection
ProQuest One Academic UKI Edition
ProQuest Central Korea
Materials Science & Engineering Collection
ProQuest One Academic
Engineering Collection
DatabaseTitleList Publicly Available Content Database
Database_xml – sequence: 1
  dbid: 8FG
  name: ProQuest Technology Collection
  url: https://search.proquest.com/technologycollection1
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Physics
EISSN 2331-8422
Genre Working Paper/Pre-Print
GroupedDBID 8FE
8FG
ABJCF
ABUWG
AFKRA
ALMA_UNASSIGNED_HOLDINGS
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
FRJ
HCIFZ
L6V
M7S
M~E
PIMPY
PQEST
PQQKQ
PQUKI
PRINS
PTHSS
ID FETCH-proquest_journals_26785791453
IEDL.DBID 8FG
IngestDate Thu Oct 10 20:17:37 EDT 2024
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-proquest_journals_26785791453
OpenAccessLink https://www.proquest.com/docview/2678579145?pq-origsite=%requestingapplication%
PQID 2678579145
PQPubID 2050157
ParticipantIDs proquest_journals_2678579145
PublicationCentury 2000
PublicationDate 20230115
PublicationDateYYYYMMDD 2023-01-15
PublicationDate_xml – month: 01
  year: 2023
  text: 20230115
  day: 15
PublicationDecade 2020
PublicationPlace Ithaca
PublicationPlace_xml – name: Ithaca
PublicationTitle arXiv.org
PublicationYear 2023
Publisher Cornell University Library, arXiv.org
Publisher_xml – name: Cornell University Library, arXiv.org
SSID ssj0002672553
Score 3.4448419
SecondaryResourceType preprint
Snippet Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to...
SourceID proquest
SourceType Aggregation Database
SubjectTerms Machine learning
Test procedures
Title AutoML Two-Sample Test
URI https://www.proquest.com/docview/2678579145
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwY2BQSUwxNE5MNEvVNTE3MdA1MTJJ1U0yS0nUBXYVDNNM0gxMUhNBu5F9_cw8Qk28IkwjoANuxdBllbAyEVxQp-Qng8bI9Y2ApaqpuaWhial9QaEu6NYo0Owq9AoNZgZWQyNzM9CSPgs3d_gYC1APsMVsjFHMgusON0EG1oDEgtQiIQam1DxhBnbwksvkYhEGMcfSknxfH4WQ8nzd4ETQIb0KIUC9ogzKbq4hzh66MMPiodFdHI9wnLEYAwuw354qwaBgmpKWYpBkYZZiBjoSxcQwCVj5JlmapaSYJKdYGiYaSDLI4DNJCr-0NAMX6OZz0GiAoakMA0tJUWmqLLB-LEmSAweCHAOrk6tfQBCQ51vnCgCRJG5w
link.rule.ids 780,784,12765,21388,33373,33744,43600,43805
linkProvider ProQuest
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3PS8MwFH7ohujNX0Pd1IJeg836kq0nEbFWbYdghd1K0pce7bZ2-O-blE4Pws7hPUIC35f35f0AuFXEA6WkYThBn-EYDdOSFLOhAi-x9NEoV42czmT8ia9zMe8Et7pLq9xgYgvUVBVOI78bW1QVk5CjuF8smZsa5X5XuxEau9DHwFK3qxSPnn81FmtjX8zBP5htuSM6hP67WpjVEeyYr2PYa1Mui_oEBg_rpkoTL_uu2IdyTXq9zNqewk30lD3GbOMs7667zv82FwygZ-N2cwaeoJJ8PZUkXUsU5NqSrw4lERYUcuWfw2ibp4vty9ewH2dpkicvs7chHLgp6E4Z4GIEvWa1NpeWKxt91R7ID8FEboc
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AutoML+Two-Sample+Test&rft.jtitle=arXiv.org&rft.au=K%C3%BCbler%2C+Jonas+M&rft.au=Stimper%2C+Vincent&rft.au=Buchholz%2C+Simon&rft.au=Muandet%2C+Krikamol&rft.date=2023-01-15&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422