Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

• Main Authors: Aleo, P. D, Engel, A. W, Narayan, G, Angus, C. R, Malanchev, K, Auchettl, K, Baldassare, V. F, Berres, A, de Boer, T. J. L, Boyd, B. M, Chambers, K. C, Davis, K. W, Esquivel, N, Farias, D, Foley, R. J, Gagliano, A, Gall, C, Gao, H, Gomez, S, Grayling, M, Jones, D. O, Lin, C. -C, Magnier, E. A, Mandel, K. S, Matheson, T, Raimundo, S. I, Shah, V. G, Soraisam, M. D, de Soto, K. M, Vicencio, S, Villar, V. A, Wainscoat, R. J
• Format: Journal Article
• Language: English
• Published: 01.04.2024
• Subjects: Physics - High Energy Astrophysical Phenomena; Physics - Instrumentation and Methods for Astrophysics
• DOI: 10.48550/arxiv.2404.01235

We present LAISS (Lightcurve Anomaly Identification and Similarity Search), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly ZTF Alert Stream via the ANTARES broker, identifying a manageable $\sim$1-5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host-galaxy features within a random forest classifier, tagging transients of rare classes (spectroscopic anomalies), of uncommon host-galaxy environments (contextual anomalies), and of peculiar or interaction-powered phenomena (behavioral anomalies). Moreover, we demonstrate the power of a low-latency ($\sim$ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host-galaxy environments. We use analogs for data-driven discovery, characterization, (re-)classification, and imputation in retrospective and real-time searches. To date we have identified $\sim$50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to: SLSNe, TDEs, SNe IIn, SNe IIb, SNe Ia-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018-2021 and absent from public catalogs ($\sim$1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the "needle in the haystack" in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.