Dear all,
I hope you will join us for the February LINCC Tech Talk session that will take place on Thursday, February 13, at 10h PT = 13h ET = 15h CLT = 19h CET on Zoom (https://ls.st/lincc-talks ). We will hear from Francois Lanusse, who will talk about multi-modal foundation models for astronomy.
One Network to Rule Them All: Multi-Modal Foundation Models for Survey Astronomy
Deep Learning has seen a recent shift in paradigm, from training specialized models on dedicated datasets, so-called Foundation Models, trained in a self-supervised manner on vast amounts of data and then adapted to solve specific tasks with state-of-the-art performance. This new paradigm has been exceptionally successful not only for large language models (LLMs) but in other domains such as vision models.
In the context of the Vera C. Rubin Observatory Legacy Survey of Space and Time, this new paradigm would translate into our community investing in a few large-scale models, trained on the totality of Rubin data and a number of external datasets. These models could then serve as a strong basis for many downstream applications without the need for individual researchers to access large amounts of training data or develop complex and expensive models on their own.
In this talk I will discuss several methodologies that can be used to reach such a goal, in particular weakly-supervised cross-modal contrastive learning and multimodal self-supervised generative pretraining. I will show how these approaches can be used to build models flexible to very diverse and inhomogeneous observations (e.g. different types of measurements such as spectra, time-series, or images, but also different instruments, etc…) and how they can then be leveraged by the end-user for a variety of downstream applications (e.g. redshift estimation, morphology classification, physical parameter inference, searching for rare objects) with very simple machine learning methods and still reach near-optimal performance.
References:
AstroCLIP: https://arxiv.org/abs/2310.03024
The Multimodal Universe: https://arxiv.org/abs/2412.02527
LINCC Tech Talks are held on the second Thursday of every month. Events are also advertised at our web page and also provided in calendar form ; and the #lincc-tech-talks LSSTC Slack channel is always available for discussions before, during, and after the talks.