The emergence of clusters in self-attention dynamics

Event Type
Seminar/Symposium
Sponsor
DECISION AND CONTROL
Location
Date
October 5, 2023 4:00 PM
Speaker
Postdoctoral Associate, Borjan Geshkovski from MIT
Cost
Registration
Contact
Rhonda Henderson
Email
rrhender@illinois.edu
Phone
217-300-8511

A picture containing text, clipart

Description automatically generated

DECISION AND CONTROL LECTURE

THE GRAINGER COLLEGE OF ENGINEERING

TITLE

The emergence of clusters in self-attention dynamics

Sponsor

Decision and Control

Date

Thursday, October 5, 2023

Time

4:00 PM

LOCATION

Coordinated science lab, rm b02

                               Speaker:  Postdoctoral Associate, Borjan Geshkovski from MIT

 

ABSTRACT

With remarkable empirical success, Transformers enable large language models to compute very powerful representations using the self-attention mechanism. We model this mechanism as an interacting particle systems to brings and demonstrate the formation of clusters as the number of layers goes to infinity. Based on joint work with Cyril Letrouit (CNRS), Yury Polyanskiy (MIT) and Philippe Rigollet (MIT).

BIO

Borjan Geshkovski is currently a postdoc at MIT Math, where he works with Philippe Rigollet. He got his PhD from the Autonomous University in Madrid under the supervision of Enrique Zuazua. His research interests are centred around control, learning and PDE.