Calendar | CS 8803

A tentative plan for the schedule (topics, deadlines, etc.) is available here.

Schedule (subject to change as the term progresses)

Jan 12

Overview

Course OverviewHow to read a paper

Feb 2

Pretraining

Chameleon: A Flexible Data-mixing Framework for Language Mo…paper

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fa…paper

Feb 4

Embeddings

Improving Text Embeddings with Large Language Modelspaper

MMTEB: Massive Multilingual Text Embedding Benchmarkpaper

Feb 9

Embeddings

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of…paper

NV-Embed: Improved Techniques for Training LLMs as Generali…paper

Feb 11

Mixture-of-Experts

Outrageously Large Neural Networks: The Sparsely-Gated Mixt…paper

Your Mixture-of-Experts LLM Is Secretly an Embedding Model …paper

Feb 16

Reasoning

BIRD: A Trustworthy Bayesian Inference Framework for Large …paper

Feb 18

Reasoning / Alignment

Direct Preference Optimization: Your Language Model is Secr…paper

Training a Generally Curious Agentpaper

Feb 23

Agent Harness

Agent Workflow Memorypaper

The OpenHands Software Agent SDK: A Composable and Extensib…paper

Feb 25

Long-Context

Efficient Streaming Language Models with Attention Sinkspaper

Mar 2

Attention

DeepSeek-V3.2: Pushing the Frontier of Open Large Language …paper

TransMLA: Multi-Head Latent Attention Is All You Needpaper

Mar 4

Reinforcement Learning

GRPO, DAPO, and "Understanding R1-Zero-Like Training"

Mar 9

Reasoning

Optimas: Optimizing Compound AI Systems with Globally Align…paper

Mar 11

Reasoning

CWM: An Open-Weights LLM for Research on Code Generation wi…paper

Mar 16

Self-Play RL

Absolute Zero: Reinforced Self-play Reasoning with Zero Datapaper

Mar 18

Self-Play RL

SPICE: Self-Play In Corpus Environments Improves Reasoningpaper

Toward Training Superintelligent Software Agents through Se…paper

Mar 30

Test-Time Scaling

Learning to Discover at Test Timepaper

Scaling LLM Test-Time Compute Optimally Can be More Effecti…paper

Apr 1

Mode Collapse

Artificial Hivemind: The Open-Ended Homogeneity of Language…paper

Apr 6

Linear Transformer

Linear transformers are secretly fast weight programmerspaper

Parallelizing Linear Transformers with the Delta Rule over …paper

Apr 8

Diffusion LM

Diffusion-LM Improves Controllable Text Generationpaper

Large Language Diffusion Modelspaper

Apr 13

Safety

Emergent Misalignment: Narrow finetuning can produce broadl…paper

On the Role of Attention Heads in Large Language Model Safe…paper

Apr 15

Mechanistic Interpretability

Scaling and Evaluating Sparse Autoencoderspaper

Sparse Crosscoders for Cross-Layer Features and Model Diffi…paper

Apr 20

Calibration

Active Task Disambiguation with LLMspaper

Learning to Route LLMs with Confidence Tokenspaper

Apr 22

Scaling Law

Scaling Laws for Precisionpaper

Training compute-optimal large language modelspaper