Calendar | CS 4650

A tentative plan for the schedule (topics, deadlines, etc.) is available here.

CS 4650 covers the foundations of Deep Learning for Natural Language Processing, including mathematical foundations, neural networks, classic algorithms such as Viterbi, the origins of language modeling development, and evaluation methods such as F1 and MT metrics.

For more cutting-edge LLM topics, we have developed a dedicated course, CS 7652 “Large Language Models”, which we recommend taking after CS 4650 or CS 7650.

Schedule (subject to change as the term progresses)

Resource

PyTorch Tutorial (w/ links to Colab notebook)

Aug 24

Course Overview: Eisenstein 1

Aug 27

Problem Set 0 (math background test) due

Aug 31

Machine Learning Review - linear classification: Eisenstein 1, J+M B, J+M 4

Sep 2

Machine Learning Review - mutliclass classification: Eisenstein 2.0-2.5, 4.1, 4.3-4.5, J+M 4

Sep 9

Neural Networks - Feedforward, optimization: Eisenstein 2.6, 3.1-3.3, J+M 6, Goldberg 1-4, J.G. Makin - Backpropagation

Sep 16

Word Embeddings, Neural Networks - CNN: Eisenstein 3.3.4, 14.5, 14.6, J+M 5, Goldberg 5; Eisenstein 3.4, 7.6, Goldberg 9

Sep 21

Sequence Models - HMM, Viterbi: Eisenstein 7.0-7.4, J+M 18, J+M A

Sep 23

Conditional Random Fields: Eisenstein 7.5, 8.3, J+M 18

Sep 28

Recurrent Neural Networks + Neural CRFs: Eisenstein 3.4, 7.6, Goldberg 10-11, J+M 14

Sep 30

Encoder-Decoder + Attention: Eisenstein 18.3 - 18.5

Oct 7

Transformer, Layer Normalization, Adam Optimizer: J+M 7.1-7.4, Vaswani+17 Transformers, Alammar’s blog post, Rush’s tutorial

Oct 12

Transformer(recap), LLM Pre-training: Masked Language Models: J+M 9, ELMo BERT

Oct 14

Knowledge Distillation, Autoregressive Language Models: J+M 7.5-7.7, Knowledge Distillation, GPT-3

Oct 19

In-context Learning, Scaling Laws: GPT-3

Oct 21

Instruction Tuning, Decoding: J+M 7, Hinton+15 Knowledge Distillation,

Oct 28

In-class Midterm (close book, close note)

TBA

Post-training of Language Models - part 3 (InstructGPT, preference optimization, decoding): InstructGPT

TBA

Open-source Language Models - part 4 (LLaMA, normalization, RoPE, AdamW, etc.): Llama 3

TBA

Open-source Language Models - part 5 (tokenization, BPE, multilinguality)): BPE