Calendar | CS 4650

A tentative plan for the schedule (topics, deadlines, etc.) is available here.

CS 4650 covers the foundations of Deep Learning for Natural Language Processing, including mathematical foundations, neural networks, classic algorithms such as Viterbi, the origins of language modeling development, and evaluation methods such as F1 and MT metrics.

For more cutting-edge LLM topics, we have developed a dedicated course, CS 7652 “Large Language Models”, which we recommend taking after CS 4650.

Schedule (subject to change as the term progresses)

Resource

PyTorch Tutorial (w/ links to Colab notebook)

TBA

Course Overview: Eisenstein 1

TBA (to be updated)

Machine Learning Review - linear classification: Eisenstein 1, J+M 4

TBA

Machine Learning Review - logistic regression, perceptron, SVM: Eisenstein 2.0-2.5, 4.1, 4.3-4.5, J+M B

TBA

Machine Learning Review - mutliclass classification: Eisenstein 2.0-2.5, 4.1, 4.3-4.5, J+M 4

TBA

Neural Networks - Feedforward, optimization: Eisenstein 2.6, 3.1-3.3, J+M 6, Goldberg 1-4, J.G. Makin - Backpropagation

TBA

Word Embeddings: Eisenstein 3.3.4, 14.5, 14.6, J+M 5, Goldberg 5

TBA

Sequence Models - HMM, Viterbi: Eisenstein 7.0-7.4, J+M 17, J+M A

TBA

Conditional Random Fields: Eisenstein 7.5, 8.3, J+M 17

TBA

Recurrent Neural Networks + Neural CRFs: Eisenstein 3.4, 7.6, Goldberg 10-11, J+M 13

TBA

Encoder-Decoder + Attention: Eisenstein 18.3 - 18.5

TBA

Transformer: J+M 8, Vaswani+17 Transformers, Alammar’s blog post, Rush’s tutorial

TBA

Convolutional Neural Networks, MT Evaluation: Eisenstein 3.4, 7.6, Goldberg 9

TBA

Pretrained Language Models - part 1 (ELMo, BERT & variants, BART/T5): J+M 10, ELMo BERT, BART

TBA

Pretrained Language Models - part 2 (GPT2/3, knowledge distillation, instruction tuning): J+M 7, Hinton+15 Knowledge Distillation, GPT-3

TBA

Post-training of Language Models - part 3 (InstructGPT, preference optimization, decoding), Midterm Review: InstructGPT

TBA

Open-source Language Models - part 4 (LLaMA, normalization, RoPE, AdamW, etc.): Llama 3

TBA

Open-source Language Models - part 5 (tokenization, BPE, multilinguality)): BPE

TBA

In-class Midterm (close book, close note)