CS 4650 - Natural Language Processing (Georgia Tech)
Natural Language Process (Georgia Tech)
- Instructor: Wei Xu
- Lecture: Mondays, Wednesdays 2:00-3:15pm
Schedule (subject to change as the term progresses)
- Resource
- PyTorch Tutorial (w/ links to Colab notebook)
- Aug 24
- Course Overview
- Eisenstein 1
- Aug 27
- Problem Set 0 (math background test) due
- Aug 31
- Machine Learning Review - linear classification
- Eisenstein 1, J+M B, J+M 4
- Sep 2
- Machine Learning Review - mutliclass classification
- Eisenstein 2.0-2.5, 4.1, 4.3-4.5, J+M 4
- Sep 9
- Neural Networks - Feedforward, optimization
- Eisenstein 2.6, 3.1-3.3, J+M 6, Goldberg 1-4, J.G. Makin - Backpropagation
- Sep 16
- Word Embeddings, Neural Networks - CNN
- Eisenstein 3.3.4, 14.5, 14.6, J+M 5, Goldberg 5; Eisenstein 3.4, 7.6, Goldberg 9
- Sep 21
- Sequence Models - HMM, Viterbi
- Eisenstein 7.0-7.4, J+M 18, J+M A
- Sep 23
- Conditional Random Fields
- Eisenstein 7.5, 8.3, J+M 18
- Sep 28
- Recurrent Neural Networks + Neural CRFs
- Eisenstein 3.4, 7.6, Goldberg 10-11, J+M 14
- Sep 30
- Encoder-Decoder + Attention
- Eisenstein 18.3 - 18.5
- Oct 7
- Transformer, Layer Normalization, Adam Optimizer
- J+M 7.1-7.4, Vaswani+17 Transformers, Alammar’s blog post, Rush’s tutorial
- Oct 12
- Transformer(recap), LLM Pre-training: Masked Language Models
- J+M 9, ELMo BERT
- Oct 14
- Knowledge Distillation, Autoregressive Language Models
- J+M 7.5-7.7, Knowledge Distillation, GPT-3
- Oct 19
- In-context Learning, Scaling Laws
- GPT-3
- Oct 21
- Instruction Tuning, Decoding
- J+M 7, Hinton+15 Knowledge Distillation,
- Oct 28
- In-class Midterm (close book, close note)
- TBA
- Post-training of Language Models - part 3 (InstructGPT, preference optimization, decoding)
- InstructGPT
- TBA
- Open-source Language Models - part 4 (LLaMA, normalization, RoPE, AdamW, etc.)
- Llama 3
- TBA
- Open-source Language Models - part 5 (tokenization, BPE, multilinguality))
- BPE