Xudong Liang (Brandon)

NLP Resource

Bayes Theorem, Hidden Markov Models, Conditional Random Field

• Understand Bayes Rule, Likelihood, Prior and Posterior
• MLE, MAP and Bayesian Inference
• A Friendly Introduction to Bayes Theorem and Hidden Markov Models
• Simple Implementation of the Viterbi Algorithm for training Hidden Markov Models (Github)
• Intuitive difference between Hidden Markov models and Conditional Random Fields
• CS838-1 Advanced NLP: Conditional Random Field
• Overview of Conditional Random Field
• Implement Conditional Random Field in PyTorch
• Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
• HMM vs RNN
• Part-of-Speech Tagging with Trigram Hidden Markov Models and the Viterbi Algorithm

Neural Networks

• A Visual and Interactive Guide to the Basics of Neural Networks
• A Visual And Interactive Look at Basic Neural Network Math
• [YouTube] Neural Networks: Zero to Hero (7-video playlist)
• "AI canon" - a list of influential resources

RNN

• The Unreasonable Effectiveness of Recurrent Neural Networks
• LSTM: From Zero to Hero with PyTorch
• Difference Between "Output" and "Hidden States (& Cell States)" in LSTM (PyTorch)
• The fall of RNN/LSTM
• LSTM time complexity

Word Embeddings: Word2Vec - CBOW, Skigram (SGNS, SVD, PPMI, Neural Word Embedding); GloVe

• The Illustrated Word2vec
• CS224n: Word Vectors: Intro, SVD and Word2Vec
• Word2Vec - Skip Gram vs CBOW
• Stanford NLP: GloVe - Global Vectors for Word Representation
• Word2Vec
• Ngram2Vec
• SVD, PPMI, SGNS, Neural Word Embedding as Implicit Matrix Factorization (Paper)
• Language Models and Skipgram Recommenders Talk @ MIT
• Skipgram Recommender Talk, Intuition and Use-Cases of Embeddings in NLP & beyond @ Qcon London
• Intuition & Use-Cases of Embeddings in NLP & beyond - YouTube
• (Notebook) 01 - Exploring Word Embeddings (Gensim)
• (Notebook) 02 - Song Embeddings - Skipgram Recommender (Gensim)
• (Paper) Real-time Personalization using Embeddings for Search Ranking at Airbnb
• (Paper) Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba
• word2vec: negative sampling (in layman term)?
• - Word2Vec: use words to predict words
• -- CBOW: predict the word in the middle
• -- Skip-gram (SGNS): predict the context window (predict neighboring word)
• - fastText
• -- Same goal as Word2Vec: learn vector rep of words
• -- But, uses character n-grams and character vector to predict words
• -- Words are represented by the sum of character n-gram vectors
• - GloVe
• -- Global word-word co-occurrence and matrix factorization for word embedding on word-context matrix
• -- PPMI

Transformers, Attention, Self-Attention

• Transformer Tokenizer Summary
• Attention
• Attention, Self-Attention, Families of Attention, and more
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
• Neural Machine Translation (seq2seq) Tutorial With Attention (Tensorflow Github)
• (Paper) Attention Is All You Need - proposed Transformer
• The Illustrated Transformer (with Self-Attention)
• The Annotated Transformer - "Attention is all you need" - Harvard NLP
• Google AI Blog - Transformer: A Novel Neural Network Architecture for Language Understanding
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)
• BERT - Slides - By Jacob Devlin - Google AI Language
• Source code for transformers.modeling_bert (Hugging Face)
• BERT Word Embeddings Tutorial
• A Visual Guide to Using BERT for the First Time (Hugging Face)
• (Notebook) 03 - Sentence Classification with BERT
• (Notebook) Attention Flow Graph Tools
• (Hugging Face Tutorial) How to train a new language model from scratch using Transformers and Tokenizers
• Hugging Face Transformers Optimizer_Schedules (e.g.: transformers.get_linear_schedule_with_warmup)
• Hugging Face
• PyTorch: Translation With A Sequence To Sequence Network And Attention
• Interfaces for Explaining Transformer Language Models
• Finding the Words to Say: Hidden State Visualizations for Language Models
• BART Explained | Papers With Code
• Computational Complexity of Self-Attention in the Transformer Model (vs. RNN)
• BERT Optimization: FasterTransformer BERT [NVIDIA]
• BERT Optimization: Effective Transformer [ByteDance]
• A BetterTransformer for Fast Transformer Inference [PyTorch]
• "AI canon" - a list of influential resources
• Attention is off by one - Evan Miller
• Transformer/Attention time complexity, encoder-decoder attention, encoder self-attention, masked decoder self-attention

LLM, GPT & Few-Shot & Zero-Shot Learning

• Byte Pair Encoding
• GPT Explained | Papers With Code
• GPT-2 Explained | Papers With Code
• The Illustrated GPT-2 (Visualizing Transformer Language Models)
• The Annotated GPT-2
• Source code for transformers.modeling_gpt2 (Hugging Face)
• Hugging Face GPT2Config
• What Makes Good In-Context Examples for GPT-3?
• Learning to Write: Language Generation With GPT-2
• AllenAI GPT-2 Explorer
• Finetuned Language Models Are Zero-Shot Learners
• Making Pre-trained Language Models Better Few-Shot Learners
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Pattern-Exploiting Training
• PROMPT WAYWARDNESS: The Curious Case of Discretized Interpretation of Continuous Prompts (investigates what prompts are actually learning)
• What Context Features Can Transformer Language Models Use (investigates what information is acutally useful from very long contexts)
• Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Perplexity in Language Models
• NLG with GPT-2 - Jake Tae
• HuggingFace: How to generate text: using different decoding methods for language generation with Transformers
• New Scaling Laws for Large Language Models
• Mismatch between logits from generate and forward with an attention mask for most GPT models
• Getting logits from T5 Hugging Face model using forward() method without labels
• How to use forward() method instead of model.generate() for T5 model -- you should not use forward() for generation, why?
• Better not bigger: How to get GPT-3 quality at 0.1% the cost
• InstructGPT: Aligning Language Models to Follow Instructions
• InstructGPT Model Card
• Training language models to follow instructions with human feedback
• Chain of Thought Prompting Elicits Reasoning in Large Language Models
• Language Models Perform Reasoning via Chain of Thought
• ChatGPT: Optimizing Language Models for Dialogue
• Lessons Learned on Language Model Safety and Misuse
• Understanding Large Language Models - A Cross-Section of the Most Relevant Literature to Get Up To Speed
• Finetuning Large Language Models - an Introduction to the core ideas and appraoches - in-context learning and finetuning
• Prompt Engineering Guide
• Prompt Engineering
• Large Tnrasformer Model Inference Optimization
• Building LLM applications for production
• [Openai] Prompt Engineering - Techniques to improve reliability
• LangChain
• MiniChain - A tiny library for coding with large language models
• MiniChain - Documents and Embeddings
• [YouTube] State of GPT (42 mins, BloombergGPT 10 mins in))
• [YouTube] Build GPT from scratch, in code, spelled out (2 hours)
• "AI canon" - a list of influential resources
• FLAN-T5 vs. FLAN-UL2
• Adapter Methods
• What is low-rank adaptation (LoRA)?
• BloombergGPT: A Large Language Model for Finance
• What's going on with the Open LLM Leaderboard? (MMLU)
• Mosaic AI - LLM Evaluation Scores
• SEC Insights - a real worl full-stack application using LlamaIndex (Demo)
• The Geometry of Truth: Dataexplorer, Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Fuyu-8B: A Multimodal Architecture for AI Agents
• Fine-Grained RLHF - Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Encoder, Decoder, Autoencoder, Decoding, Sampling, Attention

• Difference between Transformer Encoder, Decoder and Encoder-Decoder architectures
• What is the difference between encoders and auto-encoders?
• How to sample from language models - Temprature and Top k Sampling
• Seq2Seq with Attention and Beam Search
• How does Attention work in Encoder-Decoder Recurrent Neural Networks
• Beam Search Decoding in CTC-trained Neural Networks
• Introduction to Autoencoders
• Introduction to Variational Autoencoders
• Learning Disentangled Representations with Variational Autoencoders
• Unsupervised Machine Translation (Github)
• Difference between PCA and Autoencoder
• Autoencoder vs PCA: when to use?
• Decoding Strategies that You Need to Know for Response Generation - Distinguish between Beam Search, Random Sampling, Top-K, and Nucleus
• What's the difference between a Variational Autoencoder (VAE) and an Autoencoder?
• Introducing BART (vs. BERT & GPT2)

Loss Function

• What is the difference Cross-entropy and KL divergence?
• (Proof) Cross Entropy, KL Divergence and Maximum Likelihood Estimation
• Difference between Cross-Entropy Loss and (Negative) Log Likelihood Loss (PyTorch)
• PyTorch Loss-Input Confusion (Cheatsheet) - Cross_Entropy = Log_Softmax + NLL_Loss
• Advantage of log_softmax over softmax (PyTorch)
• Label Smoothing as Regularization
• Sequence Modeling with Connectionist Temporal Classification (CTC) Loss
• KL Divergence: Forward vs. Reverse?
• MultipleNegativesRankingLoss

Information Retrieval, Ranking, Search

• Understanding Ranking Loss, Contrastive Loss, Margin Loss, Triplet Loss, Hinge Loss and all those confusing names
• Siamese and triplet learning with online pair/triplet mining
• Learning to Rank: A Complete Guide to Ranking using Machine Learning

Machine Learning, Linear Algebra

• Linear Algebra exaplined in 4 pages
• R-squared vs. t-test confusion
• COMS 4771 Machine Learning
• ROC vs precision-recall curvers
• Bias-Variance Tradeoff (Noise)
• Bias-Variance Decomposition - mlxtend
• Introduction to Optimization with Genetic Algorithm
• Intro to Noise Contrastic Estimation
• CS189 - Boosting
• A Gentle Intro to Gradient Boosting Algorithm
• Difference Between Test and Validation Datasets
• What is the intuition behind beta distribution
• Beta Distribution — Intuition, Examples, and Derivation
• Is k-means a generative model and how could it be used to generate new data then?
• Optimizing Classifier Performance (AUC-ROC) via an Approximation to the Wilcoxon-Mann-Whitney (WMW) Statistic
• SMAC: AutoML for fast Hyperparameters tuning in Python
• Uncertainty Calibration
• FP64, FP32, FP16, BFLOAT16, TF32, and other members of the ZOO

Machine Learning Operations - MLOps

• Operationalizing Machine Learning: An Interview Study
• TorchServe - Serving Huggingface Transformers using TorchServe - Speed up inference with Better Transformer
• TorchServe - Workflow Examples - deploying multiple models in a pipeline with TorchServe
• NVIDIA Triton vs TorchServe for SageMaker Inference
• Paradigms of Parallelism (Data, Tensor, Pipeline, Optimizer-Level)
• Fast and Scalable AI Model Deployment with NVIDIA Triton Inference Server (Dynamic Batching)
• A100 Vs V100 Deep Learning Benchmarks
• Multi-Model Inference with Triton Inference Server

Deep Learning

• Full Implementation of Gradient Descent - no holding back
• Weight Initializers (Random, Normal, Xavier, He)
• An Overview of Gradient Descent Optimization Algorithms
• Neural Architecture Search
• A Step by Step Backpropagation Example
• Incorporating Nesterov Momentum into Adam
• Batch Normalization: The Greatest Breakthrough in Deep Learning
• How does Batch Normalization battle the fact that gradients might explode when training a Neural Network?
• Deep Learning with PyTorch - Open Source Course by Yann LeCun and NYU
• LIME - Local Interpretable Model-Agnostic Explanations
• The Gumbel-Softmax Distribution
• Merging Models with Fisher-Weighted Averaging
• A Conceptual Explanation of Bayesian Hyperparameter Optimization for Machine Learning
• Understanding Gradient Clipping (and How It Can Fix Exploding Gradients Problem)
• upsampling vs class weights in mini-batch SGD for class imbalance (and why you should favor upsampling)

Reinforcement Learning

• Proximal Policy Optimization

Miscellaneous - NLP

• Topic Modeling
• Chinese NLP Corpus
• NLP & ML Conference Deadlines
• The relationship between Perplexity and Entropy in NLP
• Python for NLP Topic Modeling
• Hierarchical Latent Dirichlet Allocation (hLDA) for Topic Modeling - Github
• Topic Modeling Gensim Python
• Free NLP Datasets
• Generating N-grams from Sentences in Python
• (Github) Ecco - python library for explaining NLP models using interactive visualizations
• Quootstrap: Scalable Unsupervised Extraction of QuotationSpeaker Pairs from Large News Corpora via Bootstrapping
• Quotebank: A Corpus of Quotations from a Decade of News (& Quobert)
• NVIDIA DCGM-Exporter
• What Is /dev/shm And Its Practical Usage
• Torch Single- and Multi-process Data Loading
• Guidelines for assigning num_workers to DataLoader

Data Sampling, Augmentation, Active Learning, Active Testing

• EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
• Active Learning Literature Survey
• Active Learning for BERT: An Empirical Study
• Active Learning by Acquiring Contrastive Examples
• On the Importance of Effectively Adapting Pretrained Language models for Active Learning
• Active Testing: Sample-Efficient Model Evaluation

Open Source Implementations - NLP

• Chinese Word Vectors
• OpenNMT-py: Open-Source Neural Machine Translation in PyTorch
• Unsupervised Machine Translation
• Facebook Research: fastText
• PyTorch Template Project
• Cleverhans Blog - Library for Benchmarking the Vulnerability of Machine Learning models to Adversarial Examples
• Interpret-Text - Github
• Predict stock market prices using RNN
• Daily Scikit-Learn Tips

Logistics: Conda, Python, Tensorboard, Jupyter

• Conda Cheatsheet
• Tensorboard Logging
• How to accumulate gradients for large batch sizes in Keras
• Running Jupyter Notebooks on GPU on Google Cloud
• Python Naming Convention Guides
• Python args and kwargs
• Primer on Python Decorators
• Python Iterables
• Python Iterators (W3 Schools)
• Apache Solr Tutorial
• fpdf2 - PDF for Python
• Jekyll: Transform your plain text into static websites and blogs
• Jekyll-now (Github)

Papers

• NLP Paper Summaries (Github)
• Dense Passage Retrieval for Open-Domain Question Answering
• Unsupervised Machine Translation Using Monolingual Corpora Only (Github)
• NSTM: Real-Time Query-Driven News Overview Composition at Bloomberg
• BERT-DST: Scalable E2E Dialogue State Tracking with BERT
• BERT-Related papers
• Contrastive Bidirectional Transformer for Temporal Representation Learning
• StateNet: Towards Universal Dialogue State Tracking
• Dialog State Tracking: A Neural Reading Comprehension Approach
• Self-training with Noisy Student improves ImageNet classification
• Adversarial Examples In The Physical World
• Improved Regularization of Convolutional Neural Networks with Cutout
• Dialgoue State Tracking SOTA
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (Github)
• Applied-ML (Github): Curated papers, articles, and blogs on data science & machine learning in production.
• Paper Digest: NeurIPS 2021 Highlights
• Scaling Instruction-Finetuned Language Models
• How many data points is a prompt worth (Outstanding Paper Award)
• A Few More Examples May Be Worth Billions of Parameters
• Efficient Natural Language Response Suggestion for Smart Reply (Ranking)
• Enabling Large Language Models to Generate Text with Citations
• From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting

CNN, CV

• CS231n Convolutional Neural Networks for Visual Recognition
• CNN Confusion Matrix With PyTorch - Neural Network Programming
• Cifar10 Classification Jupyter Notebook
• PyTorch: Resnet
• Keras: Resnet on Cifar10
• Self-Supervised Learning for Computer Vision (Github)

Speech and ASR

• Time-Delay Neural Network
• Speech2Vec Pretrained Vectors (Github)
• Understanding Speech Recognition
• How to start with Kaldi and Speech Recognition
• Mel Frequency Cepstral Coefficient (MFCC) Tutorial
• Kaldi Data Preparation
• Kaldi for Dummies Tutorial
• MFCC in Matlab
• MFCC in Python (Github) - Documentation
• Awesome Kaldi (Github)
• Great Kaldi Tutorial