LLM101n: Let's build a Storyteller

What I cannot create, I do not understand. -Richard Feynman

In this course we will build a Storyteller AI Large Language Model (LLM). Hand in hand, you'll be able create, refine and illustrate little stories with the AI. We are going to build everything end-to-end from basics to a functioning web app similar to ChatGPT, from scratch in Python, C and CUDA, and with minimal computer science prerequisits. By the end you should have a relatively deep understanding of AI, LLMs, and deep learning more generally.

Syllabus

Chapter 01 Bigram Language Model (language modeling)

Chapter 02 Micrograd (machine learning, backpropagation)

Chapter 03 N-gram model (multi-layer perceptron, matmul, gelu)

Chapter 04 Attention (attention, softmax, positional encoder)

Chapter 05 Transformer (transformer, residual, layernorm, GPT-2)

Chapter 06 Tokenization (minBPE, byte pair encoding)

Chapter 07 Optimization (initialization, optimization, AdamW)

Chapter 08 Need for Speed I: Device (device, CPU, GPU, ...)

Chapter 09 Need for Speed II: Precision (mixed precision training, fp16, bf16, fp8, ...)

Chapter 10 Need for Speed III: Distributed (distributed optimization, DDP, ZeRO)

Chapter 11 Datasets (datasets, data loading, synthetic data generation)

Chapter 12 Inference I: kv-cache (kv-cache)

Chapter 13 Inference II: Quantization (quantization)

Chapter 14 Finetuning I: SFT (supervised finetuning SFT, PEFT, LoRA, chat)

Chapter 15 Finetuning II: RL (reinforcement learning, RLHF, PPO, DPO)

Chapter 16 Deployment (API, web app)

Chapter 17 Multimodal (VQVAE, diffusion transformer)

https://github.com/karpathy/LLM101n

Diving into Deep Learning with PyTorch: A Beginner’s Guide

In this course, you learn all the fundamentals to get started with PyTorch and Deep Learning.

Deep Learning, with its potential to transform industries and the way we approach data, has taken the tech world by storm. If you've been curious about this revolutionary field and have been seeking a comprehensive introduction, then you're in the right place.

Why PyTorch?

PyTorch, developed by Facebook's AI Research lab, has rapidly gained popularity among researchers and developers alike. It is recognized for its dynamic computation graph, which means the graph builds on-the-fly as operations are created, making it highly flexible and intuitive. This is particularly useful for those just beginning their deep learning journey, as it allows for easy debugging and a more natural understanding of the flow of operations.

What Will You Learn?

In this course, you'll be taken on a deep dive into the fascinating world of deep learning. Some highlights include:

Understanding the Basics: Grasp the fundamental concepts of neural networks, how they're structured, and how they function.

PyTorch Essentials: Get hands-on experience with PyTorch's tensors, autograd, and other essential components.

Building Neural Networks: By the end of this course, you'll be constructing your very own neural networks, and training them to recognize patterns, images, and more.

Practical Applications: Witness the real-world utility of deep learning as you work on exciting projects and real-life datasets.

Beginner-Friendly Approach

This course is crafted keeping beginners in mind. Whether you're entirely new to programming, or an experienced developer wanting to switch to deep learning, you'll find the content accessible and engaging. The blend of theory and hands-on exercises ensures that you not only learn but also apply your newfound knowledge practically.

Conclusion

With the increasing demand for professionals skilled in deep learning and AI, there's no better time than now to dive in. By familiarizing yourself with PyTorch and deep learning fundamentals through this course, you're equipping yourself with the tools and knowledge necessary to be at the forefront of technological innovation.

Get started today, and embark on a journey of endless learning and opportunities!

A step-by-step roadmap for getting started with machine learning

A step-by-step roadmap for getting started with machine learning:


Foundational Knowledge

Mathematics

• Linear Algebra (vectors, matrices, eigenvalues, etc.)
• Calculus (derivatives and integrals, partial derivatives for multivariate functions)
• Probability and Statistics (Bayes' theorem, mean, median, variance, standard deviation, etc.)

Programming

• Python (most popular for ML): Get comfortable with libraries like NumPy and pandas.

Databases

• Understand relational databases (SQL) and NoSQL databases.

Machine Learning Basics

Supervised Learning

• Linear Regression
• Logistic Regression
• Decision Trees and Random Forests
• Support Vector Machines (SVM)
• k-Nearest Neighbors (kNN)

Unsupervised Learning

• Clustering (K-means, hierarchical clustering)
• Dimensionality Reduction (PCA, t-SNE)

- Regularization: L1 and L2 regularization
- Evaluation Metrics: Accuracy, precision, recall, F1 score, ROC, AUC, etc.
- Tools and Libraries: Scikit-learn




Intermediate Machine Learning

- **Ensemble Methods:** Bagging, Boosting (e.g., AdaBoost, Gradient Boosting, XGBoost)

- **Neural Networks:** Basics of feedforward neural networks

- **Deep Learning:**

- Convolutional Neural Networks (CNNs)

- Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), GRU

- **Tools and Libraries:** TensorFlow, Keras, PyTorch

- **Validation:** Understand overfitting, underfitting, and how to split data (train/test/validation splits, k-fold cross-validation)




Advanced Machine Learning & Specializations

- **Natural Language Processing (NLP):** Tokenization, embeddings, transformers, BERT, etc.

- **Computer Vision:** Advanced CNN architectures, object detection, segmentation.

- **Reinforcement Learning:** Q-learning, deep Q networks, policy gradient methods.

- **Transfer Learning:** Utilizing pre-trained models.

- **Generative Adversarial Networks (GANs):** Basics and applications.

- **Explainable AI:** Techniques to understand and interpret machine learning models.




Real-world Application & Production

- **Model Deployment:** Tools like TensorFlow Serving, Flask, FastAPI.

- **Cloud Platforms:** AWS (SageMaker), Google Cloud ML, Azure ML.

- **Model Monitoring & Maintenance:** Ensure your model stays accurate over time.

- **Optimization:** Real-time processing, reducing latency, serving models efficiently.

- **MLOps:** Continuous integration and deployment (CI/CD) for ML tools like MLflow.




Stay Updated and Continuous Learning

- **Research & Papers:** Websites like arXiv, conferences like NeurIPS, ICML, etc.

- **Online Courses & Certifications:** Coursera, edX, Udacity, and others offer advanced courses.

- **Work on Projects:** Build your own projects and participate in hackathons and Kaggle competitions.




Soft Skills & Miscellaneous

Ethics in AI: Understand the ethical implications of ML models and their decisions.
Communication: Being able to explain complex ML concepts to non-experts is crucial.
Networking: Engage with the community and attend workshops, webinars, and meetups.


Remember that the field of machine learning is vast, and it's okay not to know everything. Instead, focus on building a strong foundational understanding and then dive deeper into areas that interest you the most. The best way to learn is by doing, so working on projects and hands-on experimentation is crucial to understanding the nuances and intricacies of ML algorithms and tools.