LLM101n: Let's build a Storyteller

What I cannot create, I do not understand. -Richard Feynman

In this course we will build a Storyteller AI Large Language Model (LLM). Hand in hand, you'll be able create, refine and illustrate little stories with the AI. We are going to build everything end-to-end from basics to a functioning web app similar to ChatGPT, from scratch in Python, C and CUDA, and with minimal computer science prerequisits. By the end you should have a relatively deep understanding of AI, LLMs, and deep learning more generally.

Syllabus

Chapter 01 Bigram Language Model (language modeling)

Chapter 02 Micrograd (machine learning, backpropagation)

Chapter 03 N-gram model (multi-layer perceptron, matmul, gelu)

Chapter 04 Attention (attention, softmax, positional encoder)

Chapter 05 Transformer (transformer, residual, layernorm, GPT-2)

Chapter 06 Tokenization (minBPE, byte pair encoding)

Chapter 07 Optimization (initialization, optimization, AdamW)

Chapter 08 Need for Speed I: Device (device, CPU, GPU, ...)

Chapter 09 Need for Speed II: Precision (mixed precision training, fp16, bf16, fp8, ...)

Chapter 10 Need for Speed III: Distributed (distributed optimization, DDP, ZeRO)

Chapter 11 Datasets (datasets, data loading, synthetic data generation)

Chapter 12 Inference I: kv-cache (kv-cache)

Chapter 13 Inference II: Quantization (quantization)

Chapter 14 Finetuning I: SFT (supervised finetuning SFT, PEFT, LoRA, chat)

Chapter 15 Finetuning II: RL (reinforcement learning, RLHF, PPO, DPO)

Chapter 16 Deployment (API, web app)

Chapter 17 Multimodal (VQVAE, diffusion transformer)

https://github.com/karpathy/LLM101n

Harnessing Collective AI Wisdom: A Dive into Microsoft's AutoGen Framework

In a world where Artificial Intelligence (AI) is swiftly evolving, the race for creating more intelligent and autonomous systems is intensifying. Microsoft, a formidable player in this domain, has recently unveiled its AutoGen Framework, a pioneering platform that orchestrates interaction among multiple AI agents, aiming to streamline task execution.

AutoGen, an open-source Python library, is Microsoft’s stride into the realm of large language model (LLM) application frameworks. This framework is engineered to simplify the orchestration, optimization, and automation of workflows centered around LLMs like GPT-4. The spotlight is on the creation of "agents", which are essentially programming modules empowered by LLMs, and are designed to communicate with each other through natural language messages to accomplish diverse tasks.

What makes AutoGen an enticing proposition is its modular architecture. Developers have the liberty to create an ecosystem of agents, each specializing in different tasks yet capable of cooperating seamlessly. Every agent is perceived as an individual ChatGPT session with its unique instruction set. For instance, one agent might take on the role of a programming assistant, generating Python code based on user requests, while another could act as a code reviewer, examining the code snippets and troubleshooting them. The response from the first agent can be seamlessly channeled as input to the second, creating a coherent workflow.

The framework also extends a layer of customization and augmentation through prompt engineering techniques and external tools. These augmentations enable agents to fetch information or execute code, broadening the spectrum of tasks they can handle.

A striking feature of AutoGen is the integration of “human proxy agents”, allowing users to dive into the conversation between AI agents. This feature morphs the human user into a team leader overseeing a group of AI agents, facilitating a higher degree of oversight and control especially in scenarios requiring sensitive decision-making.

Multi-agent collaborations under AutoGen can lead to substantial efficiency gains. As per Microsoft's claims, AutoGen has the potential to accelerate coding processes by up to four times, showcasing a promising avenue for reducing developmental timelines.

Furthermore, AutoGen supports more complex scenarios through hierarchical arrangements of LLM agents, bringing a new dimension to multi-agent interactions. For instance, a group chat manager agent could mediate discussions between multiple human users and LLM agents, ensuring effective communication according to predefined rules.

As the arena of LLM application frameworks burgeons, AutoGen is squaring up against many contenders. However, what sets it apart is its emphasis on creating a collaborative environment where multiple AI agents, with a sprinkle of human intervention, can collectively drive task completion to new heights.

Despite the challenges such as hallucinations and unpredictable behaviors from LLM agents, the horizon looks promising. The evolution of LLM agents is poised to play a crucial role in the future of application development and operational systems. With AutoGen, Microsoft is not only embracing the competitive spirit of this fast-evolving field but is also laying down a robust foundation for the futuristic vision of harmonized AI-human interactions.

Introducing Falcon 180B: The Next-Gen Open-Source Language Model Surpassing Previous Benchmarks

 

The Hugging Face AI community announced the release of Falcon 180B, an open-source large language model (LLM) with 180 billion parameters trained on 3.5 trillion tokens. This latest LLM surpasses prior models, including the previously top-ranked LLaMA 2, in scale and performance. Falcon 180B, trained using Amazon SageMaker on 4,096 GPUs, competes closely with commercial models like Google's PaLM-2. The release signifies rapid advancement in LLMs, with Falcon 180B benefiting from techniques such as LoRAs and Nvidia’s Perfusion. It is expected to see further improvement as the community fine-tunes it.

Hardware requirements

Falcon 180B Training Full fine-tuning 5120GB 8x 8x A100 80GB

Falcon 180B Training LoRA with ZeRO-3 1280GB 2x 8x A100 80GB

Falcon 180B Training QLoRA 160GB 2x A100 80GB

Falcon 180B Inference BF16/FP16 640GB 8x A100 80GB

Falcon 180B Inference GPTQ/int4 320GB 8x A100 40GB