LLM compression

LLM pruning

Large language models (LLMs) consist of many components, but not all are essential for output. Such non-critical components can be pruned to maintain performance while reducing model size.

Unstructured Pruning:

• Involves removing parameters without considering the model's structure.
• Sets insignificant parameters to zero, creating a sparse model.
• It is easy to implement but hard to optimize due to its random weight distribution.
• Requires additional processing to compress and might need retraining.
• Notable advancements include SparseGPT (eliminates retraining) and LoRAPrune (combines low-rank adaptation with pruning).

Structured Pruning:

• Removes whole sections, like neurons or layers.
• Simplifies model compression and boosts hardware efficiency.
• Requires a deep understanding of the model and might significantly impact accuracy.
• LLM-Pruner is a promising technique that uses gradient information to prune without relying heavily on original training data.
• Both methods aim to optimize the balance between model size and performance.

LLM Knowledge Distillation

Knowledge distillation involves training a smaller "student" model to emulate a more complex "teacher" model, effectively creating a compact yet proficient model. In the context of LLMs, this technique has two primary categories:

Standard Knowledge Distillation:

• Transfers the broad knowledge of the teacher to the student.
• Can use prompts and responses from models like ChatGPT to train smaller LLMs, though there are constraints related to data from commercial models.
• MiniLLM, developed by Tsinghua University and Microsoft Research, improves the process by using specialized objective and optimization functions, addressing the challenge of accurately capturing data distributions.

Emergent Ability Distillation:

• Targets the transfer of a specific capability from the teacher model.
• Examples include extracting math or reasoning skills from GPT-4 to a smaller model, such as Vicuna.
• Focusing on a narrower task set makes measuring EA distillation easier, but it's essential to recognize the limitations in transferring emergent behaviors to smaller LLMs

LLM Quantization

Large Language Models (LLMs) like GPT-3 store parameters as floating-point values, with models like GPT-3 using hundreds of gigabytes of memory. To reduce this size, a technique called quantization is used, converting parameters to smaller integers.

Benefits of Quantization:

• Allows LLMs to run on everyday devices.
• Examples of quantized LLMs include GPT4All and Llama.cpp.

Quantization Approaches:

• Quantization-Aware Training (QAT): Integrates quantization during training, allowing models to learn low-precision representations. The downside is it requires training from the beginning.
• Quantization-Aware Fine-Tuning (QAFT): Adapts a pre-trained model for lower-precision weights. Techniques like QLoRA and PEQA are used in this approach.
• Post-Training Quantization (PTQ): Reduces precision after training without changing the architecture. It's simple and efficient but might affect accuracy.

For an in-depth exploration of LLM compression, the paper "A Survey on Model Compression for Large Language Models" is recommended.