Blog Index

A collection of articles on algorithms and computational techniques in Artificial Intelligence (AI), Parallel Computing, and Computational Genomics.

• 1) How to train your first LLM?

  This blog introduces the concept of training large language models (LLMs) from scratch, covering the fundamental steps and practices.

• 2) A Beginner's Guide to Tensor Parallelism in Autoregressive LLM Inference

  This blog introduces the concept of tensor parallelism in the context of autoregressive large language model (LLM) inference. It covers the basics of distributing tensor computations across multiple GPUs to accelerate model inference.

• 3) A Beginner's Guide to Tensor Parallelism + Expert Parallelism in MoE LLM Inference

  This blog explores advanced parallelism techniques in large language models, including tensor parallelism, expert parallelism, and mixture of experts (MoE) strategies.

• 4) A Beginner's Guide to Understanding Self-Attention in Autoregressive LLMs

  This blog explains self-attention in transformers from a systems perspective, focusing on runtime and memory complexity.

• 5) A Beginner's Guide to Understanding KV Caches in Autoregressive LLMs

  This blog explains KV caches in transformers from a systems perspective, focusing on runtime and memory complexity, prefix caching, and quantization techniques.

• 6) Optimization Techniques for Matrix Multiplication on CPUs

  A deep dive into various optimization techniques for matrix multiplication, including tiling, loop unrolling, parallelization, and AVX-512 SIMD on modern CPUs.

• 7) Optimization Techniques for Matrix Multiplication on GPUs

  A deep dive into matrix multiplication on A100 GPUs, exploring naive, shared memory, and WMMA tensor core approaches.

• 8) Roofline Analysis of Matrix Multiplication

  We explore the roofline analysis of matrix multiplication. The roofline plot provides insights into whether a program is memory-bound (limited by data transfer rate) or compute-bound (limited by available computational throughput).

• 9) Parallel Needleman-Wunsch Algorithm

  We explore the anti-diagonal parallelization technique for the Needleman-Wunsch algorithm.

• 10) Understanding GPU Programming on NVIDIA A100 GPUs with Parallel Prefix Sums

  This blog explores the fundamentals of GPU programming using parallel prefix sums as a case study. It demonstrates various CUDA implementations, including naive, shared memory, and dynamic parallelism, highlighting the performance benefits of efficient memory management and GPU architecture on NVIDIA A100 GPUs.

• 11) Heuristic-Based Parallel Needleman-Wunsch Algorithm

  This blog introduces a heuristic-based parallelization strategy for the Needleman-Wunsch (NW) algorithm, a cornerstone of bioinformatics used for global sequence alignment. Traditionally, the NW algorithm faces challenges in parallelization due to its sequential dependencies in dynamic programming computations. The proposed heuristic employs iterative dynamic programming with chunk-based computations, overlapping boundaries, and iterative updates, achieving both correctness and scalability.