ibragimov

Today’s transformer models contain billions of parameters, which makes it impossible to do training and inference on a single accelerator (e.g., GPU, TPU). This motivates to distribute the compute and memory across accelerators. The picture illustrates one parallelism technique called Tensor Parallelism (TP). A transformer consists of multiple identical layers, and each layer has self-attention, multi-layer perceptron (MLP), normalization, and residual add. Self-attention and MLP blocks contain most of the weights, making them ideal candidates for parallelization. The idea is to partition the big matrix of parameters in MLP and Self-Attention across accelerators. Each accelerator performs multiplications, and then the results are combined. There are a lot of interesting systems questions. What if the accelerators are heterogeneous, meaning they have different compute and memory resources? Since combining results involves network, how to ensure the computation is not stalled by the weakest compute node?

I have joined CMU’s Computer Architecture and Operating Systems (CAOS) research group to work on developing fault-tolerant OS primitives for machine learning inference. I would love to build my career on operating systems research, and it is a very long road…

I have joined CMU’s Computer Architecture and Operating Systems (CAOS) research group to work on developing fault-tolerant OS primitives for machine learning inference. I would love to build my career on operating systems research, and it is a very long road ahead…

300+ hours, 400+ commits, and 6283 lines of code. This is the amount of effort it took us to build a Unix-like, preemptive kernel that supports multithreading and has 20+ system calls within 5 weeks. One thing that CMU taught me is that there is no limit…

late night shots. leaving this place in 2 weeks 🫡

I am planning to go to main campus next semester. To fulfill my systems concentration, I am thinking of taking Operating Systems course (15-410). This course is infamous for being the hardest systems course at CMU. It requires around 60 hours per week, and…

My brother got into Stanford Summer Math Camp with full financial aid!

I’m sure many of you have heard about the tragedy at Brown University. One of the two students who was killed was an Uzbek freshman, MukhammadAziz Umurzokov. I’m forwarding a fundraiser to support his family with expenses. Any support or sharing is greatly appreciated 🙏 https://gofund.me/360c53a86

I am planning to go to main campus next semester. To fulfill my systems concentration, I am thinking of taking Operating Systems course (15-410). This course is infamous for being the hardest systems course at CMU. It requires around 60 hours per week, and professor at the beginning of the semester even says there is a high chance of failing the course. The homework projects are fun, though. Essentially, you build your own kernel from the bottom up. The first project is building your own stack allocator. Then, you build drivers for I/O, hypervisor, and other cool things I have not had a chance to read yet. If I decide to take this course, I will be sacrificing my semester exchange experience for a single CS course. The only thing I learned in this semester is being ambitious hurts your GPA 😂. I may end up taking the course, but I don’t know if I will manage it lol.

https://www.facebook.com/codingcompetitions/hacker-cup/2025

I have also recently declared Math minor 🫡

dislike button is back! feel free to use it! 😎

Every time I feel like writing a post on this channel, I change my mind immediately because mostly it’s just random thoughts that are irrelevant. I apologize for the inactivity here. I turned off the comments section a while ago to avoid spam posts. I don’t know when I am going to get back to active posting and fix the problem with spam, but, hopefully, it will be soon… ☹️

https://www.facebook.com/codingcompetitions/hacker-cup/2025

From news, I officially declared my concentration in Computer Systems.

Cloud Resource Management. To put it simply, resource management refers to the intelligent decision approach that aims to satisfy the given user demand based on the available (cloud) resources (e.g. CPU/GPU/memory). The quality at which cloud managers satisfy this demand is known as Quality of Service (QoS). QoS usually refers to the technical aspects like network bandwidth and latency. One of the recent trends in this field is so-called proactive resource autoscaling. The idea is to proactively predict the future user demand and automatically scale up or down the resources accordingly. Recently, a lot of research in proactive autoscaling has focused on developing machine learning based solutions, which turned out to be quite groundbreaking. The idea behind these models is quite simple, and they do not require extensive training. If you are interested, I suggest reading this paper. There are a lot of things to cover in the cloud computing field, and I will be slowly doing so, if my research heads there.