Parallel and Distributed Algorithms - Part 1

Introduction to Parallel and Distributed Algorithms. Part 1: Theoretical article.

Welcome back, brave coder 👨‍💻! I’m really happy and excited to be back today to writing, after a two weeks pause! These two weeks felt like a year for me, haha. But anyway, we’re back and today, we will embark on an epic adventure through the magical land of Parallel and Distributed Algorithms. If these words sound intimidating, don't worry! I was there too, we're going to turn this learning process into a fun, easy, and exciting experience.

Ready? Let’s go Ala Barakati Lah 🙏

---

What the Hell is Parallel and Distributed Algorithms?

Imagine you're a knight 🤺, and your kingdom 🏰 is under attack by an army of monsters 🧟‍♂️. Would you fight them alone, and beat them one by one? If you say yes to this, believe me, that’s like running a single-threaded program. Slow and exhausting 🥴️!

But what if you had a team of knights 🤺🤺🤺🤺, each one take on a monster at the same time? That’s a better strategy, right? That’s parallel algorithms! And if your knights are spread across different castles 🏰🏰🏰, each defending their local village, for example, that's distributed algorithms.

See? That’s easy, you know understand what’s parallel and distributed algorithms mean. Now let’s dive into each one of them, and try to explain it in computer science words!

---

Chapter 1: Parallel Algorithms

In parallel computing, the goal is to divide and conquer. Here’s the cheat sheet:

1. The Hero is the CPU or GPU
   Think of your computer's processor as the hero of the story. It has great power but also limits. Parallel algorithms help the hero (CPU or GPU) use their full potential/power.

2. The Threads or Processes
   Each thread is like a warrior in your war (task to do). They're good at handling a specific task. When they work together, things get done faster!

3. The Tasks
   Break your big quest (problem) into smaller tasks. For example, if you're baking 100 cakes, don’t do it all by yourself! Assign one cook to mix ingredients, another to bake, and another to decorate!

Example Challenge:

Imagine sorting a treasure chest full of gold coins. A parallel algorithm might:

• Have one thread sort small piles of coins.

• Have another thread combine those sorted piles.

• Done!

---

Chapter 2: Distributed Algorithms

Now, let’s say the monster’s army is so big that you need helps from neighboring kingdoms. It’s here when distributed algorithms come into play:

1. The Network Map
   Your computers (nodes) are like distant castles connected by roads (networks). These nodes work together to solve problems.

2. Communication
   Nodes exchange information via messages. It’s like sending a pigeon with a message on her leg saying, “I’ve defeated my monster, how about you?”

3. Fault Tolerance
   Sometimes, roads get blocked, or pigeons go missing (get hunted lol). Distributed algorithms plan for this, ensuring the mission continues even if a few nodes fail.

Example Challenge:

Let’s say you want to count the monsters that attacked all the kingdoms. Each node counts its local monsters and sends the number to a central castle. And the central castle adds up the numbers. Easy-peasy (as long as the pigeons don’t get lost 😅).

---

Chapter 3: Theoretical Concepts

Okay, time to learn about the concepts of both parallel and distributed algorithms. These are the big ideas you’ll need to know, don’t bother yourself with anything else:

1. Speedup and Efficiency

   • Speedup: How much faster can your team solve the problem compared to doing it alone?

   • Efficiency: Are all your team members are busy? Or is one of them relaxing?

2. Load Balancing
   You don’t want one knight fighting 100 monsters, while another knight is chilling doing nothing! Distribute the work evenly!

3. Latency vs. Throughput

   • Latency: How fast does a single task get done?

   • Throughput: How many tasks can you complete at once?

4. Synchronization
   If one team member finishes their task too soon, they might need to wait for others to catch up. That’s synchronization (maybe I will write a dedicated article about this concept soon!). Keep the team in sync!

---

Why Should You Care?

Parallel and distributed algorithms are the backbone of modern technology:

• Streaming services like Netflix? Parallel algorithms process your favorite shows.

• Online games? Distributed algorithms keep everyone in sync, whether they're in “Safi, Morocco” or in “Cartagena (9artahenna haha), Colombia”.

• NASA, SPACE X, TESLA, and all high level companies use these concepts! So, you must to understand them too!

---

Wrapping it up!

Congratulations! You’ve survived Part 1 of the Parallel and Distributed Algorithms Quest. Here’s what you’ve learned:

• Parallel algorithms are like a teamwork in the same place.

• Distributed algorithms are like a long-distance alliance between kingdoms.

• Key ideas like speedup, load balancing, and synchronization help you slay big problems.

Next time, we’ll dive into the best part which is coding! Until then, happy coding guys, see you later!

Stay tuned for the next one!

🔔 Don't forget to subscribe to my blog and YouTube channel to stay updated with all the latest content, follow me on LinkedIn too, if you like to do so!