Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Distributed Task Queue :: SysDesignMeetup :: 2022-July-02

SysDesign Meetup
2.14K subscribers
3.8K views 3 years ago #NoWar
Distributed Task Queue. We talk about different ways to postulate and reason about the problem, effectively from how would a junior engineer approach the Task Queue problem to how would a Cloud Architect view it. Thank you for 1K YouTube subscribers! #NoWar is obligatory now, as this is recorded and released in July 2022. I am hoping for the madness that is unfolding in front of our eyes to stop as soon as possible. Slides: https://tinyurl.com/sdm-taskqueue-slides 0:00 Intro 0:57 1K subscribers and meetup goals 2:31 Scope & objective 8:08 Problem statement 8:55 Scheduling tasks, retry, cron 11:42 Task dependencies & priorities | Decouple tasks creation from execution 16:32 HA / durability & exactly once | Task queue as the source of truth 25:14 The two APIs 27:26 Tasks creation API 31:17 Worker API 39:52 Keep the workers busy / avoid stalled workers 43:38 Can "shared nothing" work? 46:07 Why shared nothing might work 48:23 Why shared nothing …...more
...more

Intro

0:00

1K subscribers and meetup goals

0:57

Scope & objective

2:31

Problem statement

8:08

Scheduling tasks, retry, cron

8:55

Task dependencies & priorities | Decouple tasks creation from execution

11:42

HA / durability & exactly once | Task queue as the source of truth

16:32

The two APIs

25:14

Tasks creation API

27:26

Worker API

31:17

Keep the workers busy / avoid stalled workers

39:52

Can "shared nothing" work?

43:38

Why shared nothing might work

46:07

Why shared nothing might *not* work

48:23

MongoDB

55:36

RabbitMQ

57:43

Kafka

59:48

Q: Maybe use a dedicated topic for deleted/rescheduled tasks?

1:04:03

PostgreSQL

1:06:41

Idempotency tokens

1:12:09

Fencing tokens

1:14:20

In conclusion / Q&A

1:20:59

Q: Tasks dependencies, and why this part is eventually, not strongly, consistent.

1:22:12

Q: Is Kafka's FIFO useful? If not, what's the value of Kafka?

1:26:56

Q: Why not build an in-house solution?

1:29:17

Q: Why build an in-house solution?

1:31:55

Q: Any real-life examples where strictly-exactly-once Task Queue is needed?

1:32:52

Q: Maybe sending money like Venmo et. al.?

1:35:10

Q: What about bank transactions?

1:35:50

Q: What exactly does distributed consensus help with here?

1:37:10

Closing notes on Google Spanner and the state of the art wrt distributed consensus.

1:44:47
Distributed Task Queue :: SysDesignMeetup :: 2022-July-02
49Likes
3,864Views
2022Jul 3
Distributed Task Queue. We talk about different ways to postulate and reason about the problem, effectively from how would a junior engineer approach the Task Queue problem to how would a Cloud Architect view it. Thank you for 1K YouTube subscribers! #NoWar is obligatory now, as this is recorded and released in July 2022. I am hoping for the madness that is unfolding in front of our eyes to stop as soon as possible. Slides: https://tinyurl.com/sdm-taskqueue-slides 0:00 Intro 0:57 1K subscribers and meetup goals 2:31 Scope & objective 8:08 Problem statement 8:55 Scheduling tasks, retry, cron 11:42 Task dependencies & priorities | Decouple tasks creation from execution 16:32 HA / durability & exactly once | Task queue as the source of truth 25:14 The two APIs 27:26 Tasks creation API 31:17 Worker API 39:52 Keep the workers busy / avoid stalled workers 43:38 Can "shared nothing" work? 46:07 Why shared nothing might work 48:23 Why shared nothing might not work 55:36 MongoDB 57:43 RabbitMQ 59:48 Kafka 1:04:03 Q: Maybe use a dedicated topic for deleted/rescheduled tasks? 1:06:41 PostgreSQL 1:12:09 Idempotency tokens 1:14:20 Fencing tokens 1:20:59 In conclusion / Q&A 1:22:12 Q: Tasks dependencies, and why this part is eventually, not strongly, consistent. 1:26:56 Q: Is Kafka's FIFO useful? If not, what's the value of Kafka? 1:29:17 Q: Why not build an in-house solution? 1:31:55 Q: Why build an in-house solution? 1:32:52 Q: Any real-life examples where strictly-exactly-once Task Queue is needed? 1:35:10 Q: Maybe sending money like Venmo et. al.? 1:35:50 Q: What about bank transactions? 1:37:10 Q: What exactly does distributed consensus help with here? 1:44:47 Closing notes on Google Spanner and the state of the art wrt distributed consensus.

Follow along using the transcript.

SysDesign Meetup

2.14K subscribers