WEBVTT 0:00:00.000 --> 0:00:21.400 Programming Throwdown Episode 185: Workflow Orchestrators. Take it away, Patrick. 0:00:22.322 --> 0:00:26.760 Welcome to another guaranteed-to-be-fantastic episode. That's right. 0:00:27.300 --> 0:00:28.960 Yeah, it's money back. 0:00:29.039 --> 0:00:39.460 Yeah, I nailed it. We've been doing this too long. Okay, I'm going to try something a little different. I was thinking about this; I don't have a good formulation of it, so you can—you can sort of... 0:00:40.244 --> 0:00:40.902 Help me all. 0:00:41.729 --> 0:02:33.728 Right. But it's sort of been coming up in a number of things—and the most obvious one is when people talk about venture capital, which is, okay? You know investors, mostly tech investors, and they invest in, you know, 100 companies, but really they're expecting to do really well. They're hoping the rest don't, you know, they just assume they basically go to zero. They kind of don't care, and then they hope, you know, one or two 1000x or whatever, and pay for the risk they take over the entire pool. And part of that is this sort of like asymmetric returns. And then I began thinking it—it pops up in other things. You know, people were talking, most recently I was listening to someone talking about health recommendations, like how much protein should you consume? And people have this, you know, oh, maybe you did this much, maybe that much, maybe. And they were sort of saying, you know, up to a certain point it kind of is very, very important, and then past another point, you know, it's—you kind of get this like non-linear relationship. And then I began to think about it, and they kind of said something that led me to thinking about it in terms of work, which is if you set a certain target goal for how much you learn, you know, in a given day or continual learning, or—you know, how much output you want at work. I think internally we have this, you know, sort of everything gets kind of normally distributed that maybe some days you're going to be a little bit above, some days a little bit below, some days, you know, very rarely a lot above, you know, very rarely a lot below. And in my head at least, you know, I think we do this thing that pops up intuitively over and over again where things kind of clump together and become sort of normally, and we think that the returns there end up being, you know, kind of not super impactful, like they even out basically that the unders and the overs—you know, are okay. But... 0:02:34.234 --> 0:04:14.033 What I began to realize, and sorry, trying to bring it full circle, it's a little cloudy in my mind, but back to the VC thing: this health thing that they were sort of saying you kind of need to set your targets a little higher because actually depending on where you are on this sort of like return curve—right? That is not flat. It's not, you know, just one single slope. Um that if you are at one of these inflection points or lower, then the good days can be sort of 10x and the bad days are, you know, negative some amount. And if you are really low, what ends up happening is those negative amounts are sort of larger than the big amounts, and you can sort of lose ground or, you know, not move forward versus if you're higher up on the curve and you set your target, then even the low days, you know, you're still sort of moving forward. And so when I talk about like learning or output at work, if you, you know, sort of set your target as like my median is, I get a little bit of good stuff done, and some days I get a lot, some days I don't. Some days I make more work for myself by doing something stupid or saying something dumb at work, right? You kind of think about this as like evening out, so like slow small steps, but in reality setting yourself up for at least some of those home runs, so setting yourself up positionally, but also just where your target is, like in how you expose yourself. It sort of drifts away. Sorry, my analogy is breaking, but like you sort of shift yourself into a position where you're—you know, you still have days that are below average, but your above-average days give you this exposure opportunity for just outsized growth, right? You're taking those really, really hard home runs at work. You're, you know, going for... 0:04:14.033 --> 0:04:17.519 Just, you know, really stretch opportunities and thinking about this, you... 0:04:17.847 --> 0:04:49.049 Know it's just an interesting thing I've been juggling with, which is I think a lot of myself included—a lot of the time I don't give myself this opportunity to have this sort of outperformance, this exposure, this VC 1000x, right? You know, we talk about investment. You do the boring. Everyone always says just put all your money in the S&P 500 or the Russell 3000. You know, just put all your money in there—this very average dot return. But maybe some amount of it, maybe some small amount, you got to take those like really big swings, and you got to limit it, but if it, you know, 1000x, it could carry the whole portfolio. 0:04:49.319 --> 0:05:13.534 Uh yeah. So I can riff on this. I mean, okay, I don't know if I mentioned this in a past episode, but there's an awesome book from Nassim Nicholas Taleb, The Black Swan. Have you read The Black Swan? Yep. Yeah. So yeah, I'm a big proponent that there are Black Swans and there are White Swans. In other words, like there are—White is white swan, the... 0:05:13.534 --> 0:05:14.720 Opposite of Black Swan. 0:05:14.720 --> 0:06:10.470 So well, White Swans. Okay. First of all, White Swan is a term I literally just made up five seconds ago, but the way I would define it is like, you know, it's an unexpectedly high return, okay? Right? Because by definition, using your VC example, you know the VC funds—I've heard things like they do 100 companies and they expect one company to make up for the other 99. So it's even more extreme. But so what that means is if put yourself at the chair of a VC, when you talk to one of these 100 companies, you know you're kind of expecting them to fail because, you know statistically, they're going to be one of the 99, right? And at some point, you know, you figure out which one the one is. I'm sure. But I would consider that kind of like—you're betting on a opposite of a Black Swan, like you're betting on an unexpected, like huge boom, right? Um and so... 0:06:12.344 --> 0:06:14.504 Yeah, I think that um you 0:06:16.512 --> 0:06:33.842 know otherwise you wouldn't—you wouldn't really take the risk, right? If there wasn't the opposite of a let's call it the white swan. The white swan wasn't there, the VC wouldn't do what they do, right? If one company, you know, 5x and the other 99 went to zero, that wouldn't work. So um so 0:06:35.817 --> 0:07:29.749 they're kind of counting on that, and I think that is kind of a good way to look at things. Um I think that there's sort of certain moments where you have kind of eureka moments or you have kind of a burst or you just get lucky, and uh um and so kind of budgeting for those kind of moments, uh I think makes a lot of sense, and being being kind of ready to really capitalize—that's the other thing. Is the VC hopefully is set up to where, you know, when they do find the one that they can then, you know, 10x triple-triple down on that one. So it's the same kind of thing like you you're constantly kind of exploring, you see uh something pretty exciting, and then you kind of bet big on it. Um I do think that most returns are asymmetric. Oh, the other part I was going to say is, you know, your example with uh investing, you know, I 0:07:32.247 --> 0:08:33.860 think that um at a large enough scale things start to become normally distributed, right? So so in other words like uh if Broadcom has like a killer year, well it doesn't really matter if you've invested in I don't know if they're in the S&P 500, but if you if you invested in all S&P 500, then their killer year gets sort of diluted, right? So like at scale you don't really have black swans or white swans, at least you hope. Um uh which is which is the reason why we can all be sort of comfortable—we're not just anxiety-ridden as as as we get older, right? So so I think that that that at scale when I think about investing as someone who's not a very savvy investor at all, you know, I i just think well this the returns are going to be normally distributed around five percent or eight percent or whatever it is. But then individually or when I look at things at a smaller scale, I do think the returns are often asymmetric. 0:08:33.860 --> 0:09:38.826 Yeah, I think I might have mixed up sort of two two things. I might not. So so the one you riffed on is is good. So I think there's like taking it to career growth, not that we always have to talk about that, but taking—I actually believe I was bringing this up the other day. Most places we overestimate the impact of a failure. Like if you really if your boss came and asked you, 'Hey, do you want to try this like really hard thing?' You're like, 'Like, I'm not sure.' You know? I'm like what's really the downside of of failing? Like I think most of the time in most cases, in all cases—you know, this not this is not universal advice. I'm not a lawyer, I guess—but like most cases the risk of failing is pretty low. So it's actually really capped to the downside, you know, and it's I guess maybe similar to the VC example. The chance of the VC having money taken away from them is is non-existent. So the worst case is capped at zero, but the top case is uncapped. So if they do really bad, if 100 out of 100 fail, they lose all their money that they invested, but it doesn't mean they invested all their money. 0:09:40.362 --> 0:10:56.519 Correct, but if one goes to Facebook scale or whatever, you know, then all of a sudden, you know, they're filthy rich. And so I think it's that's probably a good mapping. I think there's another thing I was trying to slide in there, but I maybe now thinking about it, I muddy the water. I think it's it's separate, which is just like individual performance day-to-day—so like taking opportunities and getting that exposure—but then day-to-day trying to set yourself up so that like you like you're getting the reps and you're getting practice in, like you're you're getting yourself like training and skill and like setting your expectations for yourself maybe higher, knowing that sometimes you're not going to do as well. And then realizing when you practice being at a higher level that you're going to get similarly—you're it's not exactly the same though, I guess—but you're going to get those outperformance days, and those outperformance days may shock you with just like I've had that before. You call it kind of like eureka moment. You're like doing some work and being up at that higher level, you're going to potentially see more of them where it's just like what was I thinking? That like it was amazing. Like how do I get that in a bottle? Like right? I would love to be like that every day, and it doesn't happen often, but I i do feel like it happens more often when I'm operating at a high not burning out, but when I'm when I'm expecting more out of myself and performing at a high level. Yeah, I think. 0:10:56.873 --> 0:11:31.382 That makes a lot of sense. I think that uh similar to the VC, uh you know having that money to invest in the next round, like you also need to balance your work and life so that when you do have like that eureka moment, you know you can kind of uh really triple down on that, and then and then go back to a more healthy state. So I think uh I don't know if we have a term for this in my family, but but there is kind of like this notion like oh, you know, Dad's going into some work hole, you know he's gonna come back in like a couple of days or something. 0:11:31.484 --> 0:11:33.188 I call it the fugue, but yeah. 0:11:33.694 --> 0:11:34.521 The fugue. Yeah. 0:11:34.943 --> 0:11:38.875 Isn't that like an organ thing where like the organ music gets really loud and like you 0:11:39.027 --> 0:11:48.915 know I'll look at you. I've never never—that's an SAT word. I've never heard. I have to look it up maybe. Yeah, fugue state. Here we go in music. Yeah, it's a 0:11:49.877 --> 0:12:01.319 Few, a fugue state is a dissociative psychological son unexpected? Oh wait, this is bad. No, this is like this is like you have amnesia? What are you? Completely okay? Maybe I'm thinking. I guess I'm 0:12:01.335 --> 0:12:04.339 Thinking, oh Patrick's. Patrick's going dementia again, girls? Oh no. 0:12:04.900 --> 0:12:08.035 Oh no. Hang on. I don't—maybe I'm just using this. 0:12:08.035 --> 0:12:09.773 This wrong? Oh well, it's a word. 0:12:10.971 --> 0:12:15.899 Like you said. Maybe you just need to define a new word. It's like a flow state, that's what I think. 0:12:16.017 --> 0:12:16.557 Yeah, you know. 0:12:17.300 --> 0:12:18.683 Anyways, okay. But yeah, 0:12:18.683 --> 0:12:20.033 Do think I just made up? 0:12:20.033 --> 0:12:26.120 A new word for it, apparently? Because this doesn't match any existing definition. So if you use this, don't expect anyone to know what it is. 0:12:26.120 --> 0:14:20.660 You know, just okay. Kind of maybe this is too much of a segue, but you made me think of another thing. I used to coach high-level ICs at Facebook. So basically, I mean, obviously like there's—I wasn't the highest level, so there's people above me and everything, but specifically people who are trying to go either from five to six or six to seven. I did a bunch of one-on-one coaching, and then I ran this thing called the IC Circle where we would just do more like one-to-many coaching, but it was more of like a community type thing. And one thing really struck me: this person said they didn't want—this is one-on-one. They said they didn't want to go to level six because if you look at the stats, you're something like 15 times more likely to get fired if you're level six than if you're level five, for performance. And they're like, 'Well, you know, I kind of want to take this promotion.' They also were—I think they were on a visa. I mean, it's definitely part of it, but they just felt like, well, my expected value is zero if I get fired, and so keeping that probability low is actually... yeah. You're like, 'Yeah, I guess it's not zero; you get a job somewhere else.' But basically, my point is like your expected value at Meta is zero if they fire you, right? But the raise that the level six compensation is higher but not high enough to justify that, right? So anyway, what I told—what I told this person at the time was... it was basically, 'You know, okay, I totally get it. There is and I think there was kind of an issue with these people at six and above just.' 0:14:20.660 --> 0:14:39.589 Getting fired for performance.' I said, but you know, plan for the future; don't really plan for the present. So you're—hey, the present situation is you're looking around and you're seeing the level sixes on your team getting sent home, but that's—you can't kind of base the future on the present. And 0:14:41.243 --> 0:15:37.302 I think that—but I think that yeah, you have to kind of in like like the asymmetric returns kind of kicks in where you know by taking that chance and getting that promotion now, you know, you can always kind of figure out your steady state afterwards. What you can't do or it's very hard to control is having that amazing, you know, product launch that causes you to get a promotion. Like, in other words, if you have an asymmetric return and you don't capitalize on it, it might be very, very hard to bounce back from that. But if you do capitalize on it and then let's say worst case, you know, you get—you get fired for performance at the next level, I still feel like that was the right call. And just to finish the story out, the person didn't get fired; they're actually still at Facebook happily at their level. So I 0:15:38.348 --> 0:16:30.660 mean this got talked about—not not me, but by people in the field, financial people when Facebook was doing the outsized bonuses for hiring AI people. I think they've turned it down now, but they were saying a little bit of a similar thing: there's an expectation of a continuing current state. So, in other words, I don't have to take this right now because I could take it in 12 months or you know 24 months. And in reality, actually it turned out it was like a window of like 30 days. Yeah, it was like really short. But it's sort of to what you were saying: those opportunities, you think they're going to come at some even spacing or always? And that's not right. It's not true. Your ability to get a promotion, your ability to be on that project, your ability to start something new—greenfield—make your mark, whatever—those things aren't evenly distributed in time. And so yeah, I agree with you. All right? So just to 0:16:33.023 --> 0:16:46.034 Follow up. I did find the correct definition of fugue. I wasn't that crazy, although it apparently has a slightly negative connotation, but it means to do something where you're completely aware and focused, but afterwards you can't remember what happened, and so. 0:16:46.034 --> 0:16:49.560 Oh, that sounds—that's perfect. That's actually exactly what. 0:16:49.560 --> 0:16:56.969 So actually it took me a little bit of searching. Apparently, it's a very deep depth, like a... yeah, back catalog definition, but it's like, 0:16:57.154 --> 0:17:01.103 What? What idiot wrote this code? And it was you a week ago? Yes? No. 0:17:01.119 --> 0:17:14.700 No. That's what I like. You get really focused and you churn out all this stuff. My new one is writing code and then like having other people sort of help me like finish it off or whatever, and then submit it under their names, and then I think they wrote the code, and they're like, 'No, you wrote this code.' 0:17:18.974 --> 0:17:20.661 Oops. All right. All right. 0:17:20.661 --> 0:17:21.876 Time for news. You're 0:17:21.876 --> 0:17:47.239 Up first. All right. I'm up first. So it won't be this weekend when—when this comes out, but that's okay. Just recently Andre Karpathy, I believe is how you say his name, um, you know he's gained some renown recently for a little bit being the people's AI person, I guess. Like, you know, he's not—he's not running a big AI company. He clearly knows what he's doing. He can explain concepts, but he explains them in depth. Let me let 0:17:47.239 --> 0:18:09.869 Me let me add a little bit of color there. So Andre was the head of AI at Tesla, and he left, and he basically is like I don't want to say retired—that's not the right word because he's very productive, but he quit to do exactly this, like to just build things for the people and teach the people, and so he's in a way like doing some monastic work right now. 0:18:10.864 --> 0:18:13.784 Well, don't talk about 'star words,' 'monastic.' Oh, there. 0:18:13.784 --> 0:18:15.387 We go look at this. 0:18:15.387 --> 0:18:16.315 Wow, man. You 0:18:16.315 --> 0:18:20.719 Thought people thought they were coming here to talk about programming. Uh, so 0:18:22.795 --> 0:20:00.147 I—you know, in some ways actually I think to your—to your thank you for that. Actually, that's a color comment. It is very good. Um, you know, I think our podcast tries to be although I don't say we're as effective as this something in a similar vein, trying to break down stuff for folks. Um, he released a new GitHub repo out of sort of the blue, as far as I know. I don't think he was—I didn't know he was working on it, called NanoChat. And NanoChat is previously he had done some other sort of examples of training. I don't want to say like toy, but like toy Transformer networks and models and getting them up to sort of GPT-2 level, sort of, you know, trying to be two-two level, like they're like, 'Wow, okay, it's better than if you—if you remember. Oh, I don't know, you know, while ago, decade ago people were doing Hidden Markov Models trained on, you know, Harry Potter, and it would produce words that like if you didn't look too closely they kind of look like maybe sentences, but they didn't make any sense. And then, you know, we kind of got into the ChatGPTs. These GPTs were good. So he's sort of done some of that before, but now sort of what he's done is done an end-to-end pre-training doing the chat UI, you know, constructing the models and everything end-to-end in a single repo so you can really kind of study it. And really awesomely, if you don't know training the frontier models—I don't even know what the estimates are now, like 10 million dollars to train one. They're very, very, very expensive, so most training outside of maybe the 10 companies in the world is doing fine-tuning, you know, other small things but not the sort of pre-training. Um, that's reserved for people who have much more money to burn than—than myself. Uh, yeah. 0:20:00.147 --> 0:20:10.677 And 10 million I think just is one instance, so it's like if you miss a colon, I mean, you're going to catch earlier than that, but like in theory if you have some data issue, now it's another 10 million. Yeah, yeah. 0:20:10.980 --> 0:22:09.578 Yeah, you know who knows how many times they have to do it. Yeah, that's right. And that was a big thing if you recall DeepSeat getting really popular for, you know, they were had their moment. They've kind of, you know, taken a little step back, you know, in terms of the general awareness, but that was their big thing—was doing some of this pre-training cheaper. Okay? Anyways, back to the story. So Andre's Carpethys repo also made it so that you could do all of this for a hundred dollars. Now, a hundred dollars isn't nothing, but I'll make a couple observations. So first one is again compared to that other amount people spend more than that go play a round of golf, you know, go do something. You know, it—it is a lot of money. I don't want to diminish it, but it is enough like if you're doing it for learning, if you're doing it for understanding, if you're, you know, really getting into this, it's attainable. It's a hobbyist level amount that you could spend, which is awesome on sort of relatively reasonable spec hardware. And he has the scripts for doing all of it, sort of set up, which is really awesome. Um but then of course, the second observation I'll make is twofold: one, the cost for his stuff will go down if you wait, you know, six months, 12 months. I would expect it to be cheaper as new hardware and instances and supply come online. But the second thing is, as a general sort of observation, a lot of this stuff—this is where you're seeing the growth, you know, we're not seeing ChatGPT whatever five or any of these other things become 2x better, 3x better. It's not really happening, but the cost to train, the cost to run inference is all coming down dramatically. And here, you know, I think is a little bit of an example of, you know, sort of the rise in the hobbyist stuff and the decline of the cost, you know, you're coming to, you know, not exactly a meeting point but becoming and converging together to where we can start to not get to state-of-the-art but push to something that five years ago would have been like, 'Oh my gosh, like you did all of this?' Right? That would be crazy. Um so it's a reminder of the insane progress we're seeing. 0:22:09.730 --> 0:22:54.263 Yeah, totally. Yeah, I think it's a really good kind of recap. Um yeah, I mean this stuff's super exciting, I think. Um a lot of innovation now is and I think Andre actually talks about this is around, you know, reverse engineering the best possible prompt. Um I don't know if I brought this up, but I don't think we have—I had a situation at my current job about a month or two ago. Um we changed models, so a model that we were using was deprecated, and so we moved on to the next version of the same model, and the performance went way, way down, like really, really far down, like went from maybe 80 to seven percent kind of thing. 0:22:56.305 --> 0:24:52.793 Um and after a lot of debugging and and kind of, you know, looking at a lot of numbers, we realized that all we had to do is change the—so so we had a prompt that was about three or four sentences in a paragraph. If we swapped the first two sentences, the performance went back to 80 percent. So, so basically like the the ordering of the sentences, the way the thoughts are and it's not like it wasn't illogical the first way, right? It was two independent thoughts, and they just if you switch the order, massive improvement. And so what that says to me is that we humans shouldn't be writing these prompts. I mean, you know, we write the first version, but like at the bare minimum, you know, we should have a system that just like reorders them a few different ways. It picks the best ordering, like somehow breaks them into independent thoughts and then tries different permutations. But but at the limit it would be, you know, some type of feedback mechanism that's constantly updating all of these prompts. Um but that I think is where I'm seeing kind of a lot of innovation, and I think it's because of cost exactly. You know, as you said when it costs 10 million to train a model, what we ultimately need—I think NanoChat's a good way of getting there is, you know, a set of benchmarks and a set of models that are small and trained for very cheap. Um And then you can, you know, learn some insight. Someone can invent the next thing, right? Like maybe it's SSMs, maybe it's Diffusion Chat models, right? Someone invents the next thing, and then it blows everything else out of its weight class. But but it's still nowhere near ChatGPT, but but it totally dominates the very cheap ChatGPT. And so then you could say, 'Okay, this is now worth spending 10 million.' I mean. 0:24:53.097 --> 0:25:35.132 I guess that's partly where I—I think the running local models, you know, on your whatever home smart device, your personal assistant on your phone, where we were just checking before briefly. There's a, you know, cloud outage this morning, so that happens from time to time. And so I think having some of these models run locally and I think when the hardware get sophisticated enough and the models get distilled enough to be like absolutely actually useful in those things, I bet you see a lot more investment into—into that as like a focused product that you can have like what is state-of-the-art for a capped size model. 0:25:36.179 --> 0:27:32.540 Not unbounded, right? Yeah, exactly. Um all right, my news article is Pydantic AI. To be totally honest, I picked a random news article. I really just wanted to talk about Pydantic AI. I could have made it a tool, but I had a different tool. Um so this is really cool. I used this um the other day. Uh I built a very simple test like a prototype. So I downloaded—remember PC Part Picker? Yes, remember that website. Yeah, I don't know if they're still around, but um someone posted basically a CSV file of all the inventory of PC Part Picker. So there's a CSV file for heat sinks, a CSV file for processors, etc., etc. And um I use Pydantic AI, which is this way to build um AI agents, which we should probably do an entire show on, but it's it's taking off. And I think it's very interesting. Um and I only give it access to one tool, and the tool is uh, you know, DuckDB. We've probably talked about this. Yeah. So the tool literally is you can um it you pass in which CSV file you're interested in—video cards, CPU, etc.—and a DuckDB SQL query, and this this AI tool opens that CSV, runs a query, returns back the results. And so that was it. And so this whole AI agent thing with Pydantic AI, it was like 80 lines of code or something, and now I have this thing where you can ask it to build you computers, like you can literally type in, 'Hey, I want a computer with the NVIDIA GPU at least 16 gig of VRAM for under two thousand dollars,' and it will actually build a computer for you out of all those parts, and it runs SQL queries, and you can actually. 0:27:32.540 --> 0:28:19.100 See all the steps. Like first it runs like a DESCRIBE SELECT on each of the CSV files so it can get all the schemas of them all. Um and then it starts running queries. Sometimes it'll type in a query that doesn't compile. DuckDB will like send an error back, and Pydantic Connect will actually catch errors and turn them into strings and send those strings back to the AI, so the AI like got an error and was like, 'Oh, I need to change the query.' And it did, and and it eventually gets there, and it generates like uh builds PCs that that are good. So but but I just found this amazing, and I'm sure there are other tools out there, but I just—I couldn't believe how in just a matter of hours I was able to go from nothing to all that. 0:28:20.406 --> 0:28:28.118 That is really cool. And it's really—I mean like commoditizing, like yeah, it's coming coming to the fact that yeah we can do that and it's super. 0:28:28.422 --> 0:28:55.894 Cool. Yeah, so if you're doing agentic stuff, these—this is the same company that made Pydantic, which is a um uh which is kind of like a data class system that's very popular and really well done. Actually, it's the same the same people made SQLModel and FastAPI and like a ton of amazing libraries. So so this is also, you know, walking in good footsteps. 0:28:56.114 --> 0:30:55.419 My next news article is something that I actually botched, and then it was amazing how it was amazing before it was amazing. Okay? Here we go. So I saw this news article, and it was 1000 Starlink after a launch this again this weekend from SpaceX, so all the Starlink satellites up. And there's been actually a huge thing like the amount of money they're making. It's a private company still, you know, owned by SpaceX, but the number of subscribers they have, like as far as like a tech story, it's it's actually really astonishing. Um would love to have—have invested, would invest. It's private, so it's it's a bit bit of a thorny issue to try to get access, but um this is a great idea. I I'm really for it. But you know, I saw this, I was talking about it that you know there's a thousand thousand satellites, and then I—I was like, 'Wait a minute. Hang on, let me look at this again.' And it's, you know, oh, it was a thousand Starlink satellites had been launched in 2025 so far. So yeah, but I thought it was like, yeah, but I already thought it was amazing. It's like, 'Oh, there's a thousand of these satellites, you know, orbiting around.' How cool is this? It's like, 'No, that's incorrect. There's been a thousand launched to this year.' So then I wait a minute, how many are there? And it's like 8,400 and something in orbit. There are so many in orbit now that actually every single day one to two Starlink satellites incinerate back into the atmosphere from sort of end-of-life thing, which is expected. So they put them into, you know, relatively low Earth orbit, so their orbits, you know, there's a little bit of air friction and whatever they naturally decay. They're meant to do that. You don't want them staying up forever and, you know, getting obsolete and then just drifting around as space junk. So they—I actually think they do a process for, you know, intentionally sort of making it so they'll de-orbit, you know, deliberately rather than just, you know, chaotically. Um but basically there have been so many gone up. 0:30:55.420 --> 0:31:51.023 going up that one to two every single day are just basically burning up, and it's still this like insanely lucrative business doing all this crazy stuff. And there are eight thousand found a visualizer you can click and actually see all of them because they're, you know, sort of orbit information is known, and it turns out orbital mechanics is like well understood, so you can actually just go and visualize the entire like mesh of all of the orbits, which is very well planned to get nice coverage. You can see new ones that are being deployed are like clustered together and slowly drift to their, I guess station. I'm not familiar with the term, but it's absolutely bonkers just to think about the scale of this. And I don't think maybe I've talked about on the podcast before, but I was talking about it with some local friend group, you know, just sort of how if you had said this, you know, whatever 10 years ago, like, oh yeah, there's like this darling thing. It's roughly competitive with broadband. It works great. You can play video games on it like it's not super laggy. It's in 0:31:51.023 --> 0:31:53.580 this little dish, just, you know, flat dish thing, and 0:31:54.139 --> 0:31:58.300 people would be like, 'What? No.' That, and then now people are like, 'Yeah, of course. That's called Starlink.' 0:31:59.494 --> 0:32:15.002 Like, yeah, I have that in my cabin. Yeah, it's from Do they have coverage in the ocean? Or like how does that work? Is it the kind of thing where it's only in places where that can be populated, or is it literally circumnavigate? They have slightly 0:32:15.002 --> 0:32:48.904 different fee structure. I'm not clear if it's because they can or because there's a reason, but you know, people have it on like their ships, on their traveling RVs, on their boats. Yeah, there and now this weekend or whatever they're demoing. And I guess maybe some people still don't know this, but there was like a couple weekends ago or whatever they were talking about one of the airlines—I want to say United or American here in the United States—was going to be putting Starlink dishes on all of the planes, which they've used a different satellite provider. So if you've ever gotten satellite on an airplane, it's, you know, it's not 0:32:49.140 --> 0:32:49.849 Very good. Yeah. 0:32:49.900 --> 0:33:14.976 Yeah, but now they were like flying this reporter up and sort of they were able to take a FaceTime call, and it felt like normal. And they were like from an airplane, and and they were just like their mind was sort of like blown that you know flying through Spain. I think Elon Musk even ended up retweeting about it, and he's like, 'Well, if you're traveling whatever the orbital speed—I won't quote it because I'll get it wrong—of a Starlink satellite, he's like even airplanes look basically still.' And so that makes sense. 0:33:14.976 --> 0:33:15.651 Flying through an 0:33:15.651 --> 0:33:23.397 airplane. Like, oh my gosh, it's moving so fast. Aren't there going to be all of this? He's like, 'No, they basically look like they're standing still because we're flying so fast.' 0:33:24.342 --> 0:33:25.641 Wow, that's incredible. And 0:33:26.772 --> 0:33:37.032 So it's just this interesting sort of normalization of what is it? It's just absolutely it's rocket science, right? Like just all of this stuff, you know, being up there. 0:33:37.032 --> 0:33:48.220 And going around. Is do they so when they when the sat when the Starlink satellites are are, you know, shuttled—um is that shuttle reusable? Or like how does that work? 0:33:48.220 --> 0:34:32.620 Yeah, so they did the Falcon 9. So if it comes back and lands normally on the barge, so they go up, launch it in a second stage, the top part of the rocket as a fan go up, and they don't relaunch that. That's the new thing, the really big one that they make a big deal out of: Texas Starship and the booster that goes with that. The full thing is supposed to be reusable. So that's like where they see the future. But for now, the first stage, which has most of the, you know, more of the rocket engines and everything, basically goes up and then comes back and lands. And those are the videos you see. And so that's why they're able to put—well, part of why they're able to put so many satellites up in orbit, but there still is some portion of the launch vehicle that doesn't get reused. Got it? 0:34:32.620 --> 0:36:31.340 It makes sense. Cool. All right. My second news story is the introduction of the apps in ChatGPT and the ChatGPT Apps SDK. I feel like this is really powerful. I feel like there's something here. So just to explain at least how I think of it because it's still very early, basically, and I don't have a lot of details on it yet—I don't think they even kind of have a lot of details yet themselves—but the way I imagine this working is, you know, let's say you ask ChatGPT to play your favorite song or you ask it to buy eggs from—from, I guess if you say from, let's say, buy eggs from Walmart, right? So right now, if you do that, you know ChatGPT is going to say, 'Hey, I can't do any of these things. Would you like me to do something else?' It's always so polite. It's like, 'Would you like me to just talk to you about eggs?' But, you know, in the future with these apps, you know, you will install the Walmart app, and when you ask a question, you know the OpenAI folks will figure out from your question whether they should route that question to an app or not. And if they decide they should, it gets routed. And now Walmart, which probably is using ChatGPT anyways, but like now Walmart gets your question, and they're poised to handle it differently. So for example, if Walmart gets your question, they could order the eggs. They know you have your identity from you being logged into ChatGPT, and so they can order the eggs, and eggs show up at your house the next day, which is pretty cool. I do feel like this is kind of like a new kind of app store moment. 0:36:31.340 --> 0:37:23.275 Which is pretty exciting. I do feel like we'll kind of look back on this and say, 'Oh, like I wish I'd built like the—you know, whatever the most obvious app is.' Maybe groceries? Is it? I don't know, but I feel like now is the time to think about what is the app that obviously everybody wants because apps aren't even allowed on the store yet. I think they're going to announce a date when you can start submitting apps, but you have access to the SDK. So if you have something where you think ChatGPT would be a better experience than going on the web or whatever else, you would do now is the time to build that thing so that as soon as they announce the submission process, you could be first in line. I do think there's something just extraordinarily powerful here when. 0:37:23.562 --> 0:38:06.475 This was launched. I saw it was like a thousand startups are killed by one—one slide or something. It was like the headline. But I guess maybe I like the way you explain it. I guess if you develop your app, it's sort of like now you can host your own like server. If you do something local and your chat agent can query your, you know, local server that's answering something. So it's like sort of chat. You do some training or upload some workflow or something, and then OpenAI will route ChatGPT queries to that internally if you host something there, like if you are a state in their ecosystem, can you run arbitrary code? So it's not completely clear. Yeah, let me let. 0:38:06.593 --> 0:38:32.580 me see if i can explain it so the idea is you know you you you submit an app uh open ai approves it now you're on their app store right so so a person who uses chat gpt installs your app right now when that person asks questions to open ai um open ai can route some of those questions to you and uh um and and 0:38:34.184 --> 0:38:45.389 You could—you could then use OpenAI as part of helping, you know, answer the question, but they're now expecting you to respond with an answer. So in the case, 0:38:45.389 --> 0:39:13.820 Like you said, so you're a grocery store. You do work to make sure that like your OpenAI app knows how to call your inventory and pricing, yeah? You know, REST API. And so you give all of that information or whatever to your sub-component app, and then okay, and then the user installs that one, which is why it doesn't have to disambiguate across every app on the App Store. Okay? 0:39:13.820 --> 0:40:09.257 Yeah, I think that's right. And so you can also build tools like I talked about with Pydantic AI. So in addition to being sort of so open-ended that ChatGPT just calls you with a string of text, you can also have a tool. So for example, maybe you have a buy groceries tool, and that tool takes like some very specific structured data—some like the name of the grocery and how what quantity, and all of that. I mean, you still have to do all the validation yourself because you don't never know what ChatGPT is going to call your tool with. But at least that might be more structured than, you know, you just have a string of text and have to figure out what to do with that. Very cool. Yeah, definitely check this out. I mean, I feel like folks out there, you know, especially if you're a hobbyist, this is like something to get in on early. I feel. 0:40:10.337 --> 0:40:28.191 Yeah, I guess that's the what the dozen fart apps on the App Store that made, you know, tons of money in the first year of the Apple App Store. Yeah, yeah, exactly. Need the equivalent: ChatGPT Chat. So I try to make it ChatGPT Fart GPT. I don't know, Fart G. Okay, oh, never mind this. 0:40:29.035 --> 0:40:30.942 Is short devolved. Let's not ChatGPT. Oh. 0:40:33.017 --> 0:40:38.620 Boy. All right. Let's just move to book of the show. We'll have to get rid of the 'friendly' tag or whatever the safe for work tag. 0:40:38.620 --> 0:42:25.034 I don't know. We have a friendly no. Anyways, my book of the show just hard pivoting—segueing, forcing a segue is I'm repeating The Will of the Many. I had this book—I believe a couple episodes ago, I'd only just started it. I have finished it. It was really good. I really like it. So I felt like I owed the people the like follow-up that if I was going to pitch it as, you know, 'Hey, I'm starting this book,' it seems kind of interesting, I did finish it. It is good, you know. And the second one comes out probably by the time this episode's out. It'll sort of be out. So if you haven't started this book and it sounded interesting from my last pitch, I guess I could repitch it. But anyways, so you know definitely has like an equivalent of kind of a magic system—a loose magic system—where people can sort of seed their something called like a will, like sort of part of their life energy, I guess. It hasn't been fully explained yet, and they can sort of give it to somebody else. And those are assembled into the government in a kind of form of a hierarchy, a pyramid. And it's all about sort of the dynamics, the politics of that. So, of course, you know, a story about a boy who's an orphan and ends up getting, you know, sucked up into intrigue and mystery and all of that good stuff, but by the author James Islington, who I've recommended other books from before, and definitely something that I would recommend now having finished it, and I'm looking forward to the next one. I guess I could have made in my book of the show, but the next one seems like it's going to be really good as well, and I'm looking forward to reading. 0:42:25.034 --> 0:44:23.420 That all right. Um yeah, I'll definitely have to check that out. I need to get more into reading fiction. Um I my book of the show is actually a podcast episode. I'm cannibalizing our show here, but basically um I had a long drive, so long story short, uh I was. It was just me and the boys, me and my two sons, and we were driving to Houston and back, and you know they're playing, you know Nintendo Switch or whatever they want to do. So I'm like, I'm gonna listen to a podcast, and I listened to the whole six hours of this podcast with DHH, who's the person who made Ruby on Rails. Um I thought it was fascinating actually. I kind of um I kind of went in thinking, well you know this is somebody I have kind of heard about. A friend of mine told me we should have DHH on our show; that would be kind of cool. Uh DHH, if you happen to be listening, which I doubt, but or somebody who knows him or something, totally down for that. But um you know, I'd heard of him and I remember specifically him posting a lot about getting off of the cloud and he ended up moving to like Colo. Um so you know, I thought it'd be kind of interesting, but I had I'd be honestly a little bit low expectations of just any podcast episode where um uh it's it's just kind of a one-sided, where the person just basically talks for six hours. But I was like blown away actually. I felt like the content was really interesting. Um actually the whole it really goes against a lot of the sort of like common themes that you're seeing a lot of. So for example, um DHH like despises going into any 0:44:23.420 --> 0:46:22.540 office, and so he's like, you know, remote work is the only work. It's a really interesting perspective. Um you know, he talks about how to make remote work work. Well, he's been doing this for like decades, right? So he gives a lot of perspective on that. Um they uh the company, which I think is called 27 Signals, they took zero dollars of funding. So they basically bootstrapped themselves by doing um uh by doing contract work. So they did contract work until they had kind of enough money, and I guess they maybe they structured the contract work in such a way where they could share the IP and then eventually open source the IP, and it became Ruby on Rails and all of that. Um so zero dollars of venture, zero dollars of anyone investing. And he said something kind of interesting. He said, you know, people gave me a hard time because they say, well, you know, you um so their their product, which I forgot the name—oh, Basecamp. So their product is Basecamp, which is similar to Jira, and people say, 'Look, you could have made Jira.' You know, your product is actually like in many ways better than Jira, and Atlassian is like a giant company, and you're this tiny company. And his response is interesting. It's basically like it's a tiny company, but you know I own half of it, and like Jason, who's his co-founder, owns the other half. And actually like owning 50% of a tiny company is awesome, and I have more than enough money and all of that. And it was like a real kind of like I guess like it is also like I keep wanting to think of the word monastic, but but it is kind of like totally antithetical to everything you see on social media. Everything on social media is, 'Look at this big round we raise.' You know, no one ever says, 'Look at this 30 million dollars we raise; we only had' 0:46:22.540 --> 0:47:01.885 to give up, you know, 70% of the company.' Like no one ever says that part of it, right? Um and everyone seems to be uh just like extolling all the virtues of being in person. Um and yeah, I feel like just to have a different perspective, it was pretty mind-blowing. So I highly recommend folks listen to it. Um also, you know, my dad was a race car driver, so he spends about an hour or two talking about race car driving, which you know hits close to home, which I loved hearing about. Um that's more of a personal thing, but I found the whole interview fascinating, and I'd highly recommend folks check it out, and it's six hours, so I think it counts as 0:47:04.365 --> 0:48:25.819 a book, fair enough. Yeah, I think 27 Signals is private. I've looked before. I was just looking again to see, but I guess one of those interesting things about private versus public and to your point, not only they have a lot more discretion—like it just don't have to share a lot of information and get judged about decisions. And so because he doesn't have investors or all of this like even people you know here are some Hacker News post, and they're just saying like can we even estimate how many subscribers they have? How much money? And people are like throwing out numbers, but they're all, you know, just speculation. Or or you know based on some, you know sort of swag at it; like they're not accurate. And it's just kind of interesting to think about like the amounts of stress reduction that would give to someone to say, 'We're just going to make the decision can we continue to pay the people who work for us? Can we continue to make money that we're happy with?' And like there's always that question like what's enough, you know? And like if you don't have an answer to what's enough, then maybe you just end up on one of these quests to, you know, seek power and fame and fortune. And and that's okay, I guess. But if you can say this is enough, and he is saying, 'You know, I can do these really cool things—drive race cars and have this business that doesn't make me hate my life.' Then, you know, that that is awesome. It's great to see like you said sort of another way instead of it only that there's only one way. 0:48:27.222 --> 0:50:26.459 Yeah, I mean somebody said uh the host of the podcast said, 'You know, hey, you know one thing about working remote is, you know, it's kind of lonely. How do you how do you deal with that? How do you meet people?' And the guy was like, 'DHH is like get married, have a family.' He's like, 'Yeah, I have kids running around the house; I'm never lonely.' You know, I was like these are the kind of things that like just like just a different perspective. Like I've never heard anyone give an answer like that. And you know it's so different. I mean, I did we just got back from Houston, so um so I haven't had a lot of time to really like sleep on this, but but it's such a different perspective that I have to really think about it. But I think there's definitely nuggets of wisdom there absolutely. And um I do feel like uh yeah, I've kind of ping-ponged between, you know, I was in the office Covet hit; I worked remote for what four years. Now I've been in the office for the past year, and I do actually feel like um I feel like remote is better in many ways, but it has to be done correctly, and you have to be in the right situation. So I think I think remote where there is an office and like 90% of the people are in it, I think that's hard. Of course, you can make it work, but I think socially like it takes a lot of uh you have to be very active on managing that. Um I think remote where everyone's remote or in the office, you know, it's um you're on Zoom all the time anyways because there's always this is okay, this is this is my perspective. Everyone's remote, everyone's remote because you go into the office, but there's the Seattle office, or there's Bob who's remote. Like as soon as as soon as you have more than one office or a single person in the company who's remote, now you're 0:50:26.459 --> 0:50:29.464 remote because you're on Zoom all day anyways. 0:50:29.633 --> 0:52:07.420 guess my my thought here is similar to what i was saying before i it's i don't think there's one right way i think to your point it's situational it's company by company it's person by person some people hate like i couldn't be at home it's noisy other people like i can't stand the office it's noisy people dropping in on you like it's you know for almost any statement you can someone will say the same statement in positive and negative light yeah um and so like anything my how do you want to say my thought is you're gonna have to try really hard to prove that there's a singular right answer and no nuance anytime someone comes and says remote work is the worst it doesn't make sense it never works or if someone says and the office is dumb it never works we should all be like whenever you make a a statement with no wiggle room those are always the hardest to defend right and it's like come on there's there's got to be a gradient here and you know and if there's a gradient i think that you know i will say even being people fully in the office having the setup that we could do fully remote means people have some flexibility to do if they need to take a week a week they may have taken a sick week or something away or they needed to you know i was pointing out be home there's so much random stuff where people expect you to be at your place of living for delivery or for some improvement or to come in or whatever and then they change the date so having some flexibility to handle those situations has been a a net benefit regardless of sort of what percentage allocation you spend office or at home 0:52:08.166 --> 0:52:10.697 yeah totally makes sense all right tool of 0:52:10.697 --> 0:52:15.020 the show oh man i think i'm losing my voice it's not good 0:52:17.180 --> 0:52:32.700 all right mine is a is a game and it's a game i am almost certain i didn't look because i didn't want to know but i'm almost certain that i've recommended before it's uh everyone knows it everyone has it but i don't care because i got sucked back in i was i was uh free and then i am no longer free 0:52:34.491 --> 0:53:18.045 i've been playing factorio again uh and not even although it looks amazing not even the space age dlc i'm just back to playing vanilla bass no mod factorio i've been playing it on my switch and it has been it was it's always like a little awkward it's so much better on the pc with the keyboard i will admit but once you get into the groove and you're like on an airplane which which i was recently a couple times you know can sit on the couch with the family you know when they're watching some show you just really don't want to watch for the fifth time in a row which whatever anyways like you if if you can quote the show and like you know what's going to happen from the kids show you know you've seen it too many times which 0:53:18.045 --> 0:53:18.855 means they've seen it 0:53:18.855 --> 0:53:50.024 Even more than you, because you know you haven't watched it. Every time they've watched it—um, anyways, I'm sucked back into Factorio. If you've not played it, you either are going to assume it is either going to be work or amazing, or you won't understand it. I guess those are the three outcomes: Be like, 'This is just like work; it sucks, I don't want to work.' Or you're gonna go, 'I don't get it. This is like weird,' and just like this is not fun. Or you're gonna be like, 'What? Yeah, fugue!' You're gonna spend 10 hours and then you have no clue what happened, you know. 0:53:51.441 --> 0:54:19.580 i posted uh about a month ago um i had a friend a close friend who's trying to grow his twitter presence and uh just actively trying to do this and he posted saying you know i went from 1300 followers to 7 000 followers in a month uh here's a list of things i did and it was it was a list of things and i replied and i was like how i went from 1300 followers to 1300 followers in three years and it was a screenshot of factorio 0:54:21.980 --> 0:54:29.798 But Factorio is amazing. I mean, it's—it's so satisfying. It's like gotta be one of the most satisfying feelings, like building a factory. 0:54:30.085 --> 0:54:54.859 I have a confession. I have a probably—I don't want to look how many hours. I don't actually maybe the Switch does track? I don't know how to look it; it's not prominent like on Steam, so I'm not sure how many hours I played. I've never launched a rocket. What really is that bad? It's so bad, right? I need to do it. It's become like a goal, like 'I before I stop this time, I have to do it.' I just always do other stuff. I just—I don't know. Wow. 0:54:54.859 --> 0:55:06.619 That's amazing though. Actually, it's amazing that like you have built all these intrinsic rewards, you know? It's like, 'Oh, can I—like, well, give us an example of something that took you a long time that wasn't Rocket?' No, it's not. No. 0:55:07.193 --> 0:55:39.492 No, it's worse than that. I play for a while, I start to get into like a good in it, in reaches—like there's always some place and it's always a little different where you need to like do a bunch without like a good progression so you can like progress early game progress, and I'm getting better, I think actually each time. And then I go away for a while. I'm like, 'I don't have time to invest to like double my output here, build another one of these, or you know get a blueprint working, or get the bots going.' So I just like I put it away for a while, or I get busy, and then I come back. I—I don't remember where anything is, so I just start a new. 0:55:39.492 --> 0:55:41.399 game and then I just like start over. 0:55:41.399 --> 0:55:50.190 And I just like do with all the improvements. So it's just like whatever you call it, like greasing the groove. Like I just keep playing the early game over and over again. I— 0:55:50.190 --> 0:55:52.975 know. Yeah, that makes sense. That makes sense. It's bad. I'm a— 0:55:52.975 --> 0:55:58.155 bad person. Like I need to finish the game. There's so much the developer put in. I haven't even experienced my. 0:55:59.168 --> 0:56:12.145 Kids play with a Creative Mode where you're not even a person. You could just build anything anywhere. It's unlimited. Um and you kind of like teleport around the map and they just build random things with it. I— 0:56:12.617 --> 0:56:20.397 turn the biters. Like the Biters are there, but they won't attack you unless you attack them because—uh yeah, I just—I don't know. It's too anxiety-inducing, man. Well. 0:56:20.717 --> 0:57:18.025 You know, I mean, you probably know this, but the construction robots just totally changed the profile of the game. Like, um, like I have mine near the end game; mine will be set up where I'll have basically stamps that like stamp an entire screen's worth of content, and so it's like four RoboPorts, a bunch of these like electric towers, and then a bunch of solar panels, and it just stamps down. And then because the RoboPorts are part of the stamp, you know, it builds then the RoboPorts, and then it can build the next stamp, and so I'll just walk across the screen stamping like hundreds of these, and then I'm like done with power. It's like, okay, well, I don't have to think about power for the rest of the game. Nice. All right. My tool? Oh, go ahead. My tool is Nip.io, which is apparently a great way to get in trouble at work as yeah. 0:57:18.025 --> 0:57:19.206 Still have no clue. 0:57:20.438 --> 0:58:10.911 What this is? So this is like, I could see why workplaces wouldn't like this, but I'll explain what the tool is. So the idea is um there are many places where you know you can't use just an IP address; you have to use a hostname. And so you end up in your code writing all these things like, 'Oh, if I'm local, then use like this IP address; otherwise, use this hostname.' Or you end up in weird situations where like if you're running on Kubernetes, you know, then the host you want to talk to is just called 'database,' and you can like ping database, you know, curl database like from the other nodes in that Kubernetes cluster. But then 0:58:13.071 --> 0:59:52.414 when you're not running in Kubernetes, now it's like, 'Oh no, it's actually localhost.' It gets all confusing, right? And so this Nip.io is like a cool way to turn any IP address into a DNS. So basically, you take the IP address—whatever it is, you know, 1.1.1.1—and then you add dot nip.io at the end, and now you have a DNS that's going to resolve to exactly that IP address. Um so, you know, you're kind of trusting because I will say that for the reason probably is blocked for Patrick. Patrick's job is that you're kind of trusting these people to convert that DNS to the correct IP address. So, you know, you might put in like 2.2.2.2.2.2.nip.io, and it resolves to 2.2.2.2 because the people behind that website have built like a DNS lookup that does that. But they could change that code tomorrow; they could make it so everything.nip.io goes to Google or goes to a web-based Bitcoin miner or whatever like that. So, that is sort of a vulnerability that you're taking on. Um it is, you know, it's powered by a company that seems like pretty reputable, but um but that is a risk. But, you know, when you're doing development debugging stuff, especially locally on your laptop or you're getting started with a new project, it could be an easy kind of convenience. Yeah, that's my tool of the show, but 0:59:52.414 --> 0:59:57.308 I could see. Yeah, like you said, there are specific use cases where it would be 0:59:58.371 --> 1:00:10.960 really useful. Yeah, totally. Um there's a lot of code, I mean, especially third-party code that expects a domain name that just will not take an IP address, and this is a way to get around it. All right, so 1:00:11.567 --> 1:00:13.900 Oh, go ahead. No, no, no. I just. 1:00:15.449 --> 1:00:20.646 Anyways, we're gonna orchestrate the show here with a 1:00:20.646 --> 1:00:25.185 workflow orchestrator. I'm distracted today, dude. We're just like I just. 1:00:25.405 --> 1:00:28.020 Like barrel forward. We're gonna just fugue, fugue. 1:00:28.206 --> 1:00:29.708 This um we're gonna 1:00:30.889 --> 1:01:17.650 fugue it. Um oh, I'm never living that down. It's all fugued up over here. Um so, so okay, let's talk about why you need a workflow orchestrator. So um here's a very simple example, and actually kind of spoiler alert, I actually asked this as my interview question. So if if you ever end up getting interviewed by me, you're gonna probably get this question. Let's say you're building, you know, YouTube, and specifically like don't worry about like the social networking or any of that, but just, you know, you upload a video; it transcodes your video 100 different ways for mobile, and you know, low bandwidth, and all that. And then when other people go to that URL, they get your video. So um video is 1:01:19.793 --> 1:02:41.180 Maybe not the best example of this because it takes a while to upload the video, but you know, you upload the video and then it has to go through and do a whole bunch of work, right? That transcoding of that video to other formats. Maybe you uploaded it as a QuickTime video which, you know, they don't want to serve that on the web, right? So that is a time-intensive process, and so it's not practical to like just have someone wait on a web request, right? Because if that person just hits F5 or something and closes the connection, you know how is that going to work, right? There's kind of two problems here. Problem number one is sometimes you have to do things that take a long time, and then problem number two is sometimes you have to do things that are kind of non-interactive where someone will leave and you need this thing to keep working and eventually report back in some way. And the last category of why you need this would be, you know, you have a lot of things, a huge batch of things that you need to do. So for example, actually this is a very real example for us. I went through and re—oh, not. 1:02:43.172 --> 1:04:04.105 Uh, uh, re-transcribed all of the episodes of Programming Throwdown. So if you look at the transcriptions, they're better than they were before. The old system I was using was massively out of date, and I felt like well, since I upgraded it, I might as well just re-transcribe everything. And so, you know, we have 180—some odd episodes, 184 episodes—and so that actually is expensive. But even if it wasn't, it's a batch job that I have to just submit. A hundred of these and just tell me when they're all done, right? So that's the basic reason why you'll want to use a workflow orchestrator. Under the hood, they all work more or less the same way. So there's, you know, a set of message queues, and so I think we've talked about Pub/Sub and these things in the past, but basically, you know, you need a way to submit jobs, and the jobs need a way to read your submission. And so there needs to be some kind of queue to handle that. If the job is multi-step, then when the first step is done, now you know a new thing needs to be pushed onto this message queue saying, 'Okay, we're ready for the second step,' right? 1:04:06.147 --> 1:04:39.897 A job can fail. You know, like AWS could shut down that machine, the hard drive could fail on that machine, etc., etc. And so message queues are important for—or there's a lot of logic around. Basically, you send a keep-alive to the queue while you're still alive, and if the queue doesn't get a keep-alive for some amount of time you specify—let's just say 10 minutes—it assumes that you're dead and it will rerun. Like it will ask another machine to go and process that. Okay? So so message queues are a big 1:04:43.103 --> 1:05:32.940 part of it, and containerization, another really big part of it because you know you need a way to define some task on your computer but then ultimately send that task out to the cloud, and there might be hundreds, maybe thousands, tens of thousands of machines kind of running your task. So it's not practical to like `sudo apt-get install` some package all those times. So there needs to be some type of containerization, and and then also the workflow—so the thing that you know tells you the overall system what tasks need to be run and which ones depend on which other ones—that also needs to be put in the cloud somewhere and containerized. And so that's the second ingredient, and the third and final ingredient is some type of worker pool. 1:05:33.660 --> 1:05:35.179 So some type of uh 1:05:35.736 --> 1:05:47.180 you know, a pool of machines that are ready to do work. And you typically pair this with some type of auto-scaling so you have especially if you have bursty work. So imagine um 1:05:48.700 --> 1:06:09.486 Well, YouTube is so such a multinational thing that this wouldn't happen, but imagine if you were building a version of YouTube and you're just getting started where it's maybe way more popular in the US. You know, you'd want to scale those nodes down when it's the wee hours in our time zones and then scale them up when it's peak hours. Yeah. 1:06:09.705 --> 1:07:39.008 I think that becomes important too for, you know, something going viral. So if your website goes viral, you don't—you want to capture the signups, you want to capture whatever, right? But I think in that case, if the things people are signing up or doing or kicking off these workflows, I wanted to circle back to, you know, talking about failures, but I think there's a couple of things maybe that are less obvious or maybe important about these. So one is, you were mentioning, you know, something like YouTube size, but when you're kind of small, things generally fail because you did something wrong or there's a bug or a problem. But when you reach a certain size—even if you know, if you look at what I guess is called mean time between failure, like you said, a hard drive or cosmic ray or whatever it may be—but when you're running a million, you know, jobs a day, what is the chance that at least one job is going to have an issue in a day? It's non-trivial, right? It's noticeable. And so you're getting these failures because someone is walking into the server room and pulling out a server rack and like blowing the dust out of it for maintenance or whatever, right? And you can either have that be very carefully planned or just say, 'Look, it's going to get handled.' And I think kind of that is the same vein as you see stuff like—I think Netflix was famous for doing Chaos Monkey, something that goes in and just basically kills off nodes or drops nodes to make sure, you know, nothing nothing breaks. It just randomly sows chaos. I think that was. 1:07:40.054 --> 1:07:41.980 That was my job when I was there. 1:07:46.700 --> 1:07:47.980 That is my job in my family. 1:07:52.060 --> 1:09:04.379 Oh, Dad's here. Um okay, wow, I've been distracted today. But the other thing is, you know, we're talking about, you know, and I think we'll get to in a minute, but something like training and LLM. And it's, you know, ten million dollars—some of these workflows could be very, very, very expensive. And like you said, imagine there's some error, but imagine the error occurs at like a very late stage. You don't want to start all the way again from the beginning. So being able to kind of it's not exactly a hot swap, but being able to resume the pipeline with a new version of like a later node or even you just maybe it's not an error, maybe the whole thing finishes, but then you realize, 'Oh, here in this last stage or this last bit, we could do better,' or 'We could do something different.' So having a system that automatically is like preserving those stages and inputs, and and you know that kind of stuff so you can sort of rerun later parts becomes like super useful. And tying that all with the dashboard and management, which is things I have seen people not use—one of these—and then you end up just basically reinventing them. So you can either like not use them or you can reinvent one with half the features that you like do yourself and you just slowly try to re-imbuild a crappier version of like an existing solution. Yeah. 1:09:05.003 --> 1:11:04.620 Totally totally agree. Yeah, I think the part you mentioned about history and backfill is super important. Like there have been so many times where exactly what you said happens. You know, the job runs successfully, but it wasn't actually a success. Like we actually went the past three days with no artifacts because, you know, maybe the upload failed, but it didn't let the job know that something was wrong. So for example, like a very common pattern is where you catch an error in Python or in any language. You catch the error and then log an error message, and that's great and all. But because you caught the error and you didn't bubble it all the way up, now your job thinks it finished successfully. Like if you didn't set the exit code correctly on your program, and so you know that can happen. And then you find out about it even if you find out about it an hour later. If your website's popular enough, that could be millions of jobs that have run in that hour that need to be rerun. So having all of that ledger and then being able to backfill and say, 'Okay, you know, all the jobs from three days ago to now, I need to rerun all of them,' and it just goes through and does that and auto-scales to handle it. That's—that's and there's sort of this kind of like, you know, how cars have the what's it? There's basically like a backup system. So imagine so like on your HUD, imagine your HUD just like disappears. Like the OS powering the heads-up display on your car fails. It's got some kind of backup where you'll at least see like the miles per hour or something. But now that—yeah, right. But but that contingency—that's where I was thinking. There's a contingent operating system in most of these cars, but now that contingent operating system that really needs to not fail. 1:11:04.620 --> 1:11:26.804 Right, because now you're really hosed. And so similarly here, you know, the workflow orchestrator itself, if it starts failing, that can be a really difficult thing to survive. Right? Um and so to your point, you know, I've seen it as well where people try to build their own, and it happens so innocently, you know, the way it happens. 1:11:29.622 --> 1:12:18.188 Is um, you know, you're uh you have some tasks, and this happens at big companies, you know. So it's not a small company thing specifically, like you have some task that's relatively minor, and so you have this script that does it, and uh, you know, that starts to succeed. So it's like, well, you know, I'm just gonna make a Cron job and have it run this Python script every hour. And then it's like, oh, okay, now I need to run it, you know, every minute. Oh, now I need like three machines, so I'll just spin up three C2 instances all running this Python script every minute, and you kind of, like at some point, you have to just eat the technical debt and, like, set up a workflow orchestrator. But I've seen too many cases where people kind of go too long, and then there when there is an issue, it becomes very difficult, I think. 1:12:18.407 --> 1:13:00.156 The other term you'll hear with a lot of these is DAG, which isn't unique to these, but that all of these jobs form a Directed Acyclic Graph. Like no cycle graph, so they're not looping. So they have a start, they process some number of jobs with some interdependencies, and then they reach a conclusion. And so you'll hear them talked about that way. Maybe help me there, Jason? So, and not to throw a wrinkle here, but so workflow orchestration versus something that is more seen as like I would call like a pipeline, like Spark or something—what's the sort of distinction, you know, between, or where's a line, or is it blurry? Like what's the difference between something that would be viewed as more like a pipeline versus workflow orchestration? Yeah? 1:13:00.156 --> 1:14:55.180 That's a great question. Um I don't have an easy answer, but basically I would say okay, one thing: a workflow orchestrator cannot do is it cannot fuse nodes together. So you kind of explicitly define like 'do this task,' 'do that task.' Is that task? And they all depend on each other, or in a sequence or something. And uh if tasks like B and C really could be fused into one operation, it won't do that because it makes no assumption about what programming language or what version of Ubuntu is running on that node or anything like that. They're all just basically Docker containers that are getting spun up. Um you can actually um depending on the orchestrator you use, some orchestrators have it where um you can spin up a container for a particular node and that container can do multiple uh tasks. So let's say I have 10 videos I have to transcode all of them to QuickTime. I have a QuickTime node, and that node could do all 10 of them, right? But by default, most of these orchestrators you actually spin up a Docker container, do that one thing, that one time, and then tear it back down. And so there's a pretty heavy spin-up, spin-down time for uh for tests um co conversely with Spark. Like you kind of create—you do create sort of a DAG, and same with the early versions of PyTorch or if you do PyTorch Compile or you know early versions of TensorFlow, now you would create a DAG of operations, and then what the Spark compiler or PyTorch Compile or TensorFlow will do is it will fuse operations and it will create this like super optimized DAG. And it does that because 1:14:55.660 --> 1:15:12.726 all of the operations are within its own ecosystem. So like you know Spark has a limited vocabulary as does PyTorch. Um and so this being so open-ended, it can just run any Docker container with any arguments you want. Um that open-endedness comes at a cost. All 1:15:13.030 --> 1:15:13.992 Right? Yeah, that. 1:15:14.852 --> 1:17:14.299 Makes sense. Cool. Um so uh yeah, quick rundown on the steps to use a workflow orchestrator. Um typically you containerize your tasks. Um that's usually, you know, pretty straightforward; most people are using Docker anyways. Um but you also have to containerize the workflow itself. So the file that says, 'you know, run this task and then uh run the second task afterwards,' like that—that logic has to go somewhere. Um and so all of these examples that we'll talk about in the future, they all do it differently. Um and then after that, you're kind of off to the races. You submit jobs. Um one thing that is um that varies a lot from from orchestrator to orchestrator is how to pass data among the tasks. So um so the early versions of these, the only way you could pass information was through arguments. So uh so you know Job 1 finishes and then it's allowed to inject some command line arguments into Job 2. But that's it. And because it's a command line argument, you can't stick like a whole JSON blob in there, a large JSON blob or anything like that. You couldn't put an image in there or something. And so what you have to do for these early ones is um use something else to store everything. So you know the task transcodes the video and then it doesn't pass the transcoded video onto the next task, which may be like sets a flag in a database saying it's ready. Uh but what it does, it puts the transcoded video into some object store like S3 or something else, and then all it passes to the next task is maybe a unique ID of the video. Um Later versions um uh they allow you to pass 1:17:14.300 --> 1:17:59.282 arbitrary data, but under the hood they're doing the same thing. So when you pass like a Pandas DataFrame or an image or something from one task to the other, under the hood it's actually storing it in S3 in some temporary bucket which has like a low retention policy and then pulling it back out for the next um so um but yeah, that's that's all there is to it. So it's really not that hard. I would say this is in the category of things that kind of like functional programming languages where it's kind of a unique paradigm. If you've never used a workflow orchestrator, it can look very daunting, but once you've learned one, now you kind of can do pretty much all of them pretty easily. I 1:17:59.519 --> 1:18:05.037 haven't used one, but I I'm looking forward. A lot of these names are familiar, so. 1:18:05.037 --> 1:18:10.774 Uh wait, really? You haven't? Uh well what about like a proprietary one or something? I'm sure you Oh yeah uh 1:18:10.842 --> 1:18:15.617 I guess that's true, but I feel like it's a bit different. Like. 1:18:16.022 --> 1:18:16.849 If someone else already 1:18:16.849 --> 1:18:38.220 has like all of this setup. I've sort of been a part of one of them, I guess. So other people set up this stuff and I you know define some portion of some code that needs to run, and someone is sucking it up into a container and running it. Yeah, that way. But I've never sat down and had to write in the language of describing the workflow as an example that makes sense. 1:18:38.220 --> 1:19:28.551 Yeah, at Meta we had something called DataStorm, which is very similar to the ones we'll talk about next. But I never worked on DataStorm; I was mainly a customer. But they all are extremely similar, and I think a lot of these cases, like DataStorms—an example—there was something called Flume. Flume, which was similar to Spark, kind of like a combination of Spark and a workflow orchestrator. And then that someone made an open-source version of that. But yeah, I think most people will encounter workflow orchestrators; it might just be a homegrown one instead of one of these ones. 1:19:29.024 --> 1:19:29.783 Yeah, that's. 1:19:30.239 --> 1:21:28.540 True. But the big—the big market share ones are you know Airflow. Airflow is a patchy project. I don't know the history of Airflow, but it's kind of the 900-pound gorilla in the room. So if you're on AWS, there's something called MWAA, which is managed—I don't know what the W is, but the A's are Apache Airflow. And basically, you don't even have to set up Airflow yourself on any computer or anything like that; they just do everything for you. It's kind of like RDS for databases. So RDS, example, like you don't have to like install Postgres on a machine yourself or anything like that. They have the same thing for Airflow. Airflow is, you know, legacy. It's doesn't have a lot of the features of the other ones we'll talk about, but you know, you can feel confident that Airflow can do almost anything. Like if you can't get it done in Airflow, what that means is of the thousands and thousands of developers, nobody's tried to do what you're trying to do, which is maybe a sign that you're doing something wrong. It's kind of in that category where too big to fail, I guess. But but it is legacy. I think it's originally in Java, and because Airflow is so open-ended, you know, it's not hooked into any one language, that well, I would say is my biggest detractor of Airflow. The some other examples—I used Dagster a lot. Dagster is quite nice. I think it's Python only or if it's not. There's maybe a REST API that would be pretty ugly to use, but it's really meant for Python. It's nice; it gives you little decorators so you can decorate functions as Dagster nodes, and so when a function calls another. 1:21:29.340 --> 1:21:50.959 If if it calls a Dagster node as a you know function, it'll return back a Future, and Dagster kind of handles all of that for you. So it kind of pauses the operation of your function, calls that dependent task, and then comes back and gives you your function back, which is pretty clever, pretty cool. 1:21:52.950 --> 1:21:59.660 um so temporal is uh they actually just raised i think a series b i'm gonna 1:22:02.502 --> 1:23:16.540 double check um yeah they raised 2.5 oh 2.5 billion is their valuation a series c they raised um and uh it's pretty cool so it's it's it's kind of a different take you don't have the dashboard you don't have a lot of the things we talked about that are important um but it just gives you the bare thing that really matters which is uh being able to run a function and uh know you're going to get an answer regardless of how long that function takes so so temporal would be a good thing if you're building your own workflow orchestrator i think temporal would be a good thing to build it on top of and it might even be that daxter uses temporal it's very possible um but it is kind of like lower level um you know really calling individual functions uh durably um similar to temporal uh there's uh apache ray so ray comes from this uh company called any scale um ray is also pretty low level so you're calling individual functions and um uh it has to be kind of in python and so it's it's hooking in at 1:23:16.650 --> 1:23:21.038 pretty low level kind of like temporal um the thing that's really 1:23:21.038 --> 1:25:07.148 powerful about ray is just how unbelievably fast it is so with ray you um spin up all the machines in advance so you know all these other workflow orchestrators you know you're spinning up a machine doing some work and then spinning it back down with ray you spin up a whole cluster when that whole cluster is up and running you then execute a bunch of functions and those functions get farmed out to that cluster you span up for that particular task you're doing um and so it's a fully connected mesh of machines um and then when you're finished your task that entire cluster gets spun down so you're paying a bigger hit up front but you're getting like super super low latency like you never have to worry once the whole cluster is up you never have to worry about any node requiring another machine or anything like that um and the final example i have which folks should know about is cube flow um cube flow is similar in a sense to air flow um the big difference is they've integrated a dashboard where you can um uh you can upload different statistics so for example you know air flow is a dashboard where you can monitor the jobs and see oh i have five failed jobs yesterday cube flow has that too but cube flow allows you to like each run have a dashboard for that specific run and so it's like okay task three you know output a time series graph and here it is task four created a scatter plot and there it is um and so for workflows that are producing statistical outputs that's like a nice feature tab it sounds like 1:25:07.401 --> 1:25:17.580 some of them are like you said sort of lower level like rays sort of sounds like it's short shorter not shorter you want higher you 1:25:19.028 --> 1:25:20.682 know response lower response times lower latency you 1:25:20.682 --> 1:25:33.500 want to be able to do lots and lots and lots of things you want to sort of shard them all up and have them sort of distributed and then other things like you were sort of saying like transcoding a video or something you pretty much want to like 1:25:35.042 --> 1:26:08.050 Do a bunch of data reading, do a bunch of processing, kind of do a bunch of data writing, you know, as far as like externally. So you're sort of dealing with large blob objects and very long run times, and so it feels—it feels like there isn't a generic solution that just sort of like meshes over all of it, which I guess kind of makes sense. And as you were sort of saying, I think whether one sort of container can be multi-purpose or whether you really are needing different containers for all of the different jobs probably plays a huge role in sort of like how easy or hard it 1:26:08.522 --> 1:26:57.339 Is. Yeah, yeah, totally. It's one of these things that like the lines are a little blurry because I'm sure—I'm sure someone's going to email us saying, 'Oh, you could do actually this with Airflow or you know Ray could also do that.' But just generally speaking, like at the baseline level, you know Ray is not durable. So if your Ray job, uh, you know, you're responsible for doing checkpointing and things like that, you know, if your Ray job dies halfway through, it's just gone. You know whatever checkpoint you have is—whatever you created. Um, you know these other ones, you know, they'll—they'll just by their nature, they're constantly creating checkpoints. And so yeah, there's basically several things you're juggling, you know, how durable do you want these tasks to be? Um, these parts of your workflow, and then what's your latency requirement? 1:27:00.025 --> 1:28:58.460 Um, and then what—what's your sort of cost you're willing to spend? Because Ray, you know, for every so just to give an example, let's say you have some type of task and that task has a list of Python third-party libraries that need to be run, need to be included. So if you do Airflow, um, you know when that task node spins up, it spins up your Docker container where you've already put those libraries in it, right? And it's ready to go and it executes your code, and that's fine. Um, with Ray, you create a Docker container with those dependencies similar to with Airflow, but you spin up an entire cluster and every node on that cluster has all those Python dependencies. Um, you know, so if you have especially like heterogeneous things where maybe you have one task that requires Torch, which is like a two gigabyte PyPI file, right? Um, and then you have another task that's just send an email, you know, that only requires like the base Ubuntu. Um, you know, then Airflow is going to save you a little bit there. So yeah, there's no free lunch; there's like kind of several things to juggle. Um, I guess so some advice. Um, I think Ray is—you should only use Ray if latency is important. So if you're doing ML training, latency is very important. You have this GPU, it's very expensive. You don't want to waste time on it. So that's why Ray is extremely popular for uh training these AI models. Um, if you're doing ML stuff or things where you're producing a lot of um, you know, graphs and a lot of those artifacts, I think CubeFlow could be a good choice. Um, and for everything else, I would use Dagster. If what you're doing is in Python, otherwise I would use Airflow. So that's sort of the workflow diagram of which workflow orchestrator to pick. 1:28:59.567 --> 1:29:05.260 Yeah, yeah, okay. He's just all prescribed out for us, I guess this is called a flow chart, but yeah, okay. 1:29:05.355 --> 1:29:47.260 Oh, that's true. Cool. Um, yeah, I guess um you know some like things to avoid other than rolling your own orchestrator is um I think you can go the other direction and kind of make things super complicated with, you know, auto-scaling and a lot of these other things. I think you know you can take baby steps with these things just like anything. You can kind of start getting into it and end up diving super super deep and saying like, 'Oh, I'm gonna preload the nodes with these packages.' And at the end of the day, like, you know this—you want this to save you time; you don't want this stuff to cost you a lot of time. 1:29:47.695 --> 1:29:49.230 Makes sense. Thanks for the 1:29:49.551 --> 1:29:50.310 Overview. Yeah. 1:29:50.479 --> 1:29:52.639 Um, yeah, the education. Yeah, this is 1:29:54.040 --> 1:30:53.794 a good thing to learn, folks. I think this is super super useful. Um, if you are like looking to get started, imagine you have like a—you have a website that's um uh that has any type of job that needs to get done. Trying to think of a good example. Um, like like you have a website that just OCRs pictures. Someone puts an upload, drags a picture into your website and you send them back some text or something in an email. And nowadays OCR is so bloody fast, like you can just respond instantly. But let's pretend like we're still where that took a long time. Um, you just spin up Temporal. You know it's I think it's uh relatively easy to stand up. I think they have a managed service too that's very reasonably priced. And uh they look like I'm gonna call this function to do the OCR and that function is going to do the OCR and send the email, and I'm just gonna wrap it in Temporal, and now it's durable, and that could 1:30:54.486 --> 1:30:56.039 be just an easy way to get started. 1:30:58.924 --> 1:31:33.020 Cool. All right. Um, yeah, we hope you folks uh liked the episode and uh definitely keep sending us, you know, requests for new content, new new show ideas. Uh, we take them super seriously. We've implemented many of them, so our list is getting smaller. So if you have a show topic, please Slack us or Discord us—I mean, email us however you want to reach out. To thank you everyone. 1:31:35.020 --> 1:31:39.020 So. 1:31:39.020 --> 1:31:41.260 Music by Eric Barndollar. 1:31:42.940 --> 1:32:01.420 programming throwdown is distributed under a creative commons attribution share alike 2.0 license you're free to share copy distribute transmit the work to remix adapt the work but you must provide attribution uh to uh patrick and i and uh share alike in kind