WEBVTT 0:00:15.083 --> 0:00:20.955 Hey everybody, so this episode, 0:00:21.360 --> 0:02:19.820 we're going to cover DevOps. For people who are new to the show, we're going to cover it somewhere in the middle. We'll talk about news and some other things first. That is the number one comment we get is, 'Why did you take so long to talk about DevOps?' You're going to learn a whole bunch of cool stuff today. If you're a brand new listener, welcome to the show. So I wanted to kick it off with a project that I built, which I'm pretty proud of. You probably have this feeling, whether you're doing woodworking or stuff in the backyard or robotics or anything, you have a project and you say to yourself, 'This is a coin toss,' whether this is going to work or not. And it's kind of exciting. It's kind of like a rush because of that sort of randomness. And then it really is a coin toss. Half of them don't work and just decay into dust. Half the projects work really well. This is one that I feel like works pretty well. I created a LinkedIn social media bot, but it doesn't do what you think. Most of these bots go out and harass people or maybe that's not the right word, solicit people. Mine is the opposite. So mine is a defender. It defends me against the other bots. And so the way this works is it uses a large language model. So I'm using ChatGPT. It has like in the cookies file, uses a headless browser with the right cookies set up. So it's logged into my LinkedIn account, goes on LinkedIn, accepts any connection requests, and then goes to my messages, looks at the unread messages. And I ask ChatGPT, 'You know, is this a solicitation? Like, is this someone trying to sell me engineering services?' And I tell ChatGPT, 'Answer yes or no, followed by an explanation.' 0:02:19.820 --> 0:04:05.740 And the next one is, you know, is this somebody who is looking to work at my place of work? And based on these answers, it sends automatic responses. And so I actually, in the response, I tell the person, 'Hey, this is an automatic response. Your message was classified as this for this reason.' And I paste the ChatGPT answer. And then I mute the thread and put it in the other category. And this has worked amazing. So first off, after some testing, I kicked it off. I got banned three times from LinkedIn. Because, not banned, but like throttled, you know? Or like, literally, if from my desktop, I went to LinkedIn.com, I just got like a HTTP timeout or something. So like, they throttled me completely. But eventually, it went through my entire backlog of unread messages. And the other thing it does is, you know, if it doesn't catch this as a solicitation, it marks it as safe. And so I have a safe list. And so when I was done, I looked at, because I had thousands of unread messages. And it was almost all spam, right? But then when I was done, I looked. And there was all these people who were legitimate people who had interesting things to say who I now could see. It's like, basically, like I had a handful of needles, you know? Like the haystack was burnt to the crisp. And what I was left with is all the needles. And so some of the people I reached out to were surprised. One person was upset that it took me eight years to respond. Whoa. 0:04:07.620 --> 0:04:50.112 But a lot of people, it was like I built a lot of connections with people who just, you know, serendipitously, like, or just, you know, messaged me on LinkedIn. And I just couldn't see it. So it was an amazing experience. And so now what I did is created a cron job that runs on my Raspberry Pi, which runs the same process every hour. And, for example, at 4 a.m. and at 9 a.m. today, it sent automatic messages and muted to people who are trying to sell me software services. So that's just today, this morning. So it's pretty awesome. It's kind of weird, but it's 0:04:50.112 --> 0:05:06.580 pretty cool. I think the idea is interesting. I mean, I guess the incentives in the system are a bit weird because LinkedIn doesn't really have an incentive to stop the spam as long as if, like, it doesn't overwhelm and, like, mess up user retention. 0:05:08.100 --> 0:05:34.940 So, like, some people may, like, it's also a personal thing. Like, some of the stuff you might consider spam. Other people might consider, like, if you're searching for a job, some of those people reaching out may be like, 'Oh, I want to reply to them.' Like, I'm trying to do. Right? And so, yeah, I mean, getting throttled, it makes sense. It's very hard to distinguish between someone doing, like, what you're doing and someone scraping a bunch of hacked LinkedIn accounts that, you know, trying to get everyone's personal data. 0:05:34.940 --> 0:06:36.020 Yeah, I mean, I was definitely nervous when I got throttled that, like, I had inadvertently got myself kicked off LinkedIn. But, yeah, so just to double down on that, I have friends at LinkedIn and, you know, you guys totally did the right thing by throttling me. I don't feel like that was the wrong. I'm glad that it didn't last forever. But, oh, yeah, you hit on something really important, which is, you know, there's this spam, like, 'Hey, I'm in Nigeria. I have a billion dollars. I just need to accept your credit card.' And then there's what you said, which is, like, 'I think solicitations' is the better word. And, in fact, like, even this ChatGPT thing, the very first thing I ask is, 'Is this a person looking to offer me a job?' And if it is, then I mark them as safe. So, like, clearly, like, there's some solicitations I'm interested in and then others I'm not. And I don't expect, like, LinkedIn to mark those as spam, right? Or have that nuance. 0:06:36.739 --> 0:07:14.460 Yeah, so I guess, like, that's one of those, you know, I guess you call it fine-tuning or whatever. Like, at the limit, being able to describe your preferences in a maybe just a, you know, chain of consciousness style, like, way you're thinking about it and having it tuned to those preferences. Like, 'I am interested in jobs that are legitimate,' but I'm not interested in job offers that are clearly just blasts and have none of my personal information in them. Like, the person didn't bother to do research.' And you kind of list off all the things you're interested in and kind of build the, yeah, like you said, like, it's a, you can do a lot of probably what you're doing with Natural Language Filters. Or just, like, going in and doing basic stuff. It just, you wouldn't take the time. It's not worth it. 0:07:15.040 --> 0:08:25.280 Right, right. Like, for me to build some NLP thing, it would take me a long time versus this only took me maybe three, four hours. And, you know, the false alarm rate is not the best. You know, I actually, somebody texted me and I had muted them on LinkedIn and they actually knew me from a meetup that I run here in Austin and they sent me a text. So, yeah, there's definitely the false alarm rate is, let's say, let's say 5% or something, which is not great. But for an application like this, I think it's fine. I'm not landing a space shuttle or anything with this. But it definitely, when you go and look at all the people messaging you and you auto-messaging them and muting them, like, every now and then I'll go and look at it. It makes me think of what a weird world we live in where someone has, you know, a machine that messages me and my machine answers it. And it's all like natural language. Like, it looks like they messaged me, you know, manually and it looks like I responded to them manually, but it's all it's all just fake. 0:08:26.196 --> 0:09:16.740 I think I think that was on. Oh, I think it was Lex Fridman talking to Yann LeCun and LeCun was saying the same something similar, which is we always think that, 'Oh, there's going to be some propaganda campaign that, you know, somebody is going to use ChatGPT to just be super convincing with fake news.' He's like, 'What people miss is that if they can do that, you can too.' And so a lot of our email and correspondence at some point in the future, likely it was astonishing that we effectively ran open ports and like let everyone just send everything they want to us. And instead, we'll have secretaries that are trained on our behalf to like filter out that stuff, you know, and you'll just have your bots talking to their bots and some stuff we'll get through. But most of it, you know, will be sort of caught by intelligent filters on our side. And so, yeah, they'll develop together. 0:09:17.520 --> 0:10:02.300 Yeah. I mean, Scott Galloway from the NYU Business School had this talk that I watched where he was talking about Amazon having this unfair advantage, because if you search for just the word 'batteries' on Amazon.com, you got their batteries first. And how that's, you know, just a huge advantage for them. I don't think he was going down the, you know, it's not fair route. I think he's more just saying what a tremendous advantage it is for them. And so you can imagine a future where when you search for something on Amazon, there's some browser plugin that re-ranks it, you know, that has your interests. And so now like your ranking kind of fights with the server's ranking. 0:10:02.300 --> 0:10:45.640 Yeah, it's an arms race. I mean, it sort of sounds appealing to engage with social media in a way where it's like, 'I want something to go through and curate stuff on my behalf that works for me and incentivized by me,' rather than the social media companies trying to get engagement or push just absolute crap to me or try to get me to stay on longer to find the nuggets. Right? Like they're incentivized, even if they 100% know the things I'm interested in, they're incentivized to spread them out at like a specific repetition interval so that I spend more time, see more ads. And so they keep showing me other stuff that's maybe just close enough that I don't click away or whatever. And so to have somebody go through and filter all that out, it ruins all the advertising models, of course. But it does sound an interesting way to consume media. 0:10:46.280 --> 0:11:15.040 Yeah, yeah, totally. Totally. So, yeah, folks out there, if you want to connect with me on LinkedIn, I will accept everybody. And if you well, I will literally accept. Well, I won't accept it, but my AI will accept it. Convince Jason's AI bot and he'll share with you. Yeah. If you message me on LinkedIn about your software development company in Eastern Europe and how they can do software cheaper than I can or whatever it is, you will get blocked, and it will be pretty funny. 0:11:17.120 --> 0:11:51.079 All right. On to one last thing about that: the code is also totally available on GitHub. Just look me up on GitHub. Aren't them a Jason Gouchy's on my most recent repository. You can get all the code and run it yourself. All right. On to news, news. My first news. So when I was building this LinkedIn autoresponder, I used Amazon Q, which is, you know, one of these Copilot type things. Patrick, have you ever used GitHub Copilot or Amazon Q or any of these? 0:11:51.280 --> 0:12:01.032 No, I haven't. I mean, my work, it's sort of a band. And I, as I've admitted many times on here, they don't do as much coding as a hobby as I should. So, 0:12:01.032 --> 0:12:11.123 It's also banned in my work. So this is my first opportunity to use it. You know, band isn't the right word. It's more. No, yeah, exactly. It's that hasn't been considered. Maybe. Oh, 0:12:11.123 --> 0:12:18.739 No. For us, it's like, yeah, you don't want to upload your code to somewhere right in order to get comments on it. 0:12:19.180 --> 0:12:22.819 Oh, I don't know if this one even does that, though. Oh, well, or does it just run? 0:12:23.060 --> 0:12:26.024 Most of them do, or at least have clauses that they could upload. 0:12:26.024 --> 0:13:20.327 Oh, you know, that's something I didn't even think about. I just assumed that it was running on my desktop. But you're probably right, actually. OK, at any rate. So I tried, I tried this out. It was awesome. So I'll give you some examples, like really concrete examples of things I found really impressive. So and these are all going to be tiny. So this is part of it is, you know, the whole like build me an entire website ChatGPT. I've been extremely skeptical of that for a whole bunch of different reasons. The biggest one is that projects are many modules that you want to have, like some thematic consistency. Right? So I kind of went in pretty skeptical, thinking this is not really going to help. But one thing that it did was, you know, I had a variable called finished and, you know, I had while while not finished. And so then later on inside that while 0:13:21.880 --> 0:14:51.920 loop, I put I think I put finished equals. I get this right. Oh, no, sorry. I had while true and I had a variable called finished. And then I typed if finished and then it auto-completed, you know, colon new line break. But again, like not not like a big deal, but like it kind of figured out like, OK, the variable is called finished. It's inside a while true loop. So if it's finished, you want to just punch out of the loop. Right? So I was using this library called Playwright to do the web automation. And when I went to run a function, it figured out that I should set this parameter. There was like a wait time parameter and it looked further up and figured out, oh, I'd set this wait time parameter before. And so I'm going to want to set it again. And so right when I put the open parentheses, it was like wait time equals two seconds or a constant, whatever it was. So it was impressive. I actually was really, really surprised. I was very skeptical. I was expecting to uninstall it right away. There were a couple of times where it wanted to auto-generate like 30 lines of garbage, like literally just garbage characters. So it's not perfect by any means. But I would say it's a net productivity gain. You know, I was really surprised. 0:14:52.680 --> 0:15:03.120 And you feed it your context of your entire project in advance or it's doing this from like common available code that it knows about. 0:15:03.380 --> 0:15:33.340 I have no idea. I mean, basically, I'll tell you what I did as a user. I installed the extension. It makes you create an AWS Builder account, which is separate from anything Amazon or AWS accounts you have. So I had to create a new account. And now it's under the hood. It could have uploaded my entire project to the Internet. I really have no. I mean, it was an open source project anyways. I have to lose. But yeah, I really don't know what it was doing under the hood, but it was it was really clever. 0:15:33.939 --> 0:16:16.700 I look forward to this. I mean, I think this would be one of those things where it used to be. I remember that was a long time ago. Google used to sell search appliances. And, you know, for the same thing, you would put them on your in your, you know, infrastructure internal on your network at a company and it would index all of your internal websites and web pages. So you go to like an internal Google. Now, a lot of that, you know, has been integrated in other tooling and other solutions. And so we don't really think about that anymore. But I think something similar will happen here where you'll have on-premise boxes initially, because it's the only way to really guarantee, I guess, it's sort of safe for AWS instances, maybe. And you'll upload company, you know, your repositories and have this stuff crunch on it. 0:16:17.280 --> 0:16:30.620 Yeah, yeah, that makes a ton of sense. I know there's a bunch of open source versions of this that you can run locally or run in your own cloud. So I know the technology is there. It's just a matter of time. Very cool. 0:16:31.060 --> 0:17:09.659 Mine's on a pretty opposite end of the spectrum, I guess. And that is I've seen a series of these. So I just want to highlight I picked one to focus on, but there's actually two, which is there's a series of very cheap RISC-V or Risk Five. I don't think I don't remember. RISC-V. Yeah, RISC-V. Okay, that's what I was going to say. The RISC-V processors are starting to kind of roll out. And also I saw like commodity prices. So I think they're like 10 cents roughly for one of these processors. You haven't. I don't have any yet. It's on my yes. I have them in my cart on AliExpress, but they're kind of expensive. So I need to find another place to buy them from. Um, and 0:17:09.659 --> 0:17:18.780 Yeah, I found them very expensive as well. I was looking at the Mars something and it was like $200. I was like, I can get a Raspberry Pi for that. 0:17:19.300 --> 0:19:18.439 So, so, okay. So maybe this, these, these, these people are out to help us. So a YouTuber by the name of Bitlooney, B-I-T-L-U-N-I, if you've never seen him before, lots of interesting computer related projects there. He built a super cluster of 256 of these with some interesting discussions. If it's not something you've ever considered before, but if you want to have 256 processors talking to each other, how do you handle that as like putting them on a bus? And what is the communications matrix look like? And how do you organize it and think about it? And then, you know, how do you tell if they're all working or not? And he's not the, you know, of course other people have built, you know, very high core count even production processors before, but at a hobbyist level, it's kind of nuts. You would never have spent whatever it is, you know, $5 for a processor and then 256 of them, but at 10 cents each and PCB prices have come down to sort of make your own boards. And so it's just interesting time to kind of consider, consider some of these. And the nice thing about these, which I think there's. A pair of them that are pretty similar CH32V. I think one is 003 and the other is 203. So CH32V 203. These are the chips. But often what happens is they, these chips are very, very cheap. And if you get help from the company and you need to put them in like your hair dryer to display, you know, to control the temperature output or something, you could buy one of these. But if you want to do hobbyist stuff with it, getting an SDK, getting documentation is pretty hard. So you need a sort of critical mass of people to get, you know, good English translations and SDK. Okay. And the very, very popular I guess, example of this is if you hear people talking about, Oh, I just thought that STM32. What are the little WiFi chips? You were using them, Jason. Oh, why is my Pico Raspberry Pi Pico. No, now my brain failed me. 0:19:18.719 --> 0:21:17.780 Uh, this is a bad, yes. P32. Thank you. Thank you. I was going to say STM32, but yes, P32. Those are two different things. The ESP32 was similar. It was like a very cheap commodity thing. There was like going in light bulbs and it wasn't useful. And then hobbyists sort of realized, wait, this is actually, you know, we can get good documentation for it. And now there's a huge community of people developing things for ESP32 and other chips. And I don't know for the company if it made an impact or not, but for hobbyists, it's certainly cool to get access to, you know, these really cheap things from this Expressive, I believe is that company. So now we're starting to see the same thing, debug tools, compiler toolchains, for these RISC-V processors. And they're ridiculously cheap in part because they don't have to pay a licensing fee for their RISC-V architecture itself. And so you can get them even cheaper. Not that everyone always pays licensing fees, but for sure now, like they definitely can't. And then, in the show notes, I don't have a great way to do it, but if you search powering a Nixie tube from a RISC-V chip, you'll find there is a link here to a YouTuber who I hadn't run across before. But they built a like a boost buck boost controller from a microprocessor. So actually taking USB sort of five volt power and going, I think it was 200 volts to power a Nixie tube, which is a little like vacuum tube, except it displays numerals instead of like doing a transistor. I think it's, Oh, I did a CNL OHR look in the show notes, or search powering a Nixie tube from USB with a 10 cent RISC-V processor. And so they, I guess we're doing something at a not coordinating with the other person Bitlooney, but they both end up having something. They have a bunch of cool stuff on their channel as well. So worth checking out. So adding this definitely to my list of like things that would be cool to play with. And as Jason has told us on the show before, you know, there is a risk of letting out the magic smoke. 0:21:18.100 --> 0:21:24.080 So spending, you know, 10 cents on a processor makes letting out the magic smoke much less painful. 0:21:24.780 --> 0:22:03.400 Yeah. Yeah, totally. I feel like just personally at the level where like, I have to put the processor somewhere and put the RAM in another spot and the WiFi chip in our spot and hook them all together. That's like one level below where I'm interested. Like, that's why I was looking at the this Mars thing. I can't remember the exact one, but it was basically a Raspberry Pi where you have the WiFi of the Bluetooth, you have an Ethernet port, HDMI port, all that stuff, but it's RISC-V. And for some reason those are still really expensive, but I think it might be just that they're not mass producing them yet. 0:22:03.700 --> 0:22:20.159 Yeah. I think they'll come. I think the dev boards, I mean, I'm seeing dev boards in the sub-$5 range for some of these CH32s. I don't know what the proper abbreviation is, but for these, but they don't have WiFi and stuff like you're saying just yet. Got it. Yeah. 0:22:20.379 --> 0:24:19.260 That makes sense. All right. My next new story is by Three Vision release. So a bit of background here. Microsoft has been pushing a lot of things, but among them, these SML or no wait, SLM small language model. And so the idea is, you know, this is a model that I actually have one running right now on my phone and it's pretty good. You know, it runs in real time on my Samsung. It's not even really the latest model or anything. So small model, you know, it will give you some weird answers for sure. You know, it definitely doesn't have the depth of these other models, but for general queries, it works pretty well. I had something recently. Oh yeah. So there was some issue with Verizon. We ended up having to call Verizon, our cell phone company and get it sorted out. But for whatever reason, my entire family lost internet, which was actually kind of terrifying when you're on vacation. We're on vacation and we just, we still could make phone calls and texts and everything, but we lost the actual internet, you know, mobile data. And so I was, you know, downloading maps when I was on WiFi and all of that is a mess. But one of the things I did is I was asking this Five Three because it was local on my phone just for some general queries and it was giving me answers. So it's, it's amazing. And so now they released Five Three Vision, which is an actual like multimodal model that you can run locally pretty easily. So, people have done a lot of really interesting things with this. You can actually I have a friend who took a photo of his driver's license and asked Five Three Vision, Hey, pull out the birthday and the driver's license number and the person's 0:24:19.260 --> 0:25:57.860 name out of this photo. You know, and it did all of that and stole his identity. So it's but no, he ran it locally. And so, you know, it was totally safe. He knew that it wouldn't hit the cloud or anything like that with his driver's license. You know, he gave it a few other PDFs and asked it to, you know, extract information. And he said it was amazing. He said everything came out flawless. So, you know, I think this is just I think really something to pay attention to. There are better models like LLaVA, which is a multimodal model. But these require, you know, GPUs with 16 gigs RAM or maybe more than that. And so they're they're heavier models. It's really interesting to see like for an average person with an average desktop, laptop, or even a phone, what are we capable of doing here? This is really taking that to the next level. So, you can check it out. It's pretty easy to download. And there's something I'll put it in the show notes called Oh Llama, where it will basically hold your hand and you don't need to be a even an engineer or a coder to use these models. It gives you a like ChatGPT-like web interface, but for open source models. So this is a super low barrier of entry and I think there's a ton of potential here. So, if you're out there, you know, college, high school, just getting started, check this out. I think there's real powerful stuff you could build on this. 0:25:58.820 --> 0:27:39.560 Very nice. I haven't, I do have a couple of apps that will run open source. I think Hugging Face, there's an app. I don't know if it's an official app or not, but lets you from your phone hit those. Those aren't local though, but yeah, it is kind of cool to try the various ones and see various ways you do or don't that they can kind of fall over. But the multimodal stuff is cool to dovetail it right into my next one. It's talking about multimodal is OpenAI. Well, I mean, news all the time these days, controversies and new releases, but just focusing on a new release, release ChatGPT for Oh, which the O being Omni and sort of their take on this multi-modal. So doing, being able to use pictures or voice, the demos are really cool. Sometimes a bit hard to recreate the demos if you try it yourself, but in general, pretty exciting to say like, you know, Hey, I'm sitting here with someone who speaks Russian and only speak English. Like I'm going to speak English and I want you to translate it to Russian. They speak Russian, you know, just describing, you're like, Oh, I could do that with the translator app. It's like, yeah, but now you don't need to like, for arbitrary configurations, you know, it can kind of understand, but also some of the behind the scenes things that they're sort of saying that, you know, running this new version, even though it's improved, it's just like orders of magnitude cheaper. They're figuring out not really. I mean, you can kind of read the rumors about next version of ChatGPT Five, but it's kind of cool to see them get more and more efficient at running their current generation stuff sort of almost faster than you would expect. And it's just interesting to see how quickly things are moving and how the pace goes for training and running a lot of these models. 0:27:40.300 --> 0:28:11.020 Yeah, definitely. I'm a really big fan. I think, you know, they started opening. I started by doing reinforcement learning, which is obviously near and dear to my heart. I don't think, I don't know if they do much of that anymore, but still, I'm a big fan of the company. I think it's really cool. You know, there was complaints about them, like, I guess stealing somebody's voice or stealing like data from the internet. You know, they've stolen all of my code, but that's okay because I stole it from Stack Overflow. So I copy pasted it first. 0:28:12.580 --> 0:29:18.020 So how can I hold them responsible? Um, so yeah, I think, uh, well, I mean, that's a whole rabbit hole around, uh, data. And if you scrape Reddit, are you like, you know, trouble or whatever, but, uh, but either way, I mean, I think it's amazing. Yeah. This, this, uh, I think the four O the ChatGPT 4O stands for Omni because of the multimodal part. Um, and, uh, I think there's so many interesting ways you could take this. I mean, you know, one thing that just came to mind is, you know, you could run, Oh, circling back to what something you said earlier, uh, the app that lets you actually run the models on your phone is called MLC Chat. And, uh, you can get that. I think it's on iOS and Android. Um, but, uh, but yeah, imagine like you, you take a picture of like, uh, even just a restaurant receipt and it keeps track of, uh, you know, um, like, you know, what things you've eaten at restaurants. Um, I think there's a whole bunch of cool stuff you can do with this. Um, and we'll have to, uh, have to see, see where people take it. 0:29:19.000 --> 0:31:18.980 All right. Time for Book of the Show. My book of the show is not a book. Spoiler alert: my tool of the show is also not a tool. Book of the show is a podcast. This is a bit of a different podcast. And I don't want to give like a ton of disclaimers. I've been enjoying listening to it. Um, it's The My First Million Podcast. This is like two business folks who are kind of like doing marketing, but they have interviews with people who have sort of done startups or sold companies. And they talk a little bit about details about, you know, kind of exact dollars they were earning and how marketing goes. They were talking to someone recently who had a blog about gardening and then started selling products and turned it into. What I find interesting is it makes you think a lot about stuff that's pretty different but adjacent to the stuff we do. I think a lot of software engineers always have this question about entrepreneurial pursuits and like what the various trade-offs these folks are out talking to, you know, people who have done massive things, people who've done small things, and also just their observations. It's a very casual podcast. A bit long-winded. They also don't necessarily get to the topic of the podcast until late into the podcast. But, you know, they release twice a week, and it's a very casual listen. I've been enjoying it. I don't know that I'll listen to for always an ongoing, but it's been good to sort of listen to kind of the sales aspect, marketing, the kind of parts of the business that I don't normally focus on. Right? I'm always thinking about solving hard problems and machine learning models or whatever, right? Like those things are much more up my avenue. This is adjacent and people doing often software businesses, but talking about motivations and things related to personnel and how to relate to people. And anyways, personally, I've just been finding it a nice orthogonal set of views, I guess, to what I might normally pay attention to. 0:31:19.620 --> 0:31:28.379 Yeah, that's awesome. Yeah, I'll have to check this out. Jason Fried. That name sounds familiar. Is he the Basecamp person or is that somebody else? 0:31:28.840 --> 0:31:32.719 Yeah. He was one of the ones, him and DHH or whatever. 0:31:33.199 --> 0:33:32.360 Yeah. Okay. There we go. Yeah. Very cool. Yeah, I'm definitely going to check that out. It looks like fun. All right. So totally on the other end of the spectrum, my book is the one of the geekiest books I think we've had on the show. It's this research paper from Jan LeCun. That he wrote in 2022 called A Path Towards Autonomous Machine Intelligence. I have a link to the PDF in the show notes. It's not a; it's not a called an approved paper. It wasn't submitted to any conferences. It's just an open paper that's written on open review. And you can kind of tell there's some languages, you know, errors, grammatical errors, or some hand-wavy stuff. There aren't really any, actually, I think there's literally no results section. But sometimes these are the best papers, you know, it's just like raw thoughts of folks. And so this paper is really interesting. It tries to tie together these really large latent models, right? So, so Large Language Models are, and I think we're starting to use foundational models because they are multimodal, but these foundational models are forward models, which means they look ahead. So it says, you know, given I just wrote this part of a paragraph, what's the next word I should write, or what's the next handful of words I should write. And so you could extend that to really anything. You could say, well, given that I made this move in chess, what's the next move I should make, or given that I turned the steering wheel this way in this, in this, car, what should I do next with the car? So there's sort of like a theory of everything kind of thing going on here where, you know, the idea is you build this massive world model that tries to predict the future. 0:33:32.900 --> 0:35:31.740 And then you do a variety of different things like sampling and some gradient-based search and different methods to search through potential futures. So, I mean, maybe taking a step back. So you have the world you're in. Imagine you are a good problem we can use for this. Like just let's say chess, for example. What about okay, let's take chess where instead of you getting a chessboard the way a computer would expect it, you're getting pictures of a chessboard. Okay. And so you have to play chess from looking at pictures of a chessboard. There's a lot of things that really don't matter, right? Like if the pieces start casting a shadow in the opposite direction because there's a new light source, that's not relevant to the problem you're trying to solve. Right? And so if you look at ChatGPT, or ChatGPT is taking texts and trying to produce more text. So it really has to understand everything about text. But in this case, you're not really taking in a picture of a chessboard and trying to create a picture of the chessboard with your move on it. You're trying to take in a chessboard and make the move. Right? And so there's a lot of confounding information there like shadows, for example. So, so the idea here is you create like an embedding or like a low-level projection of the image. And then you try to predict the low-level prediction of the image with your move in it. So imagine I say, 'Here's a picture of a chessboard and I want you to give me a picture of the chessboard where the pawn has been moved up a square.' But instead of literally rendering that photo, which requires a lot 0:35:31.740 --> 0:37:29.400 of wasted time, you know, it creates sort of this low-dimensional representation and then a low-dimensional representation of the board with the pawn in the new spot. And then what you could do is you could physically move that pawn, take a picture, and then embed that picture and see if it worked. Right? So, so did the embedding of the future match the embedding of the present plus this function that's supposed to take it into the future. Right? And one thing that you'll quickly notice is like you can hack this or you could exploit this. You could say, 'Oh, well, my embedding is just the zero vector.' And all actions just turn the zero vector into the zero vector. And so, I embed to the zero vector. I moved the pawn. Now I have the zero vector. I embed the future, which is also the zero vector. And sure enough, the zero vector equals a zero vector. And it could seem like my embedding is amazing, but actually it's not doing anything useful. Right? And the way they get around that, which I thought was super clever, was, I'm going to see if I can explain this easily. They try to maximize the information of the embedding and the way they do it is, you know, if you have a picture, right? It's, and we've talked about this on the show in the past, like it's little RGB values for each pixel. Right? And so it's like a ton of these RGB values. And then when you crush it down into this embedding, now it's like a small number of numbers or dimensions, but it's trying to accomplish some goal. And the goal in his case is to do this forward thing we talked about, but also he wants the 0:37:29.400 --> 0:38:41.000 variance of the parameters to be as high as possible. So basically, trying to make fancy way of saying, trying to make the numbers vary a lot from one picture of a chessboard to another. So if you take a bunch of pictures of chessboards and they all collapse to the same embedding, you could assume that it's not doing a good job. Now it's possible that nothing has changed, but it's also possible that you're not doing a good job of capturing information. So, so yeah, that's just part of the paper. I mean, this is a 50-page report. So there's a lot of content here. The thing I like about this is you don't really need to know anything beyond like just algebra and some understanding of differential equations. So like grasp this, it doesn't refer to like a million other papers. That you then have to go read. So I think it's one of these things that folks who know DiffEq or even if you don't, you could read this paper and get a lot out of it. It's a good kind of starter paper, but there's a lot of content. 0:38:41.700 --> 0:39:06.120 And this is part of his thing. Like, I mean, I think he talked about this, but the contention that current machine learning stuff, like the LLMs can't really do planning. Like it can't sort of figure out how to do stuff. It can describe the text of a plan, but it itself doesn't do planning because it doesn't sort of move through the space like you're talking about. And this is related to that. 0:39:06.960 --> 0:39:53.260 Yeah, that's right. Yep. So, you know, he actually famously put on Twitter if you're in college right now, if you're getting a PhD right now, don't bother with LLMs because like the companies are all over it. He's like, do the thing after LLMs. Which I think is absolutely correct. Like this LinkedIn thing we talked about at the beginning of the show, like take an LLM and use it to do something and you will uncover like the next layer of problems. But yeah, I think it's interesting. Even in this paper, he's really casting a lot of shade on contrast of learning and LLMs and these things in favor of some of these other approaches that he talks about in the paper. 0:39:54.920 --> 0:41:52.320 Very nice. Yeah. I if you have been paying attention to, I guess X is that, is it that recently there was shade being thrown between Elon Musk and Yann LeCun about being scientists and writing papers. And anyways, feel free to go read that thread if you want some drama. Yeah, exactly. All right. It's time for Tool of the Show as a foreshadowed slash spoiled. Mine is not a tool to show, but it is a game. This game is available I think on actually various consoles, Xbox, PlayStation, but also I've been playing it on my Steam Deck and that's Dave, the Diver. It's just like one of those things. If you tell someone you're playing a game where you're like a slightly out of shape, spear diving fisherman who goes and collects fish and then runs a sushi restaurant by night, it would just be like, 'What?' But it actually works. It's just kind of, it's kind of fun. And it's a I guess you would call that I'm not, I'm going to misuse it from anyways. It's a bit of a rogue-like game. And that you sort of have the cycle of going down and doing the fishing repeatedly. You're not meant to sort of win it. You're just progressing, and you can do better or worse at going down and catching the fish, but then you go through the cycle of serving the sushi and earning money and upgrading your equipment. And then you can go again and you can learn, you know, get equipment that helps you dive deeper and have better weapons and fend off things trying to eat you in the sea. And so the game sort of progresses. I've not beaten it, presuming that there is an end game. I've not gotten there yet. Still, and there's a story along with it that I won't spoil. But it's just sort of like a fun, never, you don't have to really play for a long time. The controls are kind of cleverly done for how to sort of line up your spear and shoot your spear out. If you're not into, I guess, fish cruelty, definitely stay away. I mean, it does involve eating sushi, so it has to be procured somehow. But you know, definitely like a fun game. Well done. Like the art is kind of a pixel art style. 0:41:52.320 --> 0:42:07.200 So, it's kind of fun that way. But the gameplay just is cool. And, you know, if you only have like five minutes to play, you can go on one of the dives or whatever. You can play for longer, but you don't have to sit down and be like, 'Okay, I'm putting 90 minutes into this.' And so I've been enjoying that. 0:42:07.600 --> 0:42:13.420 But what happens when you die? Like, how does he make it to the next day if a fish literally eats him? 0:42:13.420 --> 0:42:31.420 So, I mean, I think it just sort of hand waves over that part. So if you're collecting up fish and there's like a limit to how many you can carry, like a weight that you die, like you lose everything you're carrying. But I, I presume he has some sort, I don't know that I remember exactly how they hand wave away the fact that you returned to the boat and you can try again. 0:42:31.779 --> 0:42:36.420 Got it. Okay. That's cool. I'll have to check. I've heard good things about that game. So I'll have to check it out. 0:42:36.680 --> 0:42:37.977 Yeah. I was very skeptical. 0:42:37.977 --> 0:44:36.280 But it works. Very cool. All right. My tool of the show is also a game, but very, very relevant. It's called Turing Complete. And I learned a ton about like low-level engineering, like electrical engineering, I guess, or computer engineering from this game. So literally the first level they give you, and Patrick, you're going to know this. They basically give you one of the gates and you have to build all the others. I think NAND is that right? Okay. Yeah. They give you a NAND gate and they're like, make a NOT gate. They're like, make an OR gate, make an AND gate. And so you build all the gates from a NAND gate. And then as you build things, they end up on your toolbar. So it's because it's very modular, right? So it's like, okay, now I don't need to make the NOT gate. As soon as I build it, I just have it now. And then you get all the way to where you like build registers and you build like an ALU. And then you end up like creating your own assembly code. So you learn like in the beginning, they give you assembly code without arguments. So like, you literally have to be like if you're moving memory from this register to that register, you need to have a named instruction for all possible pairs of registers. And then it gives you like assembly code with arguments where you could just have like a move operator or like multi-symbol, I guess, assembly code. And then in the end you end up like creating things that shoot down asteroids and solving mazes and doing all sorts of crazy stuff. Yeah. You play Space Invaders. You build an AI to play Space Invaders. And he, you can actually then like drill down, like you could build the AI to play Space Invaders. 0:44:36.460 --> 0:45:47.720 And if you're inclined, drill all the way down to like an individual NAND gate in this computer you built. It is unbelievably satisfying. It actually taught me a lot because as people know, like Computer Engineering is not my background. So I learned a lot by doing it. It gets really, really hard. You know, I was able to get through the whole game, but I think it's, I would say the mid to end game is pretty much inaccessible for people without an engineering degree. I don't know if I'm just like taking my skills too highly there, but I feel like if a person is going to have a really hard time with the last few levels of this game, but that's okay because you've experienced so much of the content already and maybe would encourage people to go and get like a four-year degree or something. But there's a lot of complexity, but a ton of fun. The UI is really well done the way you place the circuits, and you can color code the traces and everything. Highly recommend it. Definitely worth the, I don't think it's that expensive. Definitely worth the 20 bucks. 0:45:47.720 --> 0:46:13.980 I think we mentioned that one in passing before, but this is like definitely a good review of it. I've not tried it. So now you make me want to go do it, but I know I'll also hate myself having done that bottoms-up thing a few times in a few different contexts, the pain and the suffering, and knowing that yes, you can just look up how a flip flop works if you already know what a flip flop is, but you're supposed to figure it out again from scratch. 0:46:14.600 --> 0:46:43.780 Yeah, I did my, that was another thing. Is I did my best to not cheat at all. There was something, I think it was like an XOR gate or something there. One of these gates were like, I felt like there was an incredibly simple solution, but I had an incredibly convoluted solution because I didn't want to cheat. All right. On to the topic DevOps, or Developer Operations. Patrick, what is DevOps? 0:46:44.240 --> 0:47:09.759 All right. This is a fair question. So words are hard, but I know DevOps, I think is in the set of responsibilities that take you from compiling your code, testing your code, deploying your code—the sort of life cycle of your code into an application from development and into production—and the processes that go around that. 0:47:10.459 --> 0:47:20.200 Yeah, that's right. I feel the same way. What is, I've heard of Site Reliability Engineer. What is that? Is that the same as DevOps or are they different? 0:47:20.779 --> 0:48:15.360 So this is a fair question. I had to double-check just to make sure my thinking around this was good because sometimes you end up with an experiential definition versus an actual definition. So, I think SRE is a term that started out of Google, but has gained acceptance, and a Site Reliability Engineer is responsible for basically keeping production up and working. And so the two do have in some ways overlap, but SRE is responsible for the production environment. So making sure that things, databases are staying performant, web sites are staying up, monitoring traffic, alerting if something goes down and either you know, restarting it, fixing it, whatever themselves are involving engineers or DevOps and sort of that. So, you know, DevOps can definitely make an SRE's job a lot harder if they're not doing a good job helping to check along it, but SREs are really the people responsible for keeping it up and going in the end. 0:48:16.180 --> 0:49:31.296 Yep. Yep. Yep. That's right. And DevOps is extremely, extremely important. You know, outages, I mean, depending on where you work, outages may or may not make the news. They definitely have a huge impact on your company. If people feel like they can't trust your company to be up all the time, that's going to create problems. I mean, as I said, when we were on vacation and we lost mobile data at like a huge impact. And one of the things we thought is maybe we shouldn't have the entire family on one plan. I mean, we're still going to do that because it'd be a huge hassle otherwise, but it made us think about that. Right. So like that. So I think it can have a huge impact on your business, your company's reputation. Also your engineering time, right? So if DevOps is done correctly, then your engineers are getting very quick feedback. Anyone who's built something knows, you ever go back and look at your code from a year ago and you're like, 'What idiot wrote this?' Like this guy needs to be fired. And then you look at the top of the file and it's your name on it or something, right? No, never. 0:49:31.296 --> 0:49:32.798 Done that. Never happened. 0:49:33.160 --> 0:50:29.520 So, you know, if you have a problem that you don't find for three months, you've lost context; you've moved on to something else. If you have a problem and you find out in five minutes, well, that's a totally different story, and you can rectify that pretty quickly. You know, at worst case, you can roll back, even if you don't know what's causing it; you can at least roll back five minutes, and just say, 'Okay, let's take our time and figure this out.' If it's a problem that's maybe been there for three months, and people are just starting to hit it now, that can be extremely stressful for your engineering team and for yourself. So, DevOps is meant to address all of these issues, and it's absolutely critical if you're doing any type of software that goes to other people. 0:50:30.160 --> 0:51:28.160 Yeah. I mean, Jason mentioned outages potentially making the news, but I mean, it can be an absurd amount of money if you think of something like Amazon going down, and everybody's role is to make sure that doesn't happen to some extent. But even in, like you mentioned in smaller companies, if you're trying to, if you can't figure out why something is acting different today than it did yesterday or the day before—and like, you're doing a lot of running around because of some of the techniques that we're going to talk about, that DevOps, not just being a specific person or group of people, but really everyone's responsibility to do some of the stuff that's involved in this and really making sure that you can quickly get to the bottom of what's going on. Nothing worse than banging your head. If you ever encountered that, like my code, I haven't changed anything, but it's doing something completely different, or I changed this one line and everything is different. It's very, very frustrating. And that can happen in more than just your compiling if you're not careful. 0:51:29.020 --> 0:53:28.779 Yep. Yep. Yep. That's right. So yeah, jumping into it, maybe we'll just start. The general DevOps cycle is build, test, release. And so we'll structure this episode in the same way. Looking at the very first step there, building—building software that you tend to give to someone else or to another machine is actually really hard. We've talked about this in the past, but you have a bunch of libraries on your machine. Stepping back a bit here, you have your operating system, which has a bunch of core C libraries. Then you have a whole bunch of packages that you've installed—OS packages. And then some of those packages are themselves package managers, like Python's pip or NPM. So you have all of that context. If you build a piece of software, it could be using any or all of that context. And then you give that software to somebody else or to a machine in the cloud, or to a customer, and that software doesn't work, right? Or it doesn't work the way you intend. So this is called—I always have a hard time saying this name, but it's idempotency, idempotency. And what that means is something is idempotent if it has no external dependencies. Right? Imagine like a function that doesn't use any global variables or system calls or anything. That function has no external dependencies. Every time you call two plus two, you get four every single time, no matter what the context is or time of day—anything like that. And so you want your build system to be like that. The challenge is it becomes kind of getting a hundred percent test coverage. 0:53:28.780 --> 0:53:54.411 where it just becomes exceedingly difficult to get that much idempotency. For example, one really popular tool for building software, at least in C, C++, and Python is Bazel. What you can actually do with Bazel is you can actually have Bazel build its own. 0:53:56.082 --> 0:54:49.440 you know, compiler and have Bazel build its own C libraries and all of that. And then when it builds other software, it's using those compilers and C libraries that it built. So that's kind of the ultimate—I mean, I think the ultimate idempotent solution is, Bazel runs inside of a Docker container that has like an empty Linux image, and inside that Docker container, Bazel goes and builds everything from the ground up. Now again, that's extremely expensive. There's a bunch of challenges there. It's hard to actually execute that. And so for most people, they don't need to go that far, but using Bazel at all is a good first step to get idempotency. 0:54:49.800 --> 0:56:31.340 Yeah. I've heard this called hermetic builds as well. I'm not sure if there's a slight difference between the two, but I think like you mentioned, there's the first line of things, which is, we pin all of the libraries to a known version. Everyone's on the same OS version and you can get 99% of the benefits that if things are reproducible across everyone's, but it still may be—I'm trying to think of like some specific examples, but depending on what you have in the build as well, not just your build infrastructure. If you go to the level of I want the hash of my output binaries to be identical in the matter whose machine I build it on. Like it is literally a bit identical regardless, then you can end up with all sorts of interesting things that are getting picked up or used or the ordering of things. Or, if you really go to an extreme that everybody's machine builds a bit identical binary, it can take a lot of running things down and making sure that things are exactly as you think they are. But the big unlock of having, as Jason mentioned, even like building it from source is one, but at least knowing the version of something that everyone is on is definitely super useful and what everybody's depending on. Because if you do none of it at the opposite, it's not just that your builds aren't reproducible, but some—I've had this on teams. Someone sends you something and you can't get it to compile because there's 15 dependencies that aren't called out anywhere because they just got picked up from their computer. And so you need to one by one resolve, brew install this, pip install that. And you're just going at it one by one, trying to find the set of packages to install. And it's super frustrating. 0:56:32.080 --> 0:58:25.700 Yep. Yep. Yeah, exactly. So getting it completely hermetic or idempotent is really difficult. One thing that's a nice middle ground, which I think is the most popular approach, is to have a build server which matches your production hardware. Imagine if you are deploying a website to an AWS EC2 instance, you probably don't want to build that website on your desktop and copy it over. You probably want to do at least that final build on another EC2 instance with exactly the same specs and hardware and software setup. That way, if there is a problem—let's say you're running GCC 10 or like LibC 2.13 and they're running 2.6 on the cloud. Then what will happen is the build will work fine on your machine. But then when you try to do that final build before pushing it to customers, that final build would happen on the EC2 instance and it would throw some error. And at least you would catch it before it made it to the production servers. So there's a bunch of tools for this. This is called a build farm or a test farm. There's a bunch of tools for this. I've used BuildBarn in the past. I've also used Jenkins. There's GitHub now has GitHub Actions. And so these do a lot of different things. You can run commands off triggers. But at least in the case of BuildBarn, it also is just transparently hooked up to Bazel. So the way it works is you configure your Bazel. Bazel actually has an API for sending, submitting jobs and getting results back. 0:58:25.700 --> 0:59:12.980 And so you can set up Bazel to point to a BuildBarn instance and say, 'Hey, you know, you can build the code locally on my machine, but you could also build it on this BuildBarn instance.' You can also say like only build it remotely. Don't do anything on my machine. And then you can have a lot more control over the machines that are doing the building. And then when you go to actually build the release candidate—maybe you do that every day or once a month or what have you—you could use something like Jenkins, which would spin up a machine in the cloud, do your build on that clean machine and then tear it down. And so getting a good handle on these tools is super important. 0:59:13.440 --> 0:59:56.820 I noticed we did skip over one thing that we probably just took for granted, which is having code in source control and every commit being checked that it builds. So we're talking about building it, but all of this starts from a basis that you're not compiling code. That main branch is currently broken or whatever; you're on is not buildable. And that everybody knows that you just had to do these six things and then it builds, it's been on that before. So you aren't allowed to merge to Main unless that's proven that that branch afterwards will be good. 0:59:57.520 --> 1:01:56.320 Yep. Yep. That's a good point. I think there's a lot of complexity there, but yeah, I think on the surface level, you should have a set of smoke tests and say, look, if this PR breaks one of these tests, or if the Bazel test command isn't clean, then we can't merge a PR or you need like a special override flag or what. So I think there's sort of a meta point here, which is, we've talked a lot about setting up build farms and testing farms and Jenkins and all of that. But then there's those meta question of like, how do you set that up? How do you guarantee that your Jenkins cluster matches your production cluster? And if somebody, let's say upgrades the production cluster to Ubuntu 22, but the build cluster didn't update the build cluster to Ubuntu 22. How do you keep those errors from happening? And the answer is this concept called Infrastructure as Code. I think its also, the acronym is IAC or IaC. They take the A in infrastructure. So what you want to do here is you don't actually want in a corporate setting or a business setting, you don't actually want to go into the Amazon web interface and like click through and create a new VM and assign it an Ethernet adapter. Doing these things is great as a hobbyist, or if you're doing a research project or a one-off thing, but for production, you don't want to be using the Amazon or Google Cloud, any of these UIs and spinning things up because you might just click an option that someone else didn't thinking that's the right way to go. And then it becomes really hard to debug that. 1:01:56.320 --> 1:03:55.960 Or what if you need to spin up like a ninth machine, but Bob spun up the first eight machines and Bob isn't there. So everyone's waiting for Bob. So what the most popular tool for this is, or language for this is Terraform. Terraform is a tool and a language. I believe there is a .tf files that have their own domain specific language. There's some actually issue, some drama around this where I guess there's like an open source fork and HashiCorp, which runs the closed fork. There's some issues there, but I think from what I've been reading, Terraform isn't going anywhere and it's not going to get completely proprietorized or anything like that. So, I'd highly recommend Terraform. The way it works is it's really clever. So Terraform looks at your cloud whenever Terraform create—okay, let me take a step back. So Terraform is this script that procedurally lists a bunch of things that should exist. Like you should have this VM. It should have this much memory. It should have this Ubuntu image on it. And then it should also be able to tunnel through to this other VM, et cetera. So the first time you run this, you have an empty cloud. And so Terraform goes through the list and creates all of those pieces of software infrastructure, right? But when it creates them, it puts special tags and then it creates this cache file, which you put into your source control. And that keeps a history. So Terraform knows, 'Hey, I Terraform created this VM from this git commit of Terraform code.' And so then all you have to do is change that Terraform file and say, 'Oh, actually it has' 1:03:55.960 --> 1:05:13.560 32 gigs of RAM. And then the next time you run TF apply Terraform again, because it's tagged everything, it knows what it created versus what, you know, Patrick created. And so it looks at all the things it created. It looks at the new Terraform file and it realizes what it needs to change. And so it will actually just add RAM to a VM or rename it or something like that instead of just deleting everything and starting over. You can, as you would expect, you can do weird things that make Terraform very confused. And then it's like, I don't really know how to fix this, but other, if you're not trying, you'll almost never have a situation like that. And practice Terraform pretty easily knows, you know, how to make that leap. And so this will help guarantee that everything's on the same page. So imagine, you know, you'd have one place in Terraform that says, 'This is my production Linux image.' And someone just changes that one line to like Ubuntu 22 and under the hood Terraform might like reformat a thousand machines, but it's going to do it all in one step. And that's going to keep inconsistencies popping up. 1:05:13.560 --> 1:05:41.069 I think for, you know, not only being able to do that, but also having as code checked in and a history of the changes is of course definitely useful. So having something that you can have other people approve before you just apply, because I think that's another instance everyone's heard about is someone goes in and just makes a change and nobody knew or it wasn't documented and then everything's broken and no one knows why. So, yep. 1:05:41.940 --> 1:07:22.820 Another popular thing to do is what's called blue-green deployment. Have you heard this term? Is that AB deployment? I, yeah, pretty much. So blue-green just means you have two copies of your infrastructure. For example, like the upgrading Ubuntu example. So you want to upgrade all the machines at once and just have one clean atomic operation. The problem is of course, you know, it takes time to upgrade a thousand machines. And so you don't want your website to be down while you're doing that. And so blue-green is just a fancy way of saying have two copies. So you upgrade the first copy. And while you're doing that, all the traffic goes to the second copy and then you switch the traffic over. It's kind of like if you've ever done a ropes course, you know how like you're, you have two harnesses. And so you disconnect one harness from a rope to connect it to another rope. But if while you're connecting it to other rope, you fall, it's okay because you still have the second harness. One time I went to a place where I don't know how they did this, but it's some kind of mechanical or magnetic thing where basically if, yeah, I think it was, it was a circuit. Like if you hadn't completed the circuit by locking one of the locks, then the other one couldn't be opened. So like you physically couldn't unhitch both of the harnesses at the same time and do something dangerous. And so blue-green is kind of like that where, while you're making a change to blue, you can't send any traffic to it. And then it flips. 1:07:24.420 --> 1:08:57.480 I think not only this being singular stuff, but I think part of what we're talking about is leading to making sure that all of these things are repeatable and then they happen frequently. So, you know, saying, 'Oh, we haven't deployed a new one of these or checked on anything in a month or three months or four months or six months.' You build up a lot of changes, assuming your code's going under active deployment or packages that you're using or getting upgraded, and, you know, security vulnerabilities we've not talked about, but monitoring and updating because of those. And so making sure that you're continuously integrating your code into the system and then deploying those things out at some cadence. I mean, it's not in the mathematical sense continuous, but just doing it at a constant repetitive interval prevents there from being large, gigantic changes. It somewhat becomes a self-fulfilling problem where you wait a really long time because each time you do it is hard, but then the longer you wait, the worse it becomes. And so then of course it's hard because more stuff has changed and more stuff can break. If you're doing that process of build, test, deploy, just consistently and constantly, there's not really too much to be afraid of. And so you may still have an acceptance testing phase in there or whatever. So it's not, although it has its own merits, but it's not the classical 'build, go fast, and break things.' I'm not saying it has to be that way. It's just being always in the process of doing these things. So they're not a we do this once a year and everyone forgets what to do in the meantime. 1:08:58.100 --> 1:09:15.639 Yep. Yep. Yep. Totally right. And once you have a lot of this automated and in code, then you can, you can, you can do this continuously. Imagine if you had a software librarian who was cutting a release every five minutes. It's just not practical. 1:09:16.240 --> 1:09:50.300 I mean, to be clear, there is a role software librarian. I'm sure in some companies are still there, but yes, I think we used to work at a place that had a software librarian who at once a week interval actually was responsible for making sure all the code compiled. I would not recommend that. Yeah, no, I wouldn't recommend that either, but to your point, they couldn't do it every five minutes. So they got kicked out to say once a week, it is this person's job to take all the code, review all the changes, make summary reports and what we now just do constantly with computer systems they would do by hand. 1:09:50.899 --> 1:09:59.600 Yeah. Yeah, exactly. I remember getting just like software librarian coming over to my desk. 'Hey, so, broke the build again.' 1:10:01.467 --> 1:11:01.140 It's like now we have a machine that continuously comes to our desk and puts you on blast. Yeah, that's right. Have you ever done anything interesting? Like any interesting real-life triggers? I'll tell a couple from my background at one point we built, we found a way where, okay, you know, those they're like cardboard, but they can be life-size, like life-size cardboard. Cardboard cutout, yeah. Like statues, I guess is what you'd call them, you know? So there was this one guy who kept breaking the build and we found a thing, I think it was Zazzle or something where we could get a life-size cardboard cutout of this engineer. And we had a Raspberry Pi or some Arduino, something set up where his eyes, we put LEDs for eyes and whenever the build broke, the LEDs flickered. What's the craziest build breaking thing you've seen? 1:11:01.640 --> 1:11:38.679 I don't think I've only ever been on one team. I think did it. They had some stuffed animal that they would take from whoever was like most recently, yeah, screwed something up and they would take it to them and that they would put it on their desk or whatever. Cause they were the most recent person that but no, in general, I've always heard stories or, you know, there's a lava lamp somewhere now still does that. There's like lava lamps because they take a while to heat up and start going. So if the build is broken, the lava lamp turns on. And then, you gotta try to fix it as fast as you can so that the whole office doesn't see that the lava lamp is happily bubbling away and the build is broken. 1:11:39.620 --> 1:11:51.745 This is amazing. We should have like a New York Times Square style ball that drops from the ceiling. And if the build is broken for a whole day, it hits the ground and then it opens up and 1:11:51.745 --> 1:11:52.580 there's a pink slip. 1:11:53.560 --> 1:11:59.507 I was thinking there's like a kaleidoscope of multicolored lights, but yeah, maybe firing is just 1:11:59.609 --> 1:12:02.190 opens up and there's a little receipt printer. 1:12:03.625 --> 1:12:07.179 Dot matrix printer prints your pink slip. 1:12:10.340 --> 1:13:53.840 Yes. That. Oh man. Oh man, that was so good. We, this is another reason why we need a hybrid workplace that we can haunt each other. Just working remote just isn't the same. Oh man. Okay. So how do we measure good and bad DevOps? I think is a really interesting part of this. You know, one thing actually, maybe I'll start by saying you should measure it. In general, my philosophy here, maybe this is like a pessimistic view, but if you don't measure something, you should expect it to be the worst thing possible. That's kind of my overall view of it. And so, or maybe like the worst within some common sense reasoning. And so DevOps is the same way. If you don't measure it, then it's not going to be very good. Or you know what, another thing that I've seen happen where, you know, you don't measure something. And what actually happens is a few people heroically kind of keep it up and running. But then they end up getting burnt out and just feeling like they're not really productive and hurts morale. So, so, measuring things extremely important, how to measure DevOps? I'll just start with the simplest one: build times, you know, build times, test times, deployment time. Is it taking three weeks to build your deployment image? That's a huge problem. You know, our engineers just sitting there, like wrapping their fingers on the desk, waiting to build. That's not good. And so just having a graph of that, I think is a good place to start. 1:13:54.747 --> 1:14:12.559 We didn't talk too much about that, but I think you're absolutely right. Like builds, if not managed properly by everyone, they just become sprawling and take a long time. And, you know, it really kills your ability to iterate. So it's very valuable to do what you can to make sure your builds run fast. 1:14:13.360 --> 1:14:59.520 Yeah. I heard a story from the games industry where they weren't using DLLs. There was this philosophy that like, we just want one EXE and the linker was taking like 90 minutes. And then at some point somebody, you know, just broke the paradigm and said, you know, look, we're going to put DLLs and we'll put the DLLs in the same directory as the EXE. Nobody will even notice. And then, and then the linker took like 40 seconds, but people were waiting an hour and a half. It's like, oh, I missed, well, I guess if you miss a semicolon, you get a compiler error, but it's like, it's like, oh, this number was a three. It should have been a four—90 minutes. You know, and that's just, that just destroys, you know, the morale and the productivity. 1:15:01.060 --> 1:16:58.320 Another thing that's really important are things like release cadence, things like bug tracking, you know, how long does it take to fix a bug? What's the bug rate, you know, per week and trying to get that number down. And then the last one, which is the most important, are surveys. People hate surveys, you know. I mean, outside of work, you know what I mean? You know, if someone comes to your door and wants you to know, you're like, are you voting for this school board member or something, or tell us the survey on like this, or, you know, do you have energy efficient lights? Like it feels like one of these things that you just want to slam the door in somebody's face. But at work, especially if it's done right, surveys are extremely important. You know, what, one example of a case where surveys worked really well, when we built the machine learning backend at Facebook, whenever someone canceled a job, there was a survey, and the way it worked is, you know, they could click cancel and then just, there was a submit button they could submit. And it would, the survey wasn't in any way forcing people, but there's a little radio button said like, why are you canceling this job? Again, you didn't have to choose, but you could choose like, you know, user—I think it was called like, I don't want this job anymore. Or I made, there's basically, there's a choice. I was like, I made a mistake. Then there was a, then the rest were all infrastructure. Like the job isn't finishing or the system is crashed or the machines keep going down. Like a whole bunch of choices that were all relevant to things we were trying to do better. And then when you clicked one of those, it popped up a little feedback form, free form. And that was unbelievably useful. 1:16:58.320 --> 1:17:47.600 So for one, you know, it immediately set a target so people could try to do better on those forms or get the button press radio button press rates down. The second thing is the free form gave us like a whole bunch of really useful context. And what we ended up settling on, which I think every company should do this is, is a regular survey where we ask people like, what is their percent time doing work that they enjoy? And then calling out like, what do they not enjoy? Now some of these things are, you know, oh, I don't enjoy that. Like we don't have chocolate milk or something. Okay. It's like, okay, that's fine. But some of them were like, you know, Hey, I don't enjoy just sitting here waiting for my build for an hour. Right. And that is extremely useful information. 1:17:48.880 --> 1:18:06.860 Have you ever personally, like on any of the teams you've managed, ever done like a survey? You made me think. I don't think I've ever done that. I've always had like higher level organizational surveys, but have you ever done a survey of just like, you know, send out, Hey, everyone send in information about stuff specific to your team's roles? Yeah. 1:18:07.160 --> 1:19:59.020 So, it's a good point. I'll do it at a low cadence because, you know, the company will do a survey every quarter. Sure. And so sometimes I'll ask for questions to be added to that survey. But no, I think trying to remember if it's every, if it's twice a year or once a year. But yeah, we'll do a survey and we'll ask, you know, basically how's it infrastructure? You know, what are things that are slowing you down? What are things that like frustrate you? And yeah, I think it's extremely nice to get that, get that feedback. One interesting, yeah. One interesting thing about surveys is, the best results are actually from the people who are the most reluctant to give feedback. Not because they're afraid or anything, but you'll have the people who are angry and the people who are ecstatic. You get the feedback from them like that day. Right. But it's the people who are like on the margin and the people who are like, I'm too busy to do this survey. Those are actually often the people who have the most interesting things to say. And so it becomes really difficult because you really have to push for, you know, as close to a hundred percent feedback as possible, because as you get closer to a hundred percent, the data gets richer and richer. All right. So, I guess we'll end it with any DevOps horror stories. I'll tell one real quick. This wasn't a company I was working at, but we hired somebody. This was, oh gosh, probably like 10 years ago. We hired somebody who's telling me a story about their past company, which was a startup. 1:20:00.546 --> 1:21:25.292 And, they had this, so all of their code was Tomcat, which was this Java web service library. And so the way it works is you create a JAR file and the JAR file, I don't think it spins up a web server, but I think it answers web requests and then you hook it up to a web server. I'm starting to like get a little hand wavy here, but basically a JAR file for folks who don't know is literally a ZIP file. It's a ZIP file that has certain files in certain places of the ZIP and it's just renamed to JAR. That's what it is. So, so this JAR file like knew about how to handle web requests and it had a bunch of Java bytecode in it. And the way they would do DevOps or deployment is when they made changes to the source code, they would remember what files they changed, like what dot Java files. They would take those dot class files from their machine and they would inject them, like replace them in the JAR, in the ZIP. So they would like SSH into the production machine that's serving all the customer traffic. And they would open this in like a zip editor and they would drag their class files from their machine. And the website you said would go down like once a week because of this, no one would know how to fix it. Oh, 1:21:25.292 --> 1:21:35.340 Wow. I don't think I, other than like having an actual software librarian, like we were talking about being ridiculous now, I don't think I have any particular horror stories. 1:21:35.991 --> 1:21:41.320 What about like a downtime ever? Have you ever been on the crunch where something went down? 1:21:41.820 --> 1:22:28.719 No, see, we did mostly like internal-facing stuff. So we've never had like a problem with SLAs, but you know, I guess we've had issues, you know, just the kind of normal stuff, like somebody built, you know, with a on a machine that had like a really old version of a library and we didn't even know. And then, you know, for some reason, the code that was, everybody was trying to run, which is not work. We couldn't figure out why no one could reproduce it. And there's always this thing when, you know, when you can't reproduce some problem to just hand wave and say, well, it's not broken now. But then every time they would build it, it would, you know, it would have the same issue. And so finally we, we tried to figure it out and it turned out, oh yeah, like you need to upgrade this library. Like this, this library had a known bug in it. You missed the email that told you like, 'Please update.' 1:22:30.740 --> 1:24:16.827 I got one last story. This is what happens when DevOps goes too far in the other direction. I was at this place, which the PRCI. So in other words, it's a test that runs when you have a pull request that you want to get merged in. We were taking like a day and somebody pushed a change where it was a JavaScript or a React bug that they introduced where basically the submit button was never available. Like it was always grayed out. So you could create a machine learning job to define the job and everything, but you couldn't click the submit button to actually launch your job. And so people were having a hard time. I said, for now, let's just always let people click the submit button. And if people submit a bad job, it'll just fail on the backend. And that'll just get us through today. Cause a bunch of people were upset about it. So I submitted a just a one-line change. So literally in the React, it just like in the submit button, the code it's like active equals a bunch of logic. I just deleted that part of that HTML tag. So the button could always be pressed. And then the PRCI took like half a day and then failed because of some other change. And basically, you know, people were super upset and I couldn't get this one-line HTML change in for two days. It's just a total disaster. So, so I think that's a case where a lot of tests were running and I'm sure they're catching things in general, but it was also kind of destroying productivity in a different way. Yeah. 1:24:17.856 --> 1:24:29.640 That's true. We didn't talk about that. Sometimes you may need to just do the crazy thing and just push a change and just get rid of all the other stuff because it's in a bad state and the bad state's not letting it get out of the bad state. 1:24:30.700 --> 1:24:53.480 Yeah. That's right. So yeah, I mean, you know, software librarians probably not a thing anymore, but DevOps is going to be a thing for a long time. DevOps is not going anywhere. If this kind of stuff is really interesting to you, you could definitely get a nice career for many years doing DevOps and I think it's a great field. 1:24:54.500 --> 1:25:06.640 Well, thank you everyone for listening through another episode and appreciate all of our listeners, all the feedback and people on Patreon. So definitely a shout out to everyone who makes the world go round. 1:25:06.960 --> 1:25:10.220 Yeah, totally. Thanks everybody. We'll catch y'all later. Have a good one. 1:25:23.500 --> 1:25:54.299 Mind. Thank you.