WEBVTT 0:00:15.200 --> 0:00:20.880 Programming throwdown episode 174 DevOps. Take it away, Jason. 0:00:21.280 --> 0:04:50.970 Hey, everybody. So this episode we're going to cover DevOps. For people who are new to the show, we're going to cover it somewhere in the middle. We'll talk about news and some other things. First, that is the number one comment we get, is, why did you take so long to talk about DevOps? And you're going to learn a whole bunch of cool stuff today if you're a brand new listener. Welcome to the show. So I wanted to kick it off with a project that I built, which I'm pretty proud of. It's one of these things. You probably have this feeling, whether you're doing woodworking or stuff in the backyard or robotics or anything, you know, you have a project and you say to yourself, this is a coin toss whether this is going to work or not. And it's kind of exciting. It's, it's like, you know, it's kind of like a rush because that, that sort of randomness, right? And then, you know, as it really is a coin toss, half of them, you know, don't work and just decay into dust. Half the projects work really well. This is one that I feel like works pretty well. I created a LinkedIn social media bot, but it doesn't do what you think. You know, most of these bots go out and like, harass people, or maybe that's not the right word, solicit people, right? Mine is the opposite. So mine is a defender. It defends me against the other bots. And so the way this works is it uses large language models. So I'm using ChatGPT. It has like in the cookies file, uses a headless browser with the right cookies set up. So it's logged into my LinkedIn account, goes on LinkedIn, accepts any, you know, connection requests, and then goes to my messages, looks at the unread messages, And I ask ChatGPT, you know, is this a solicitation? Like, is this someone trying to sell me engineering services? And I tell ChatGPT, you know, answer yes or no, followed by explanation, and. And the next one's, you know, is this somebody who is looking to work at my place of work? And based on these answers, it sends automatic responses. And so I actually, in the response, I tell the person, hey, this is an automatic response. Your message was classified as this for this reason. And I paste the ChatGPT answer and then I mute the thread and put it in the other category. And this has worked amazing. So first off, after some Testing, I kicked it off. I got banned three times from LinkedIn because, yeah, not banned, but like throttled, you know, or like literally if, if from my desktop I went to LinkedIn.com I just got like a HTTP timeout or something. So like they throttled me completely. But. But eventually it went through my entire backlog of unread messages and. Oh, and the other thing it does is, you know, if it, if it doesn't catch this as a solicitation, it marks it as safe. And so I have a safe list. And so when I was done, I looked at. Cause I had thousands of unread messages and it was almost all spam, right? But then when I was done, I looked and there was all these people who were legitimate people who had interesting things to say, who I now could see. It's like, basically like I had a handful of needles, you know, like the haystack was burnt to the crisp and what I was left with is all the needles. And so I, some of the people I reached out to were surprised. One person was upset that it took me eight years to respond, whoa. But a lot of people is like, I built a lot of connections with people who just, you know, know, serendipitously, like, or just, you know, message me on LinkedIn and I just couldn't. Couldn't see it. So. So it was an amazing experience. And so now what I did is created a cron job that runs on my Raspberry PI, which runs the same process every hour. And for example, at 4am and at 9am today, it sent automatic messages and muted to people who are trying to sell me software services. So that's just today, this morning. So it's pretty, it's pretty awesome. It's kind of weird, but it's pretty cool. 0:04:51.450 --> 0:05:34.940 I think the, the idea is interesting. I mean, I guess the incentives in the system are a bit weird because LinkedIn doesn't really have an incentive to stop the spam as long as if, like, it doesn't overwhelm and like, mess up user retention. So like, some, some people may, like, it's. It's also a personal thing. Like some of the stuff you might consider spam other people might consider. Like if you're searching for a job, some of those people reaching out may be like, oh, I. I want to reply to them. Like, I'm, I'm hunting, right. And so, yeah, I, I mean, the getting throttled, it makes sense. It's very hard to distinguish between someone doing, like, what you're doing and someone you Know, scraping a bunch of hacked LinkedIn accounts that, you know, trying to get everyone's personal data. 0:05:34.940 --> 0:06:36.220 But yeah, I mean, I was definitely nervous when I got throttled that like I had inadvertently got myself kicked off LinkedIn, but. But yeah, so I just. To double down on that for friends. I have friends at LinkedIn and you know, you guys totally did the right thing by throttling me. I don't feel like that was. I'm glad that it didn't last forever, but. Oh yeah, you hit on something really important, which is, you know, there's this spam, like, hey, I'm in Nigeria, I have a billion dollars. I, I just need you to send me your credit card. And then there's what you said, which is like, like I think solicitations is the better word. And in fact, like, even this ChatGPT thing, the very first thing I ask is, is this a person looking to offer me a job? And if, if it is, then I mark them as safe. So like, clearly, like, there's some solicitations I'm interested in and then others I'm not. And I don't expect like LinkedIn to mark those as spam. Right. Or have that nuance. 0:06:36.540 --> 0:07:14.520 Yeah. So I guess, like, that's one of those. I, you know, I guess you call it fine tuning or whatever. Like at the limit, being able to describe your preferences in a, maybe just a, you know, chain of consciousness style, like way you're thinking about it and having it tuned to those preferences. Like, I am interested in jobs that are legitimate, but I'm not interested in job offers that are clearly just blasts and have none of my personal information in them. Like the person in bott to do research and you can kind of list off all the things you're interested in and kind of build the. Yeah, like you said, like it's a, you can do a lot of probably what you're doing with natural language filters or just like going in and doing basic stuff. It just, you wouldn't take the time. It's not worth it. 0:07:14.840 --> 0:08:25.420 Right, right. Like, for me to build some NLP thing, it would take me a long time versus this only took me maybe three, four hours. And you know, the, the false alarm rate is not the best. You know, I actually somebody text me and I, I had muted them on LinkedIn and they actually knew me from a meetup that I run here in Austin and, and they sent me a text. So yeah, there's definitely the false alarm rate is, let's say, let's say 5% or something, which is not great. But for an application like this, I think it's, it's, it's fine. I'm not landing a space shuttle or anything with this. But it definitely, when you go and look at all the people messaging you and you auto messaging them and muting them, like every now and then I'll go and look at it. It makes me think of what a weird world we live in where someone has, you know, a machine that messages me and my machine answers it and it's all like natural language. Like it looks like they messaged me, you know, manually and it looks like I responded to them manually. But it's all, it's all just fake. 0:08:26.140 --> 0:09:16.730 I think, I think that was on. Oh, I think it was Lex Friedman talking to Yann Lecun and, and lecun was saying the same, something similar, which is, we always think that, oh, there's gonna be some propaganda campaign that, you know, somebody's gonna use, you know, chatgpt to just be super convincing with fake news. He's like, what, what people miss is that if they can do that, you can too. And so a lot of our email and correspondence, at some point in the future, likely, it was astonishing that we effectively ran open ports and like let everyone just send everything they want to us. And instead we'll have secretaries that are trained on our behalf to like filter out that stuff, you know, and you'll just have your bots talking to their bots and some stuff will get through, but most of it, you know, will be, be sort of caught by intelligent filters on our side. And so they'll develop together. 0:09:17.370 --> 0:10:02.430 Yeah, I mean, Scott Galloway from the NYU Business School had this talk that I watched where he was talking about Amazon having this unfair advantage because if you search for just the word batteries on Amazon.com, you got their batteries first and how that's, you know, just a huge advantage for them. I don't think he was going down the, you know, it's not fair route. I think he was more just saying what a tremendous advantage it is for them. And so you could imagine a future where when you search for something on Amazon, there's some browser plugin that re ranks it, you know, that has your interests. And so now like your ranking kind of fights with the, with the server's ranking. 0:10:03.230 --> 0:10:45.700 Yeah, it's an arms race. I mean, it sort of sounds appealing to engage with social media in a way where it's like, I want something to go through and curate stuff on my behalf that works for Me and incentivized by me. Rather than the social media companies trying to get engagement or push just absolute crap to me or try to get me to stay on longer to find the nuggets. Right. Like they're incentivized. Even if they a hundred percent know the things I'm interested in, they're incentivized to spread them out at like a specific repetition interval so that I spend more time, see more ads. And so they keep showing me other stuff that's maybe just close enough that I don't click away or whatever. And so to have somebody go through and filter all that out, it ruins all the advertising models, of course, but it does sound an interesting way to consume media. 0:10:46.180 --> 0:10:59.820 Yeah, yeah, totally. So, yeah, folks out there, if you want, connect with me on LinkedIn, I will accept everybody. And then you can see if you. Well, I will literally accept. Well, I won't accept it, but my AI will accept it. 0:10:59.820 --> 0:11:02.100 Convince Jason's AI bot and he'll share with you. 0:11:02.500 --> 0:11:51.200 Yeah, if you, if you Message me on LinkedIn about your software development company in Eastern Europe and how they can do software cheaper than I can or whatever it is, you will get blocked and it will be pretty funny. All right, on to. Oh, one last thing about that. The code is also totally available on GitHub. Just look me up on GitHub. Aren't them a. Jason Gauches Find my most recent repository. You can get all the code and run it yourself. All right, onto news. News. My first news. So when I was building this LinkedIn autoresponder, I used Amazon Q, which is, you know, one of these copilot type things. Patrick, have you ever used GitHub, Copilot or Amazon Q or any of these? 0:11:51.280 --> 0:12 No, I haven't. I mean, my work, it's. It's sort of banned. And I, as I've admitted many times on here, they don't do as much coding as a hobby as I should. 0:12:00.560 --> 0:12:11.170 So it's also banned at my work. So this was my first opportunity to use it. You know, banned isn't the right word. It's more just like it's. That hasn't been considered. Maybe is better. 0:12:11.170 --> 0:12:18.850 Oh, no, for us, it's like. Yeah, it's, it's. You don't want to upload your code to somewhere, right. In order to get comments on it. 0:12:19.010 --> 0:12:22.770 Oh, I don't know if this one even does that though. Oh, does it do that or does it just run? 0:12:22.850 --> 0:12:26.850 Most of them do, or at least have clauses that they could upload Portions of yours. 0:12:27.170 --> 0:14:52.010 Oh, you know, that's something I didn't even think about. I just assumed that it was running on my desktop. But you're probably right, actually. Um, okay. At any rate, so I tried, I tried this out. It was awesome. So I'll give you some examples, like really concrete examples of things I found really impressive. So. And these are all going to be tiny. So this is part of it is, you know, the whole like build me an entire website chatgpt. I've been extremely skeptical of that for a whole bunch of different reasons. The biggest one is that projects are many modules that you want to have like some thematic consistency. See, right. So I kind of went in pretty skeptical, thinking this is not really going to help. But one thing that it did was, you know, I had a variable called finished and you know, I had while while not finished. And so then later on inside that while loop I put, I put, I think I put finished equals at least if I get this right. Oh, no, sorry. I had while true and I had a variable called finished and then I typed if finished and then it auto completed, you know, colon, new line break again, like not, not like a big deal, but like it kind of figured out like, okay, the variable is called finished. It's inside a while true loop. So if it's finished, you want to just punch out of the loop. Right. There was another thing where. Oh yeah, so I was using this library called Playwright to do the web automation. And when I went to run a function, it, it, it figured out that I should set this parameter. There was like a wait time parameter and it looked further up and figured out, oh, I had set this wait time parameter before and so I'm gonna wanna set it again. And so right when I put the open parentheses, it was like wait time equals two seconds that are a constant or whatever it was. So it was impressive. I actually was really, really surprised. I was very skeptical. I was expecting to uninstall it right away. There were a couple of times where it wanted to auto generate like 30 lines of garbage, like literally just garbage characters. So it's not perfect by any means, but it was, I would say it's a net productivity gain. You know, I was really surprised. 0:14:52.630 --> 0:15:03.710 And you feed it your like context of your entire project in advance or it's doing this from like common available code that it knows about on GitHub. 0:15:03.710 --> 0:15:33.450 I have no idea. I mean, basically I'll tell you what I did as a user. I installed the extension. It makes you create an AWS Builder account which is separate from any other Amazon or AWS accounts you have. So I had to create a new account and that was it under the hood. It could have uploaded my entire project to the Internet. I really have no, I mean it was an open source project anyways, so I had nothing to lose. But yeah, I really don't know what it was doing under the hood, but it was, it was really clever. 0:15:33.770 --> 0:16:16.700 I look forward to this. I mean, I think this will be one of those things where it used to be, I remember that was a long time ago, Google used to sell search appliances and you know, for the same thing you would put them on your, in your, you know, infrastructure, internal, on your network at a company and it would index all of your internal websites and web pages. So you go to like an internal Google. Now a lot of that, you know, has been integrated in other tooling and other solutions and so we don't really think about that anymore. But I think something similar will happen here where you'll have on premise boxes initially. Cause it's the only way to really guarantee, I guess it's, it's sort of safe or AWS instances maybe and you'll upload company, you know, your repositories and have this stuff crunch on it. 0:16:17.270 --> 0:16:30.790 Yep, yep, that makes a ton of sense. I know there's a bunch of open source versions of this that you can run locally or run in your own, in your own cloud. So I know the technology is there. It's just a matter of time. Very cool. 0:16:31.030 --> 0:16:58.690 Mine's on a pretty opposite end of the spectrum, I guess. And that is, I've seen a series of these, so I just wanted to highlight. I picked one to focus on, but there's actually two, which is there's a series of very cheap Risks V or RISC V I don't think, I don't remember. Yeah, RISC V. Okay, that's what I was gonna say. The RISC V processors are starting to kind of roll out and also I saw like commodity prices. So I think they're like 10 cents roughly for one of these processors. 0:16:58.850 --> 0:16:59.730 Do you have any? 0:17:00.129 --> 0:17:08.770 I don't have any yet. It's on my. Yes, I have them in my cart on AliExpress. But they're kind of expensive there, so I need to find another, another place to buy them from. 0:17:09.330 --> 0:17:18.920 And I found them very expensive as well. I was looking at the Mars something and it was like 200 bucks. I was like, how can you get a Raspberry PI 5 for that? 0:17:19.080 --> 0:19:12.600 So, so okay, so maybe this, these These, these people are out to help us. So a YouTuber by the name of Bitloon B I T L U N I if you've never seen them before, lots of interesting computer related projects there. He built a super cluster of 256 of these with some interesting discussions. If it's not something you've ever considered before, but if you want to have 256 processors talking to each other, how do you handle that as like putting them on a bus and what does the communications matrix look like and how do you organize it and think about it and then, you know, how do you tell if they're all working or not? And he's not the, you know, of course other people have built, you know, very high core count, even production processors before, but at a hobbyist level it's kind of nuts. You would never have spent whatever it is, you know, $5 for a processor and then 256 of them, but at $0.10 each and PCB prices have come down to sort of make your own boards. And so it's just an interesting time to kind of consider, consider some of these. And the nice thing about these, which I think there's a pair of them that are pretty similar, CH32V I think one is 003 and the other is 203. So CH32V 203. These are the chips. But often what happens is they, these chips are very, very cheap and if you get help from the company and you need to put them in like your hair dryer to display, you know, to control the temperature output or something, you could buy one of these. But if you want to do hobbyist stuff with it, getting an SDK, getting documentation is pretty hard. So you need a sort of critical mass of people to get, you know, good English translations and SDK. And the very, very popular, I guess example of this is if you hear people talking about, oh hey, why? I just. No, not the STM 30. What are the little WI FI chips? You were using them, Jason. 0:19:12.600 --> 0:19:16.600 Oh, why, why is my, the Pico Raspberry PI Pico. 0:19:16.600 --> 0:19:19.320 Ah, no, now my brain failed me. Oh, this is the best. 0:19:19.320 --> 0:19:20.680 Oh, the ESP32. 0:19:20.680 --> 0:21:22.920 Yeah, thank you, thank you. I was going to say STM32 but ESP32, those are two different things. The ESP32 was similar. It was like a very cheap commodity thing that was like going in light bulbs and it wasn't useful. And then hobbyists sort of realized, wait, this is actually, you know, we can get good documentation for it. And now there's a huge, you know, community of people developing things for ESP32 and other chips. And I don't know for the company if it made an impact or not, but for hobbyists it's certainly cool to get access to, you know, these really cheap things from this Espressif, I believe, is that company. So now we're starting to see the same thing. Debug tools compiler tool chains for these RISC V processors and they're ridiculously cheap in part because they don't have to pay a licensing fee for the RISC V architecture itself. And so you can get them even cheaper. Not that everyone always pays licensing fees, but for sure now, like they, they definitely can't. And then in the show notes, I don't have a great way to do it, but if you search powering a Nixie tube from a RISC V chip, you'll find there is a link here to a YouTuber who I hadn't run across before, but they built a like a Boost Buck Boost controller from a microprocessor. So actually taking usb sort of 5 volt power and like going, I think it was 200 volts to power a Nixie tube, which is a little like vacuum tube except it displays numerals instead of like a transistor. I think it's. Oh, I did a CNL Ohr look in the show notes or search powering a Nixie tube from USB with the 10 cent RISC V processor. And so they, I guess we're doing something at a. Not coordinating with the other person Bitloonie. But they both end up having something and they have a bunch of cool stuff on their channel as well. So worth checking out. So adding this definitely to my list of like things that would be cool to play with. And as Jason has told us on the show before, you know, there is a risk of letting out the magic smoke. So spending, you know, 10 cents on a processor makes letting out the magic. 0:21:22.920 --> 0:22:03.530 Smoke much less painful. Yeah, yeah, totally. Yeah. I feel like just personally at the level where like I have to put the processor somewhere and put the RAM in another spot and the WI FI chip in another spot and hook them all together, that's like one level below where I'm interested. Like, like the, that's why I was looking at the. This mars thing. I can't remember the exact one, but it was basically a Raspberry PI where you have the WI Fi, you have the Bluetooth, you have an Ethernet port, HDMI port, all that stuff. But it's RISC V. And for some reason those are still really expensive. But I think it might be just that they're not mass producing them yet. 0:22:03.850 --> 0:22:18.900 Yeah, I think they'll come. I think the dev boards, I mean, I'm seeing dev boards in the sub $5 range, um, for some of these CH32s. I don't know what the proper abbreviation is, but for these. But they don't have WI fi and stuff like you're saying just yet. 0:22:19.220 --> 0:25:57.880 Got it. Yep, that makes sense. All right, my next news story is PHI3 Vision release. So a bit of background here. Microsoft has been pushing, well, a lot of things, but among them, these sml, or no, wait, slm, Small language model. And so the idea is, you know, this is a model that I actually have one running right now on my phone. And it's. It's pretty good. You know, it runs in real time on my Samsung. It's not even really the latest model or anything. So small model, you know, it will give you some weird answers for sure. You know, it definitely doesn't have the depth of these other models, but, you know, for general queries, it works pretty well. I had something recently. Oh, yeah. So there was some issue with Verizon. We ended up having to call Verizon, our cell phone company, and get it sorted out. But for whatever reason, my entire family lost Internet, which was actually kind of terrifying when you're on vacation. We were on vacation and we just. We still could make phone calls and texts and everything, but we lost the actual Internet, you know, you know, mobile data. And so I was, you know, downloading maps when I was on WI fi and all of that, it was. Is a mess. But one of the things I did is I was asking this Phi 3 because it was local on my phone, just for some general queries, and it was giving me answers. So it's amazing. And so now they released five three Vision, which is an actual, like, multimodal model that you can run locally pretty easily. So people have done a lot of really interesting things with this. You can actually. I have a friend who took a photo of his driver's license and asked 53 vision, hey, pull out the birthday and the driver's license number and the person's name out of this photo, you know, and it did all of that and stole his identity. So it's. No, I'm just kidding. But he ran it locally and so, you know, he's totally safe. He knew that it wouldn't hit the cloud or anything like that with his driver's license. You know, he gave it a few other PDFs and asked it to, you know, extract information and he said it was amazing. He said everything came out flawless. So, you know, I think this is just, I think really something to pay attention to. There are better models like Lava, which is a multimodal model, but these require GPUs with 16 gigs RAM, maybe more than that. And so they're heavier models. It's really interesting to see like for an average person with an average desktop, laptop or even phone, you know, what are, what are we capable of doing here? This is really taking that to the next level so you can check it out. It's, it's pretty easy to download and there's something I'll put it in the show notes called Olama where it will basically hold your hand and you don't need to be even an engineer or a coder to use these models. It gives you a like chatgpt like web interface but for open source models. So this is a super low barrier of entry and I think there's a ton of potential here. So if you're out there, you know, college, high school, just getting started, you know, check this out. I think there's something real powerful stuff you could build on this. 0:25:58.520 --> 0:27:39.630 Very nice. I haven't. I do have a couple apps that, that will run open source. I think Hugging Face, there's an app. I don't know if it's an official app or not, but lets you from your phone hit those. Those aren't local though. But yeah, it is kind of cool to try the various ones and see various ways you do or don't that they can kind of fall over. But the multimodal stuff is cool to dovetail it right into my next one that's talking about Multimodal is OpenAI. I mean news all the time these days, controversies and new releases. But just focusing on a new release released chat GPT4O which the O being omni and sort of their take on this multimodal. So doing being able to use pictures or voice. The demos are, are really cool. Um, sometimes a bit hard to recreate the demos if you try it yourself but in general pretty exciting to say like, you know, hey, I'm sitting here with someone who speaks Russian and I only speak English. Like I'm going to speak English and I want you to translate it to Russian. They speak Russian you know, just describing, you're like, oh, I could do that with the translator app. It's like, yeah, but now you don't need to like, you know, for arbitrary configurations, you know, it can kind of understand. But also some of the behind the scenes things that they're sort of saying that, you know, running this new version, even though it's improved, is just like orders of magnitude cheaper. They're figuring out. Not really. I mean, you can kind of read the rumors about next version of, you know, ChatGPT 5, but it's kind of cool to see them get more and more efficient at running their current generation stuff. Sort of almost faster than you would expect. And it's just interesting to see how quickly things are moving and how the pace goes for training and running a lot of these models. 0:27:40.320 --> 0:29:18.100 Yeah, definitely. I'm a really big fan. I think, you know, they started OpenAI started by doing reinforcement learning, which is obviously near and dear to my heart. I don't think. I don't know if they do much of that anymore, but still, I'm a big fan of the company. I think it's really cool. There was complaints about them, like, I guess, stealing somebody's voice or stealing data from the Internet. They've stolen all of my code, but that's okay because I stole it from Stack Overflow, so I copy pasted it first. So how can I hold them responsible? So no, I think. Well, I mean, that's a whole rabbit hole around data. And if you scrape Reddit, are you like, you know, trouble or whatever? But, but either way, I mean, I think it's amazing. Yeah, this, this. I think the 4 0, the ChatGPT 4 O stands for Omni because of the multimodal part. And I think there's so many interesting ways you could take this. I mean, you know, one thing that just came to mind is, you know, you could run. Oh. Circling back to what some of you said earlier, the app that lets you actually run the models on your phone is called MLC Chat. And you can get that. I think it's on iOS and Android. But. But yeah, imagine like you, you, you take a picture of like even just a restaurant receipt and it keeps track of what things you've eaten at restaurants. I think there's a whole bunch of cool stuff you could do with this. And we'll have to see where people take it. 0:29:19.220 --> 0:31:19.070 All right, time for Book of the Show. My Book of the show is not a book. Spoiler alert. My Tool of the show is also not a tool. Book of the show is a podcast. This is a bit of a different podcast and I don't want to give like a ton of disclaimers I've been enjoying listening to it. It's the. My First Million podcast. This is like two business folks who are kind of like doing marketing but they have interviews with people who have sort of done startups or sold companies and they talk a little bit about details about, you know, kind of exact dollars they were earning and how marketing goes. They were talking to someone recently who had a blog about gardening and then started selling products and turn it into. What I find it interesting is it makes you think a lot about, you know, stuff that's pretty different but adjacent to the stuff we do. I think a lot of software engineers always have this question about entrepreneurial pursuits and like what the various trade offs. These folks are out talking to, you know, people who have done massive things, people who've done small things and also just their observations. It's a very casual podcast, a bit long winded. They also don't necessarily get to the topic of the podcast until late into the podcast but you know, they, they release twice a week and it's, it's a very casual listen. I've been enjoying it. I don't know that I'll listen to for, you know, always and ongoing but it's been good to sort of listen to kind of the sales aspect marketing the kind of parts of the business that I don't normally focus on. Right. I'm always thinking about solving hard problems and machine learning models or whatever. Right. Like those things are much more up my avenue. This is adjacent and people doing often software businesses but talking about motivations and things, you know, related to personnel and you know, how to relate to people and anyways, personally I've just been finding it a nice orthogonal set of views I guess to what I might normally pay attention to. 0:31:19.550 --> 0:31:28.510 Yeah, that's awesome. Yeah. I'll have to check this out. Jason Freed, that name sounds familiar. Is he the basecamp person or is that somebody else? 0:31:28.750 --> 0:31:33.510 Yeah, he was one of the ones. Him and DHH or whatever. Yeah. 0:31:33.510 --> 0:38:41.290 Okay, there we go. Yeah, very cool. Yeah, I'm definitely going to check that out. It looks like fun. All right. So totally on the other end of the spectrum. My book is the. One of the geekiest books I think we've had on the show. It's this research paper from Yann Lecun that he wrote in 2022 called A Path towards Autonomous Machine Intelligence. I have a link to the PDF in, in the show notes. It's not a, it's not a. Called a, an approved paper. It wasn't submitted to any conferences. It's just an open paper that's written on open review. And you can kind of tell, you know, there's some languages, you know, errors, grammatical errors, or some hand wavy stuff. There aren't really any actually. I think there's literally no results section. But sometimes these are the best papers. You know, it's just like raw thoughts of, of, of folks. And so this, this paper is really interesting. It tries to tie together, you know, these, these really large latent models, right? So, so large language models are. And I think we're starting to use foundational models because they are multimodal. But you know, these foundational models are, you know, forward models, which means they look ahead. So it says, you know, given I just wrote this part of a paragraph, what's the next word I should write? Or what's the next handful of words I should write? And so you could extend that to really anything. You could say, well, you know, given that I made this move in chess, what's the next move I should make? Or given that I turn the steering wheel this way in this car, what should I do next with the car? So there's sort of a theory of everything kind of thing going on here where the idea is you build this massive world model that tries to predict the future and then you do a variety of different things like sampling and some gradient based search and different methods to search through potential futures. So I mean, maybe taking a step back so you know, you have the, the world you're in, imagine you are. It's a good problem we can use for this. Like just let's say chess, for example. What about, okay, let's take chess where instead of, you know, you getting a chess board the way a computer would expect it, you're getting pictures of a chessboard, okay? And so you have to play chess from looking at pictures of a chessboard. There's a lot of things that really don't matter, right? Like if the pieces start casting a shadow in the opposite direction because there's a new light source that's not relevant to the problem you're trying to solve, right? And so if you look at ChatGPT or ChatGPT is taking text and trying to produce more text, so it really has to understand everything about text. But in this case, you're not really taking in a picture of a chessboard and trying to create a picture of the chessboard with your move on it. You're trying to take in a chessboard and make the move, right? And so there's a lot of confounding information there, like shadows, for example. So the idea here is you, you create like an embedding or like a low level projection of the image, and then you try to predict the low level prediction of the image with your move in it. So imagine I say, here's a picture of a chessboard and I want you to give me a picture of the chessboard where the pawn has been moved up a square. And so, but instead of literally rendering that photo, which, you know, requires a lot of wasted time, you know, it creates sort of this low dimensional representation and then a low diretical representation of the board with the pawn in the new spot. And then what you could do is you could physically move that pawn, take a picture, and then embed that picture and see if it worked. Right? So, so did the embedding of the future match the embedding of the present plus this function that's supposed to take it into the future? Right. And one thing that you know, you'll, you'll quickly notice is like, you can hack this or you could, you could exploit this. You could say, oh, well, my embedding is just the zero vector. And all actions just turn the zero vector into the zero vector. And so, so I embed to the zero vector, I move the pawn. Now I have the zero vector. I embed the future, which is also the zero vector. And sure enough, the zero vector equals a zero vector. And it could seem like my embedding is amazing, but actually it's not doing anything useful, right? And the way they get around that, which I thought was super, super clever, was I'm going to see if I can explain this easily. They try to maximize the information of the embedding. And the way they do it is, you know, if you have a picture, right? It's, and we've talked about this on, on the show in the past, like, it's little RGB values for each pixel, right? And so it's like a ton of these like RGB values, right? And then when you crush it down into this embedding now it's like a small number of numbers or dimensions, but it's trying to like accomplish some goal. And the goal in his case is to do this forward thing we talked about. But also he wants the variance of the parameters to be as high as possible. So basically trying to make, fancy way of saying, trying to make the numbers vary a lot from, from one picture of a chessboard to another. So if, if you take a bunch of pictures of chessboards and they all collapse to the same embedding you could assume that it's not doing a good job. Now it's possible that like nothing has changed, but it's also possible that you're not doing a good job of capturing information. So. So yeah, that's just part of the paper. I mean this is a 50 page report, so there's a lot of content here. The thing I like about this is you don't really need to know anything beyond like just algebra and, and, and some understanding of like differential equations. So like grasp this. It doesn't like refer to like a million other papers that you then have to go read. So I think it's one of these things that like folks who know Diffeq or even if you don't, you could read this paper and get a lot out of it. It's a good kind of starter paper, but there's a lot of content too. 0:38:41.850 --> 0:39:06.210 And this is part of his thing. Like, I mean, I think he's talked about this, but the contention that the current machine learning stuff like the LLMs can't really do planning. Like it can't, it can't sort of figure out how to do stuff. It can describe the text of a plan, but it itself doesn't do planning because it doesn't sort of move through the space like you're talking about. And this is, this is related to that. 0:39:06.850 --> 0:39:53.380 Yeah, that's right. Yep, yep. So you know, he actually famously put on Twitter, like if you're in college right now, if you're getting a PhD right now, don't bother with LLMs because like the companies are all over it. He's like, do the thing after LLMs, which I think is, is absolutely correct. Like, like this LinkedIn thing we talked about at the beginning of the show, like take an LLM and use it to do something and you will uncover like the next layer of problems. But, but yeah, I think it's interesting even in this paper he's really casting a lot of shade on contrastive learning and, and LLMs and these things in favor of some of these other approaches that he talks about in the paper. 0:39:55.140 --> 0:40:11.860 Very nice. Yeah. If you have been paying attention to, I guess X is that recently there was shade being thrown between Elon Musk and Yann Lecun about being scientists and writing papers. Anyways, feel free to go read that thread if you want some drama. 0:40:12.260 --> 0:40:13.140 Yeah, exactly. 0:40:13.700 --> 0:42:07.280 All right, it's time for Tool of the show as foreshadowed slash spoiled. Mine is not a tool of the show, but it is a game. This game is Available I think on actually various consoles, Xbox, PlayStation. But also I've been playing it on my, my Steam deck and that's Dave the Diver. This is like one of those things if you tell someone you're playing a game where you're like a slightly out of shape spear diving fisherman who goes and collects fish and then runs a sushi restaurant by night, it would just be like, what? But it actually works. It's this. It's kind of, it's kind of fun. Uh, and it's a. I guess you would call that. I'm not. I, I'm gonna misuse this from. Anyways, it's a bit of a rogue, rogue like game in that you sort of have the cycle of going down and doing the fishing repeatedly. You're not meant to sort of win it, you're just progressing. And you can do better or worse at going down and catching the fish, but then you go through the cycle of serving the sushi and earning money and upgrading your equipment and then you can go again and you can learn, you know, get equipment that helps you dive deeper and have better weapons and fend off, you know, things trying to eat you in the sea. And so the game sort of, sort of progresses. I've not beaten it. Presuming that there is an end game. I've not gotten there yet still. And there is a story along with it that I won't, I won't spoil, but it's just sort of like a fun never. You don't have to really play for a long time. The controls are kind of cleverly done for how to sort of line up your spear and shoot your spear out. If you're not into, I guess, fish cruelty, definitely stay away. I mean, it does involve eating sushi, so it has to be procured somehow now. But definitely a fun game. Well done. The art is kind of a pixel art style, so it's kind of fun that way, but the gameplay just is cool. And if you only have five minutes to play, you can go on one of the dives or whatever. You can play for longer, but you don't have to sit down and be like, okay, I'm putting 90 minutes into this. And so I've been enjoying that. 0:42 --> 0:42:13.470 What happens when you die? How does he make it to the next day if a fish literally eats him? 0:42:14.020 --> 0:42:31.380 So I, I mean, I think it just sort of hand ways over that part. So if you, you're collecting up fish and there's like a limit to how many, you can carry like a weight so you die like you lose everything you're carrying. But I, I, I presume he has some sort. I don't know that I remember exactly how they hand wave away the fact that you return to the boat and you can try again. 0:42:31.780 --> 0:42:36.580 Got it. Okay, that's cool. I'll have to check. I've heard good things about that game, so I'll have to check it out. 0:42:36.580 --> 0:42:38.900 Yeah, I was very skeptical, but it works. 0:42:39.670 --> 0:43:08.920 Very cool. All right. My, my tool of the show is also a game, but very, very relevant. It's called Turing Complete. And I learned a ton about like low level engineering, like electrical engineering I guess, or computer engineering from this game. So literally the first level they give you and Patrick, you're going to know this. They basically give you one of the gates and you have to build all the others, I think. Nand. 0:43:09.240 --> 0:43:10.040 Nand, right? 0:43:10.040 --> 0:45:47.780 Okay. Yeah, they give you a NAND gate and they're like make a not gate. They're like make an or gate, make an and gate. And so you build all the gates from a NAND gate and then as you build things they end up on your toolbar. So because it's very modular, right? So it's like, okay, now I don't need to make the not gate as soon as I build it, I just have it now. Um, and then you get all the way to where you like build registers and you build like an alu and then you end up like creating your own assembly code. So you learn like in the beginning they give you assembly code without like arguments. So like you literally have to be like, if you're moving memory from this register to that register, you need to like have a named instruction for like all possible pairs of registers. And then it gives you like assembly code with arguments where you could just have like a move operator or like multi symbol, I guess, assembly code. And then in the end you end up like creating things that shoot down asteroids and solving mazes and doing all sorts of crazy stuff. You play Space Invaders, yet you build an AI to play Space Invaders Invaders. And you can actually then drill down, you could build the AI to play Space Invaders and if you're inclined, drill all the way down to an individual NAND gate in this computer you built. It is unbelievably satisfying. It actually taught me a lot because as people know, computer engineering is not my background. And so I learned a lot by doing. Gets really, really hard. I was able to get through the whole game, but, but I, I think it's, I would say the mid to end game is pretty Much inaccessible for people without an engineering degree. I don't know if I'm just like taking my skills too highly there, but I feel like, if I feel like a person's gonna have a really hard time with the last few levels of this game. But. But that's okay because you've experienced so much of the content already and maybe you would encourage people to go and get like a four year degree or something. But it's, there's. There's a lot of complexity, but a ton of fun. The UI is really well done, the way you place the circuits and you can color code the traces and everything. Highly recommend it. Definitely worth the. I don't think it's even that expensive. Definitely worth the 20 bucks. 0:45:48.820 --> 0:46:14.100 I think we mentioned that one in passing before, but this is definitely a good review of it. I've not tried it, so now you make me want to go do it. But I know I'll also hate myself having done that bottoms up thing a few times in a few different contexts. The pain and the suffering and knowing that, yes, you can just look up how a flip flop works if you already know what a flip flop is, but you're supposed to figure it out again from scratch. 0:46:14.850 --> 0:46:44.030 Yeah, I, I did my. That was another thing is I did my best to not cheat at all. There was something, I think it was like an XOR gate or something there. One of these gates where like, I felt like there was an incredibly simple solution, but I had an incredibly convoluted solution because I didn't want to cheat. All right, onto the topic. DevOps or developer operations. Uh, Patrick, what is DevOps? 0:46:44.510 --> 0:47:09.790 All right, this is, this is a fair question. Uh, so words are hard, but. But I, you know, DevOps, I think, is in the set of responsibilities that take you from compiling your code, testing your code, deploying your code, the sort of life cycle of your code into an application, from development and into production and the processes that go around that. 0:47:10.720 --> 0:47:20.240 Yeah, that's right. I feel the same way. What is, what is a. I've heard of site reliability engineer. What is that? Is that the same as DevOps or are they different? 0:47:20.720 --> 0:48:15.350 So this is a fair question. I had to double check just to make sure my thinking around this was good, because sometimes you end up with a experiential definition versus a, you know, actual definition. So I think SRE is a term that started out of Google but has gained acceptance. And a site reliability engineer is responsible for basically keeping production up and working. And so the two do have in some ways overlap. But SREs are responsible for the production environment. So making sure that things, databases are staying performant, websites are staying up, monitoring traffic, alerting if something goes down and either, you know, restarting it, fixing it, whatever themselves, or involving engineers or DevOps and sort of that. So, you know, DevOps can definitely make an SRE's job a lot harder if, if they're not doing a good job, helping to check along it. But SREs are really the people responsible for keeping it up and going in the end. 0:48:15.990 --> 0:50:29.570 Yep, yep, yep, that's right. Yeah. And DevOps is extremely, extremely important. You know, outages. I mean, depending on where you work, outages may or may not make the news. They definitely, you know, have a huge impact on your company. If people feel like they can't trust your company to be up all the time, that's going to create problems. I mean, as I said, when we were on vacation and we'd lost mobile data had like a huge impact. And one of the things we thought is maybe we shouldn't have the entire family on one plan. I mean, we're still going to do that because it'd be a huge hassle otherwise. But it made us think about that. Right. So, like that. So I think it can have a huge impact on your business, your company's reputation, also your engineering time. Right. So if DevOps is done correctly, then your engineers are getting very quick feedback. Anyone who's built something knows. You ever go back and look at your code from a year ago, you're like, you know, what idiot wrote this? Like, this guy needs to be fired. And then you look at the top of the file and it's your name on it or something, right? No, never done that, Never happened. So, so you know that, you know, so if you, if you have a problem that you don't find for three months, you know, you've lost context, you've moved on to something else. If you have a problem and you find out in five minutes, well, that's a totally different story. And you can rectify that pretty quickly. You know, at worst case, you can roll back. Even if you don't know what's causing it, you can at least roll back five minutes and just say, okay, well, let's, let's take our time and figure this out. If it's a problem that's maybe been there for three months and people are just starting to hit it now, that can be extremely stressful for your engineering team and for yourself. So DevOps is meant to address, you know, all of these, these issues and it's, it's absolutely critical if you're doing any type of software that goes to other people. 0:50:29.890 --> 0:51:28.320 Yeah, I mean, Jason mentioned, you know, outages potentially making the news, but I mean, it can be an absurd amount of money if you think of something like Amazon, you know, going down. And everybody's role is to make sure that doesn't happen to some extent. But even in, like you mentioned in smaller companies, if, if, you know, you're trying to, if you can't figure out why something is acting different today than it did yesterday or the day before, and like, you're doing a lot of running around because of some of the techniques that we're going to talk about that, you know, DevOps not just being a specific person or group of people, but really everyone's responsibility to do some of the stuff that's involved in this and really making sure that you can quickly get to the bottom of what's going on. Nothing worse than banging your head. Have you ever encountered that? Like my code, I haven't changed anything, but is doing something completely different, or I change this one line and everything is different. It's very, very frustrating. And that can happen in more than just your compiling if you're not careful. 0:51:28.800 --> 0:54:49.570 Yep, yep, yep, that's right. So, yeah, jumping into it. Maybe we'll just start. So, so, you know, the, the general DevOps cycle is, you know, build, test, release, and so we'll structure this, this episode in the same way. So looking at the very first step they're building, you know, building software that you tend to, that you intend to give to someone else or to another machine is actually really hard. You know, we've talked about this in the past, but you have a bunch of libraries on your machine. So stepping back a bit here, you know, you have your operating system which has, you know, a bunch of, you know, core C libraries. Then you have a whole bunch of packages that you've installed, you know, OS packages, and then some of those packages are themselves package managers, like Python's PIP or npm. So you have all of that context. And so if you build a piece of software, it could be using any or all of that context, and then you give that software to somebody else or to a machine in the cloud or to a customer. And, and that software doesn't work right, or it doesn't work the way you intend. And so this is called, I always have a hard time saying this, this name, but it's idempotency. Idempotency. And what that means is something is Idempotent if it has no external dependencies, right? And so imagine like a function that doesn't use any global variables or system calls or anything. That function has no external dependencies. Every time you call two plus two, you get four every single time, no matter what the context is or a time of day, anything like that. And so you want your build system to be like that. The challenge is it becomes kind of like getting 100% test coverage where it just becomes exceedingly difficult to sort of get, get that much idempotency. So, for example, you know, one really popular tool for building software, you know, at least in C C and Python, is, is Basel. So Basel. What it, what you can actually do with Bazel is you can actually have Bazel build its own, you know, compiler and have Bazel build its own, you know, C libraries and all of that. And then you have, when it builds other software, it's using those, those compilers and C libraries that it built. And so that's, that's kind of the ultimate, I mean, I think the, the ultimate idempotent solution is, you know, Bazel runs inside of a Docker container that has like an empty Linux image. And inside that Docker container, Basil goes and builds everything from the ground up. Now again, that's extremely expensive. There's a bunch of challenges there. It's hard to actually execute that. And so for most people, they don't need to go that far. But just using Basil at all is a good first step to get item potency. 0:54:49.890 --> 0:56:31.510 Yeah, I've heard this called Hermetic builds as well. I'm not sure if there's a slight difference between the two, but I think like you mentioned, there's the first line of things which is, you know, hey, we pin all of the libraries to a known version, everyone's on the same OS version, and you can get 99% of the benefits that, you know, if things are reproducible across everyone's. But it still may be. I'm trying to think of like some specific examples, but depending on what you have in the build as well, not just your build infrastructure. If you go to the level of I want the hash of my output binaries to be identical no matter whose machine I build it on. Like it is literally bit identical regardless, then you can end up with, you know, all sorts of interesting things that are getting picked up or used or the ordering of things or, you know, if you really, really go to an extreme that everybody's machine builds a bit identical binary, it can take a Lot of running things down and making sure that things are exactly as you think they are. But the big unlock of having, you know, as Jason mentioned, even like building it from source is one, but at least knowing the version of something that everyone is on is definitely super useful and what everybody's depending on. Because if you do none of it at the opposite, it's not just that your builds aren't reproducible, but some. I've had this on teams. You, someone sends you something, you can't get it to compile because you. There's 15 dependencies that aren't called out anywhere because they just got picked up from their computer. And, and so you need to one by one resolve, you know, brew, install this, pip, install that. And you're just going at it one by one by one, you know, trying to find the set of packages to install. And it's super frustrating. 0:56:31.830 --> 0:59:12.950 Yep, yep. Yeah, exactly. So, you know, getting it completely hermetic or idempotent is really difficult. One thing that's a nice middle ground, which I think is the most popular approach, is to have a build server which matches your production hardware. So imagine if you are deploying a website to an AWS EC2 instance. You probably don't want to build that website on your desktop and copy it over. Probably want to do at least that final build on another EC2 instance with exactly the same specs and software set up. That way, if there is a problem, let's say you're running the GCC 10 and, or like libc, you know, 2.13 and they're running 2.6 on the cloud, then what will happen is the build will work fine on your machine, but then when you try to do that final build before pushing it to customers, that final build would happen on this EC2 instance and it would throw some error and at least you would catch it before made it to the production servers. And so there's a bunch of tools for this. You know, this is called Build Farm or Test Farm. There's a bunch of tools for this. I've used Build Barn in the past. I've also used Jenkins. There's, there's GitHub now has GitHub Actions. And so these, these do a lot of different things. You can, you can run commands off triggers, but at least in the case of Build Barn, it also is just transparently hooked up to Bazel. So the way it works is you configure your bazel. Bazel actually has an API for sending submitting jobs and getting results back and so you can set up Bazel to point to a build barn instance and say, hey, you know, you can build the code locally on my machine, but you could also build it on this build barn instance. You could also say like, only build it remotely, don't do anything on my machine. And then you can have a lot more control over the machines that are doing the building. And then when you go to actually build the release candidate, maybe you do that every day or once a month or what have you. You, you could use something like Jenkins, which would spin up a machine in the cloud, do your build on that clean machine and then tear it down. And so getting a good handle on these tools is super important. 0:59:13.990 --> 0:59:56.920 I noticed we did skip over one thing that we probably just took for granted, which is having code in source control and every commit being checked that it builds. So we're talking about building it. But all of this starts from a basis that you're not compiling code, that's a thing, or that main branch is currently broken, or whatever you're on is not buildable. And that, yeah, everybody knows that you just had to do these six things and then it builds then been on that before. So you aren't allowed to merge to main unless that's proven that that branch afterwards will be good. 0:59:57.320 --> 1:05:12.880 Yep, yep, yep, that's a good point. Yeah, I think, yeah, I mean this is, this is, there's a lot of complexity there. But yeah, I think on the surface level you should have a set of smoke tests and say, look, if, if this PR breaks one of these tests, you know, if the basal test command isn't clean, then we can't merge a pr or you need like a special override flag or whatever. So I think there's sort of a meta point here, which is we've talked a lot about setting up build farms and testing farms and Jenkins and all of that. But then there's those meta question of like, how do you set that up? How do you guarantee that your Jenkins cluster matches your production cluster? And if somebody, let's say upgrades the production cluster to Ubuntu 22, but the build cluster, they didn't update the build cluster to Ubuntu 22, how do you keep those errors from happening? And the answer is, is this concept called infrastructure as code? I think it's also the acronym is IAC or iaac. They take the A in infrastructure. And so what you want to do here is you don't actually want, you know, in a, in a corporate setting or a business setting, you don't actually want to go into the Amazon web interface and like click through and create a new VM and assign it an ethernet adapter. Like doing these things is great as a, as a hobbyist or if you're doing a research project or a one off thing. But for production you don't want to be using the Amazon or Google Cloud, any of these UIs and spinning things up because you might just click an option that someone else didn't, thinking that's the right way to go and then it becomes really hard to debug that. Or what if you need to spin up like a ninth machine but Bob span up the first eight machines and Bob isn't there and so everyone's waiting for Bob. Right? So, so, so what the most popular tool for this is or language for this is Terraform. Terraform is a tool and a language. I believe there is a, you know, TF files that have their own domain specific language. There's some actually issue, some drama around this where I guess there's like an open Source Fork and HashiCorp which runs the closed, closed fork. There's some issues there. But I think from what I've been reading, Terraform isn't going anywhere and it's not going to get completely proprietorized or anything like that. So I'd highly recommend Terraform. The way it works is it's really clever. So Terraform looks at your cloud whenever Terraform. Okay, let me take a step back. So Terraform is this script that procedurally like lists a bunch of things that should exist. Like you should have this vm, it should have this much memory, it should have this, you know, Ubuntu image on it and then it should also be able to tunnel through to this other vm, et cetera, et cetera. And so the first time you run this, you have an empty cloud. And so Terraform goes through the list and creates all of those pieces of software infrastructure, right? But when it creates them, it puts special tags and then it creates this, this cache file which you put into your source control and that keeps a history. So Terraform knows, hey I Terraform, you know, created this VM from this git commit of Terraform code. And so then all you have to do is change that Terraform file and say, oh, actually it has 32 gigs of RAM. And then the next time you run tfapply Terraform again because it's tagged everything, it knows what it created versus what Patrick created. And so it looks at all the things it created, it looks at the new Terraform file and it realizes what it needs to change. And so it will actually just add RAM to a VM or rename it or something like that instead of just deleting everything and starting over. You can, as you would expect, you can do weird things that make Terraform very confused. And then it's like, I don't really know how to fix this, but. But other if you're not trying, you'll almost never have a situation like that in practice. Terraform pretty easily knows you know how to make that leap. And so this will help guarantee that everything's on the same page. So imagine, you know, you'd have one place in Terraform that says, this is my production Linux image, and someone just changes that one line to like Ubuntu 22. And under the hood, Terraform might like reformat a thousand machines, but it's going to do it all in one step. And, and that's going to keep inconsistencies. 1:05:12.960 --> 1:05:42.010 Popping up, I think, for, you know, not only being able to do that, but also having AS code checked in and a history of the changes is of course definitely useful. So having something that you can have other people approve before you just apply, because I think that's another instance everyone's heard about is someone goes in and just makes a change and nobody knew or wasn't documented and then everything's broken and no one knows why. So yep. 1:05:42.170 --> 1:05:46.980 Another popular thing to do is what's called blue green deployment. Have you heard this term? 1:05:47.380 --> 1:05:49.540 Is that AB double M I? 1:05:49.700 --> 1:07:22.980 Yeah, pretty much. So blue green just means you have two copies of your infrastructure. And so for example, like the upgrading Ubuntu example, so you want to upgrade all the machines at once and just have one clean atomic operation. The problem is of course, you know, it takes time to upgrade a thousand machines. And so you don't want your website to be down while you're doing that. And so blue green is just a fancy way of saying have two copies. And so you upgrade the first copy and while you're doing that, all the traffic goes to the second copy and then you switch the traffic over. And so it's kind of like if you've ever done a ropes course, you know how like you're, you have two harnesses and so you disconnect one harness from a rope to connect it to another rope. But if while you're connecting it to other rope, you fall, it's okay because you still have the second harness. And one time I went to a place where I don't know how they did this, but it's some kind of mechanical or magnetic thing where basically if, if, if. Yeah, I think it was, it was a circuit. Like if you hadn't completed the circuit by locking one of the locks, then the other one couldn't be open. So like you, you physically couldn't unhitch both of the harnesses at the same time and you know, do something dangerous. And so blue, green is kind of like that where you know, while you're making a change to blue, you can't send any traffic to it and then it flips. 1:07:24.660 --> 1:08:57.600 I think not only, you know, this being singular stuff, but I think part of what we're talking about is leading to making sure that all of these things are repeatable and that they happen frequently. So, you know, saying, oh, we haven't deployed a new one of these or checked on anything in a month or three months or four months or six months. You build up a lot of changes. Assuming your code's going under, you know, active deployment or packages that you're using are getting upgraded and you know, security vulnerabilities we've not talked about, but monitoring and updating because of those. And so making sure that you're continuously integrating your code into the system and then, you know, deploying those things out at some cadence. I mean, it's not in the mathematical sense continuous, but just doing it at a constant, repetitive interval prevents there from being large, gigantic changes. And it somewhat becomes a self fulfilling problem where you wait a really long time because each time you do it is hard. But then the longer you wait, the worse it becomes. And so then of course it's hard because more stuff has changed and more stuff can break. If you're doing that process of build, test, deploy just consistently and constantly, there's not really, not too much to be afraid of. And so you may still have an acceptance testing phase in there or whatever. So it's not, although it has its own merits, but it's not the classical build, go fast and break things. We're not saying it has to be that way. It's just being always in the process of doing these things. So they're not a. We do this once a year and everyone forgets what to do in the meantime. 1:08:57.920 --> 1:09:15.760 Yep, yep, yep, totally right. And once you have a lot of this automated and in code, then you can, you can, you can do this continuously. Imagine if you had a software librarian who was, you know, cutting a release every five minutes. It's just not practical. 1:09 --> 1:09:31.210 I mean, to be clear, there is a role software library. And I'm sure in some companies are still there, but I. Yes, I think we used to work at a place that had a software librarian who at a once a week interval actually was responsible for making sure all the code compiled. I would not recommend that. 1:09:31.530 --> 1:09:33.450 Yeah, no, I wouldn't recommend that either. 1:09:34.250 --> 1:09:50.410 But to your point, they couldn't do it every five minutes. So they got kicked off to say, hey, once a week it is this person's job to take all the code, review all the changes, make summary reports. And you know what we now just do constantly with computer systems they would do by hand. 1:09:50.990 --> 1:10:06.190 Yeah, yeah, exactly. Yeah. I remember getting just like software librarian coming over to my desk. Hey, so broke the build again. It's like, okay, now we have a machine that continuously comes to our desk. 1:10:06.589 --> 1:10:08.110 Now GitHub puts you on blast. 1:10:08.750 --> 1:10:44.150 Yeah, that's right. Have you ever done anything interesting? Like any interesting real life triggers? I'll tell a couple from my, from my background. At one point we built. We found a way where. Okay. You know those, they're like cardboard, but they can be life size. Like life size cardboard? Yeah, yeah, like statues I guess is what you'd call them, you know. So there was this one guy who kept breaking the build and we found a thing, I think it was Zazzle or something where we could get a life size cardboard cutout of this engineer. 1:10:44.390 --> 1:10:44.870 Okay. 1:10:44.870 --> 1:11:01.190 And, and we had a Raspberry PI or some Arduino something set up where his eyes. We put LEDs for eyes and whenever the build broke, the LEDs flickered. What's the, what's the craziest build breaking thing you've seen? 1:11:01.510 --> 1:11:38.850 I. I don't think I've only ever been on one team I think did it. They had some stuffed animal that they would, you know, take from to whoever was like most recently. Yeah. Screwed something up and they would take it to them and that they would, you know, put it on their desk or whatever because they were the most recent person that. But you know, in general, I, I've always heard stories or you know, there's a lava lamp somewhere now still does that. There's like lava lamps because they take a while to heat up and start going. So if the build is broken, the lava lamp turns on and then, you know, you gotta try to fix it as fast as you can so that, you know, the whole office doesn't see that the lava lamp is happily bubbling away and the build is broken. 1:11:39.870 --> 1:11:49.630 This is amazing. We should have like a New York Times Square style ball that Drops from the ceiling. And if the build is broken for a whole day it hits the ground. 1:11:49.710 --> 1:11:52.750 And then it opens up and there's a pink slip. 1:11:54.430 --> 1:12:00.190 I was thinking there's like a kaleidoscope of multicolored lights. But yeah, maybe firing is just opens. 1:12:00.190 --> 1:12:01.870 Up and there's a little receipt printer. 1:12:04.110 --> 1:12:07.270 A dot matrix for printer prints your. Your picture. 1:12:10.310 --> 1:12:11.830 Yes, that. 1:12:12.950 --> 1:13:53.840 Oh man, oh man, that's so good. We, this is another reason why we need a hybrid workplace that we can taunt each other. Just working remote just isn't the same. Oh man. Okay, so how do we measure good and bad? DevOps I think is a really interesting part of this. You know one thing actually, maybe I'll start by saying you should measure it. In general, my, my philosophy here, maybe this is like a pessimistic view, but if you don't measure something, you should expect it to be the worst thing possible. That's kind of my overall view of it. And so. Or maybe like the worst within some like common sense reasoning. And so DevOps is the same way. If you don't measure it, then it's not going to be very good. Or you know what, another thing that I've seen happen where you don't measure something and what actually happens is a few people heroically kind of keep it up and running, but then they end up getting burnt out and just feeling like they're not really productive and hurts morale. So measuring things, extremely important. How to measure DevOps. I'll just start with the simplest one. Build times, build times, test times, deployment time. Is it taking three weeks to build your deployment image? That's a huge problem. Our engineers just sitting there, wrapping their fingers on the desk, waiting the build. That's not good. And so just, just having a graph of that I think is a good place to start. 1:13:54.720 --> 1:14:12.640 We didn't talk too much about that, but I think you're absolutely right. Like builds, if not managed properly by everyone, they just, they become sprawling and take a long time and you know, it really kills your ability to iterate. So it's very valuable to do what you can to make sure your builds run fast. 1:14:13.120 --> 1:17:47.400 Yeah, I heard a story from the games industry where they weren't using DLLs there, there was this philosophy that like, we just want one exe and, and the linker was taking like 90 minutes. And then at some point somebody, you know, just broke the paradigm and said, you know, look, we're going to put DLLs and we'll put the DLLs in the same directory as the EXE, nobody will even notice. And then the linker took like 40 seconds or something, but people were waiting an hour and a half. It's like, oh, I missed. Well, I guess if you miss a semicolon, you get a compiler error. But it's like, it's like, oh, this number was a three, should have been a four. 90 minutes. And that's just, that just destroys, you know, the morale and the productivity. Another thing that's really important are things like release cadence, things like bug tracking. You know, how long does it take to fix a bug? What's the bug rate, you know, per week and trying to get that number down. And then the last one which is the most important are surveys. People hate surveys. You know, I mean, outside of work, you know, I mean, you know, if someone comes to your door and wants you to know, you like, are you voting for this, you know, school board member or something? Or tell us, you know, the survey on like this or, you know, do you have energy efficient lights? Like, it feels like one of these things that you just want to slam the door in somebody's face. But at work, especially if it's, if it's done right, surveys are extremely important. You know, one, one example of a case where surveys worked really well. When we built the, you know, the machine learning backend at Facebook, whenever someone canceled a job, there was a survey. And the way it worked is, you know, they could click cancel and then just there was a submit button they could submit and it would. The survey wasn't in any way forcing people, but there's a little radio button said like, why are you canceling this job? Again, you didn't have to choose, but you could choose like, you know, user, I think it was called, like, I don't want this job anymore. Or I made. There's basically, there's a, there was a choice. I was like, I made a mistake. Um, then there was a. Then the rest were all infrastructure, like the job isn't finishing or the, the system is crashed or the machines keep going down. Like there's a whole bunch of choices that were all relevant to things we were trying to do better. Um, and then when you clicked one of those, it popped up a little feedback form, free form. And that was unbelievably useful. So for one, you know, it immediately set a target so people could try to like, you know, do better on those forms or get, get the, you know, the button press, radio button press, rates down. The second thing is the freeform gave us like A whole bunch of really useful context and what we end up settling on, which I think every company should do this is, is a, a regular survey where we ask people like, what is their percent time doing work that they enjoy? And, and then calling out like, what do they not enjoy? Now some of these things are, you know, oh, I don't enjoy that. Like we don't have chocolate milk or something. Okay. It's like, okay, that's fine. But, but some of them were like, you know, hey, I don't enjoy just sitting here waiting for my build for an hour. Right. And that, that is extremely useful. And for. 1:17:49.150 --> 1:18:05.790 Have you ever personally, like on any of the teams you've managed, ever done like a survey? You made me think, I don't think I've ever done that. I've always had like higher level organizational surveys. But have you ever done a survey of just like, you know, send out, hey, everyone, send in information about stuff specific to your team's roles. 1:18:06.670 --> 1:21:25.860 Yeah. So it's a good point. I'll do it at a low cadence because, you know, the company will do a survey every quarter. Sure. And so sometimes I'll ask for questions to be added to that survey. But, but no, I think trying to remember if it's every, if it's twice a year or once a year. But yeah, we'll do a survey and we'll ask basically, how is IT infrastructure? What are things that are slowing you down? What are things that frustrate you? And yeah, I think it's extremely nice to get that, get that feedback. Yeah. One interesting thing about surveys is the best results are actually from the people who are the most reluctant to give feedback. Not because they're afraid or anything, but you'll have the people who are angry and the people who are ecstatic. You get the feedback from them like that day. Right. Um, but it's the people who are like on the margin and, and the people who are like, I'm too busy to do this survey. Those are actually often the people who have the most interesting things to say. And, and so it becomes really difficult because you really have to push for, you know, as close to a hundred percent feedback as possible. Because as you get closer to a hundred percent, the data gets richer and richer. All right, so I guess we'll end it with any DevOps horror stories. I'll tell one real quick. This wasn't a company I was working at, but we hired somebody. This was, oh gosh, probably like 10 years ago. We hired somebody who's telling me a story about their past company which was a startup and they had this. So all of their code was tomcat, which was this Java web service library. And so the way it works is you create a JAR file and the JAR file, I don't think it spins up a web server, but I think it, it answers web requests and then you hook it up to a web server. I'm starting to like get a little hand wavy here, but basically a JAR file for folks who don't know is is literally a zip file. It's a zip file that has certain files and certain places of the zip and it's just renamed to jar. That's, that's what it is. So, so this JAR file like knew about how to handle web requests and it had a bunch of Java bytecode in it. And the way they would do DevOps or deployment is when they made changes to the source code they would remember what files they changed, like what dot Java. They would take those dot class files from their machine and they would inject them, like replace them in the jar, in the zip. So they would like SSH into the production machine that's serving all the customer traffic and they would open this in like a zip editor and they would drag their class files from their machine. And the website he said would go down like once a week because of this. No one would know how to fix it. Oh wow. 1:21:26.100 --> 1:21:35.220 I don't think I, other than like having an actual software librarian. Like we were talking about being ridiculous. Now I don't know that I have any particular horror stories. 1:21:35.860 --> 1:21:41.300 What about like downtime ever? Have you ever been on the crunch where something went down? 1:21:41.620 --> 1:22:28.800 No. See we did mostly like internal facing stuff. So we've never had like a problem with like SLAs but you know, I guess we've had issues, you know, just the kind of normal stuff like somebody built, you know, with a, on a machine that had like a really old version of a library and we didn't even know. And then you know, like for some reason the code that was, everybody was trying to run, which is not work, we couldn't figure out why no one could reproduce it. And there's always this thing when you know, when you can't reproduce some problem to just hand wave and say, well it's not broken now. But then every time they would build it it would, you know, it would have the same issue. And so finally we, we tried to figure it out and it turned out, oh yeah, like you need to upgrade this library. Like this, this library had a known bug in it, you missed the email that told you like, please update. 1:22:30.880 --> 1:24:18.210 I, I got one last story. This is a, this is what happens when DevOps goes too far in the other direction. I was at this place which the, the prci. So in other words, the tests that run when you have a pull request that you want to get merged in were taking like a day and somebody pushed a change where it was a JavaScript or a react, a react bug that they introduced where basically the submit button was never available. Like it was always grayed out. So you could like create a machine learning job to define the job and everything, but you couldn't click the submit button to actually launch your job. Um, and so people were having a hard time and I said for now, let's just like always let people click the submit button. And like, if people submit a bad job, it'll just fail on the back end and that'll just get us through today because a bunch of people are upset about it. So I submitted a. Just a one line change. So literally in the react, it just like in the submit button the code, it's like active equals a bunch of logic. I just deleted that, that part of that HTML tag so the button could always be pressed. And then the PRCI took like half a day and then failed because of like some other change. And basically like, you know, people were super upset and I couldn't get this like one line HTML change in for like two days. It's just a total disaster. So I think that's a case where, you know, a lot of tests were running and you know, I'm sure they're catching things in general, but it was also kind of destroying productivity in a different way. Yeah, that's true. 1:24:18.210 --> 1:24:29.690 We didn't talk about that. Sometimes you may need to just do the crazy thing and just push a change and just, you know, get rid of all the other stuff because it's in a bad state and the bad state's not letting it get out of the bad state. 1:24:30.810 --> 1:24:53.620 Yeah, yeah, that's right. So yeah, I mean, you know, software librarians, probably not a thing anymore, but DevOps is going to be a thing for a long time. DevOps is not going anywhere. If this kind of stuff is really interesting to you, you could definitely get a nice career for many years doing DevOps and I think it's a great field. 1:24:54.740 --> 1:25:06.680 Well, thank you everyone for listening through another episode and appreciate, appreciate all of our listeners, all the feedback and people on Patreon. So definitely a shout out to everyone who makes the world go round. 1:25:07.080 --> 1:25:45.940 Yeah, totally. Thanks, everybody. We'll catch you all later. Have a good one. Share Distribute Trans work to rebuild Adaptive and share alike in kind.