WEBVTT 0:00:15.600 --> 0:00:21.440 Programming throwdown episode 132, funding open source projects. Take it away, Jason. 0:00:22.160 --> 0:00:31.360 Hey, everybody. So if that voice sounded different to you, that's because we have Adam from CO recursive on the show. So thanks so much, Adam, for coming on. 0:00:31.820 --> 0:00:33.420 Thanks for having me. I'm super excited. 0:00:33.820 --> 0:01:07.540 Cool. Yeah. Patrick is getting a tan. He is on vacation. And part of us wanting to do two shows a month and trying to make the show more oriented or focused is trying to make sure we actually put out that show and that content. We have a ton of stuff we want to talk about, and so I think it's great that when one of us is out, instead of us just not making that content and pushing everything back, that we find folks who, you know, are friends of ours who love podcasting and love tech to jump in. And so Adam's doing that for Patrick. So thank you again, Adam, for coming on. 0:01:08.020 --> 0:01:10.580 Yeah, no problem. I'm excited. Anytime, by the way. 0:01:10.580 --> 0:02:21.780 Yeah, cool. We'll take you up on that. Yeah, I think I'm planning on being out in July, so we'll see how that all goes. But. Cool. So, yeah, I wanted to start by just jumping into gaming setups. I recently redid the gaming setup I had in my living room, and I was pretty stoked about it. You know, I have. I'm really big into the whole emulator scene. You know, I worked on MAME for a long time and all of that. And so I've always had a kind of special place in my heart for those, like, retro games like SNES and N64 and all of these things. And so I have accumulated over the years, like, USB versions of all of these controllers. I don't know if they even had wireless when I started this collection, but I just kind of stuck with the whole wireless. And so I have, you know, those, like, big. They call them fight sticks, but they're big. It looks kind of like you took the. The controller part of an arcade cabinet and just ripped it out and just. It's this big thing. It's like it weighs. It has a weight in it. So it's heavy. It's got the big buttons and it's got the octagonal joystick and everything. And so you just kind of. It's like putting an arcade controller in your lap. 0:02:22.420 --> 0:02:25.700 That's awesome. So you rest it on your lap and then. Does it give you the feel? 0:02:26.470 --> 0:04:14.580 Yeah, yeah, exactly. Yeah, it gives you a feel. It's got the. Yeah, the big buttons, everything. So I've got those. I have two of those I have two of the SNES controllers. The ones I got are the like Buffalo brand or Ibuffalo brand. They're solid. I mean, they've lasted forever. And so similar. I have all sorts of different ones. And so here I am with all these controllers. They're all usb. And you know, when I play on my computer, that'd be fine, but I wanted to play in the living room. And so what I used to do is I used to have just a really long USB extension. And I think USB can go like 50ft or something. I think occasionally we would lose like an input, but it wasn't too bad. But it's kind of janky and you know, people are tripping over it and everything. So what I found online on Amazon, which I didn't know this existed, but blew my mind, is it's a USB dongle. And what comes out the other end is an ethernet port. And then you can run ethernet cable like through the wall in your house. So I have this going from the TV through the wall and then it comes up where my couch is. And so I actually took the. The other end of it is a USB hub. So the ethernet plugs into this USB hub and I have that kind of attached to my couch. And so I can actually plug controllers into the couch and they work with the tv. I am a big fan. Yeah, it was amazing. I found really low latency, which I guess in hindsight isn't that surprising because, you know, we have Cat 5 running through the house for, you know, online gaming and all of that. But yeah, for some reason I just thought that with all these moving with all these parts that there would be some kind of issue. But it works flawlessly. And we have a little cabinet right behind the couch, you can kind of reach behind, you pull out, you know, a SNES controller and then have at it. 0:04:15.220 --> 0:04:20.110 That's amazing. We have this mocha. Have you ever heard of mocha? 0:04:20.430 --> 0:04:21.310 No, I haven't. 0:04:21.390 --> 0:04:48.050 Okay. So our house does not have ethernet throughout the house. So this is not so much gaming as Internet. So mocha is, I think, multimedia over coax. So the coax cables that they did run through the house, you can get this like mocha controller and you plug ethernet in one side and then coax the other side and then it runs high speed ethernet over top of these coax cables. 0:04:48.130 --> 0:04:49.170 Wow, really? 0:04:49.490 --> 0:05:08.050 Yeah. So that's what we're talking on right now. So for whatever reason, like the hookup, I have fiber coming into the house that's super fast. But then like WI fi could barely make it from the basement where the setup is like anywhere else. Right. But you put in these mocha things and like it's super high speed. Like, it doesn't slow it down at all. 0:05:08.940 --> 0:05:22.300 That is awesome. You'll have to look into that. Yeah. My house, actually, I ran this Cat 6 cable just for this, but other than that, it has. Similar to you has cable, has coax all over the house, which we're not even using right now. 0:05:22.860 --> 0:05:29.580 Yeah. And like a lot of people will try to get. There's like power line Ethernet where you run it over your plugs. 0:05:29.900 --> 0:05:33.260 Oh, I've seen that in the store, but I didn't really understand it. Totally. 0:05:33.500 --> 0:05:41.880 Yeah. So I guess it's not as good because, like, there's other things going on. Like, you know, you get interference from the app, actual products that you plug in. 0:05:41.880 --> 0:05:43.720 Right. That's what I thought would happen. Yeah. 0:05:43.720 --> 0:05:50.520 Yeah. Where your coax cable, I guess has lots of bandwidth, but the mocha controllers are more expensive. Like it's a proprietary thing. 0:05:51.320 --> 0:05:52.120 Oh, I see. 0:05:52.200 --> 0:05:54.040 But it's awesome. I totally recommend it. 0:05:54.280 --> 0:06:51.600 That is super cool. My parents recently ditched their cable provider and got 5G home Internet, which I didn't even know is a thing. But yeah, that was pretty cool. I helped them set it up. It's like, you know, this is like every tech son has to do this, like go to their parents house, try to fix their Internet. And so it came in the mail from T Mobile and it took me forever to figure out, like, where to plug in the cable line because I didn't know what we were. What they had gotten. And then I'm like reading instructions and finally it hit me. It's like, oh, this is like a 5G thing. It just plugs into the wall and that's it, you're done. And my first reaction was, oh, well, this Internet can't be that great, but it's amazing. Like the speed test, I think it was 15 megabytes per second down, which is huge. Yeah. I mean, I don't. It's probably more than what I have. And yeah, phenomenal quality. And it's just this little tower looks like a Mac Pro trash can, you know, like one of those form factors. And yeah, I thought that was really impressive. 0:06:52.480 --> 0:06:57.720 So it's basically a cell phone that. Using 5G that provides the Internet. 0:06:57.720 --> 0:07:08.720 That's all it is. But yeah, it gets an amazing. Maybe it has a stronger antenna, not sure. But it gets amazing transfer speeds. It's actually pretty big, so it must be Basically just a giant antenna. 0:07:08.800 --> 0:07:12.640 Yeah, A whole bunch of composite antenna inside somehow. 0:07:13.040 --> 0:07:17.840 Yeah, right, Cool. So your gaming setup is Nintendo Switch. 0:07:18.240 --> 0:07:24.960 Oh, yeah. So I'm not a hardcore gamer, but like, me and my wife always like to play Mario Kart. 0:07:25.280 --> 0:07:26.320 Oh, yeah. Who's better? 0:07:27.060 --> 0:07:28.340 She. She is clearly better. 0:07:28.580 --> 0:07:32.500 My wife is way better too. There's just something about it. 0:07:33.620 --> 0:07:50.980 I forget which. I think it was on the Wii maybe years ago. And yeah, she just played it so often against me and. And she just got so good. And now we play online. Like we have like Nintendo online or whatever. Oh, yeah, we're competing against, you know, 13 year olds from all over the world. 0:07:51.380 --> 0:08:14.910 Yeah, that's right. I remember. I mean, this is. I'm sure it's a very common thing, but you play in your social network, your local hyperlocal network, and you think, I have to be the best in the world. I mean, I've beaten every person in my neighborhood. And then you go to play Mario Kart online, it's like, oh, yeah, I guess you could leave a banana peel right there so that someone misses the jump and dies. Like, you know, it's something I'd really thought about. 0:08:15.950 --> 0:08:20.920 Yeah, they're amazing. If you play online, it's really. I mean, I guess that should be a surprise to no one, but. 0:08:20.920 --> 0:08:47.120 Yeah, yeah, yeah, exactly. Same thing happened with starcraft. This is a really long time ago, but this is like 1998 or something. Starcraft won. You know, we had a little tournament, a land party tournament at our high school, and there was this guy who won it. And I also. I think I came in second or third. And then we thought, oh, if we form a team of the two of us, you know, we have to be the best in the world. And we just got completely demolished by people, as you said, people from all over the world. 0:08:48.340 --> 0:08:50.820 Have you ever seen that documentary, the King of Kong? 0:08:51.300 --> 0:08:52.500 Oh, that is amazing. 0:08:52.820 --> 0:08:53.860 Such a good documentary. 0:08:53.860 --> 0:08:57.460 What is that gentleman's name? The guy with the long hair? I'll look it up. 0:08:58.100 --> 0:09:02.740 Yeah, but it's about a guy. It's Pac man, right? No, it's King Kong. 0:09:02.740 --> 0:09:04.340 Right? King Kong. Yeah, that's right. 0:09:05.060 --> 0:09:12.740 He's gonna beat the world champion. Surprisingly, a lot of drama in a thing about people competing with arcade games. 0:09:13.330 --> 0:09:29.410 Billy Mitchell. That's who it is. Yeah. So Billy Mitchell was. I think he made hot sauce. Yeah, his like day job was making hot sauce. And then on the side he just played Donkey Kong endlessly. I guess Donkey Kong's way harder than I thought before I saw the Movie. 0:09:29.650 --> 0:09:39.330 Yeah, there's whole intricacies to it that I wasn't aware of. But also he was quite the character, like quite a. At least portrayed as quite a villain. 0:09:39.890 --> 0:09:55.990 Yeah, yeah, definitely. Definitely larger than life character. That's a phenomenal movie, by the way. King of Kong. Highly recommended. Have you seen the movie? Somewhat similar in the sense of like a documentary, A class action park on hbo. 0:09:56.070 --> 0:09:59.910 Max, is this the water slide place? Yeah. Yeah. 0:10:00.550 --> 0:11:14.600 Oh my gosh. So folks out there should check this out. There's this water park in, I think the 80s and the 90s called Action Park. And their claim, one of their claim to fames was a loop de loop, a vertical loop de loop water slide. So basically, like, it's exactly what you would think, like you'd see in a roller coaster where, you know, you go down this water slide and you build up so much speed you can literally do a loop de loop and just the G forces are holding you against the edge of the loop de loop until they don't. So it turns out if you're weight, if you're either too light or too heavy from this, like pretty tight bound, then you would not get enough momentum. And what would happen is you would get near the top and then you would smack the inside of the wall of the loop on your way down. And this is just one of the attractions that they talk about in Class Action Park. The thing that blew my mind was not only the obvious, like, oh my God, I can't believe that they're able to run for 20, 30 years, but then also the fact that a lot of the rides later came from those ideas. So those ideas actually spawned a lot of, you know, safe rides. So in a sense it was like a necessary evil or something. But man, it's an unbelievable story. 0:11:14.840 --> 0:11:31.400 Yeah, they were quite innovative, but horrible. Like they, I think in the thing it said, like, the local municipality was upset because they had to send ambulances there so much to get people. And so they bought them an ambulance. They're like, here you go. Yeah, problem solved. 0:11:31.480 --> 0:12:15.070 Dedicated ambulance for this theme park. They had one section where you would drive golf carts with armor and little cannonball cannons that would shoot tennis balls. And if the tennis ball hit the armor, there was some vibration sensor on there. And if it felt enough of an impact, it would start spinning the tank around. And these are all gasoline powered and people were passing out from the fumes and all this crazy stuff. But then it turns out I went to Disney. This is years ago, but much, many years after Action park had Shut down. And Disney actually had the same ride. I mean, of course, much safer electric and they're using foam balls, not tennis balls and everything. But it's actually super, super fun. And it was inspired by action parks. 0:12:15.230 --> 0:12:18.110 That's awesome. Yeah. Innovators. 0:12:18.990 --> 0:16:07.190 Yeah, definitely. Yeah, it's kind of that yang, that kind of like chaotic energy, you know. Cool. All right, on to news. So my news first news item is really interesting. The title is I was wrong. CRDTs are the future. And this is from Joseph Gentle. Yeah, Joseph Gentle. And so people who might have heard of Joseph. So Joseph worked on Google Wave, which was a really old product that died, but kind of, you know, a lot of that carried over into Google Docs and some of these other things. So Google Wave, I think predates all of that stuff. And it was the first time where you could really edit a document in real time with other people and they could see your changes in real time. And so it was really exciting seeing sort of that tech on the web. And then of course, Google Docs now it's, it's ubiquitous. And so all of that worked with something called Operational Transformations or ots. And the way it works is basically, you know, imagine just take a string, for example. There's some long string and I go and I delete a section of that string, right? Maybe I delete three lines. Assuming it's a multi line string, I go and delete like three lines of it. So OT will send a command to the server saying, you know, Jason deleted, you know, this position to this position. And then let's say Adam at the same time deletes something else like a, like maybe a non fully overlapping section of text. So Adam submits a delete right at the same time. So the server is going to get those two commands and the server is going to be smart enough to reconcile that conflict. So it's going to say, okay, now Jason's command came in first. Let's do the delete. When Adam's command comes in, in addition to the delete command and what he wants to delete, there's also a hash saying this is sort of a hash of the state of the string when I wanted to do my delete. Or you could use a timestamp, there's other ways to do it. But so the system will say, okay, when Adam was doing his delete, this text was there, that's been deleted. And so now if I sort of fast forward that change, then we only actually need to delete this much text. And now we have Adam Semantically we have what Adam wanted to delete and what Jason wanted to delete, right? And so you have this server that's similar to when you do git pull, right? You have the server that's like fast forwarding these requests as they come in because the requests are based on old state, right? And that server is also taking those requests once they've been reconciled and pushing them so that everyone has the latest information. Right? And so there's this constant back and forth going on now. It's complicated to get that right. There's ShareJS and like a bunch of libraries that do this for you now. Complicated to get it right, but it's simple to understand. The problem with OT is you have that server, you have to have that one authority who's making all of those judgment calls and doing all of that fast forwarding. And so you have naturally a bottleneck there. So CRDTS is a truly distributed way to do it where you don't need a server. You're kind of achieving a consensus among a group of people. And so it could be in theory completely serverless and peer to peer. And so super, super exciting, really hard to get right because as you can imagine there's no authority. And so that just makes everything so much harder. But like everything, you know, over time hard things become easier. And Joseph Gentle has decided that after spending a decade on OTS that CRDTs have just gotten to the point now where that's the way to go. And he's actually going to drop all of his OT work and switch over to CRDTs, which is pretty wild to see. 0:16:07.610 --> 0:16:30.490 That's cool. I think that Martin, you know the guy, Martin Kleppman I think is his name, who wrote. Yeah, that's right, Designing data intensive applications or something. He's been working on CRDTs for some time now, I think one of the JavaScript projects. So just seeing him interested in it made me think like, oh, I think there's something here, you know that this can be unlocked. 0:16:30.730 --> 0:17:10.880 Yeah, totally agree. Yeah, and he's called out by name in this article. And so yeah, I actually, I know a lot about ots. I implemented an operational transform. That's basically how mamehub and how the network MAME stuff works, I guess tying this back to the intro. So you would do button presses and then it would sort of fast forward your button presses. But yeah, so I definitely want to learn about CRDTs and maybe we'll do a follow up show on CRDTs and how they work. Because at the moment I know almost nothing. I'm just really excited that all these people who know this field better than anyone are getting on board. So there's clearly something amazing here. It just to digest it. 0:17:12.400 --> 0:17:39.120 Yeah, the conflicts I think is the tricky part and I guess they've figured that out. That's why it's conflict free. So for myself I saw this article that was called how we lost 54,000 GitHub stars. Kind of a sad story. So there's this open source library called I don't know if it's HTPI or HTP. Anyways, I've used it before. Let's say it's HTTP httpI. 0:17:39.200 --> 0:17:40.240 I think it's PI. 0:17:40.320 --> 0:18:33.140 Yeah, yeah. So it's a replacement for curl with some really nice defaults so that it's. It's easy to get back like a JSON document, it auto formats it and like it pretifies it, it has some nice defaults. So like it assumes a get. If it makes sense, you could send like key values in a really nice way. So it's has a lot of nice syntactic sugar for making these rest requests. And it's not a new project, it's been around for 10 years, has a website, you know, very heavily invested and well polished open source project. And so the author of it, he was working on the project and also working on a personal project and I think he wanted to make a change to something on his personal GitHub and so he decided to make it private. So he hit the like yes, in GitHub, I confirm to make this private. But he wasn't under his own name actually he was under the HTTPI organization. 0:18:33.700 --> 0:18:34.500 Oh no. 0:18:34.900 --> 0:18:53.780 So accidentally took it private and then when he turned it back public, the star count had reset to zero. So somewhere inside the internal implementation details of GitHub, if you take a project private and then bring it back like that counter resets. So 10 years of people following on GitHub was lost. 0:18:54.450 --> 0:19:06.530 That is unbelievable. I mean it kind of makes sense like if something is private, then only the people who have access to it can star it. And so it probably in this case is nobody. And so. But that's rough. 0:19:06.690 --> 0:19:28.200 Yeah, it's rough. So I mean the blog post was actually nice though because he kind of said like, oh here's what happened. So FYI, if you were following us, you might want to do it again. And then he also said like here's the things that we've planned, like we're still doing lots of stuff and so he's focused on the positive, I guess. But it was an interesting story and a warning for us who have open source projects not to turn it on and off again. 0:19:29.640 --> 0:19:56.540 Yeah, that is unbelievable. Yeah, I mean, it's really rough. I mean, there's not really any way to. It's. I think in general, configuration changes are the cause of so many issues. It's just extremely hard to test something like that. And you know, almost every time, I don't want to say every time, but almost every time you go to the news and you see Google is down, YouTube is down, Facebook is down, it's almost always a configuration change. 0:19:56.940 --> 0:19:57.900 Yeah, totally. 0:19:57.980 --> 0:19:59.660 99 out of 100 times. 0:20:00.060 --> 0:20:04.540 It's like the thing that you can't test well. Right. You just kind of got to make the change. 0:20:05.020 --> 0:20:09.900 Yep. Yeah, exactly. Especially you start dealing with DNS and these other things. It's really hard. 0:20:10.460 --> 0:20:11.660 DNS is the worst. 0:20:12.060 --> 0:20:20.040 DNS is brutal. I mean, it's like email. You know, the protocol was invented so long ago and it's just, you know, everyone's on it and what are you going to do? 0:20:20.040 --> 0:20:33.040 You know, I mean, the DNS might actually be, you know, perfectly fine and a great protocol. It's just, it's a distributed protocol. Right. Where people are caching things and so it's very. Has corner cases. 0:20 --> 0:23:03.240 Yeah, exactly. Yeah. It'd be nice if DNS worked more like git, where somehow like in your. I guess it does. Or it could like. It'd be nice if you could somehow change like every DNS change could somehow only affect your company. And so then, you know, you would, you would kick your entire company off of Google or Facebook or something before you kick the public off. But then you could somehow like push that change to everyone else. I mean, that's actually. DNS would be another really interesting show, but we'd have to get somebody who deeply understands it. But yeah, I think that stuff's fascinating. So my next topic is Dall E making fun of the. Or I guess an homage to Salvador Dali creating images from text. This is from OpenAI. This is super, super cool. And yet they're getting a lot of flack saying they cherry picked these examples and everything. I mean, I don't want to dispute any of that, but all I have to say is this is really cool. I mean, the way they presented this is really exciting. The choices are really nice. It's extremely compelling. And so what this does is you give Dall E a text prompt and the AI generates images and it's remarkable. Like I'm just looking at the site here. This says the first one's an illustration of a baby daikon radish in a tutu walking a dog. And it just does it like, there it is, you know, and an armchair in the shape of an avocado. And it's exactly what it sounds like. It's a chair in the shape of an avocado. Yeah, just amazing. I mean, it blows my mind. I feel like there's so much potential here. There's so many amazing things people could do. I think that, you know, there's an uncanny valley here. So I'm looking at the last one. It says a cat on top and a sketch or, sorry, the exact same cat on top as a sketch on the bottom. So in other words, it wants the AI to generate a picture of a cat and then a sketch of the same cat below it. That seems really hard. I mean, it does it. The thing is, you know, anything that's, you know, an animal or a person, you know, there's this, it has to be super, super accurate. And so this is just weird looking, to be honest. But for things that are kind of abstractions like concept art, it does an amazing job. And I mean, I'm just waiting for somebody, some really creative person to do something truly, you know, earth shattering with this because you have effectively an infinite amount of graphical content here. 0:23:05.240 --> 0:23:20.600 Yeah, it's so neat to play. Like, I don't have access to it, but there was some earlier versions of this that were open source and we're more limited. But it's very fun to like put in a phrase and see what you get. Like. Yeah, it's just, it's amazing. 0:23:21.160 --> 0:23:29.910 Yeah. You know, I mean, here's a very simple thing and maybe someone out there should do this. If you do this, I would love, you know, at us on Twitter. Adam, what are you on Twitter? Are you co recursive? 0:23:29.910 --> 0:23:31.030 Adam Gordon Bell. 0:23:31.190 --> 0:24:27.460 AdamGordenbell okay, AdamGordenbellural netsforlife. Add us if you do this. I'd love to see this. Basically just take a text game and just put a picture next to it. So whatever the description that comes out of the text game, you just set up a two paned type thing and just have a picture next to it. I think that'd be amazing. There is, there's this thing called Z Machine, which is a sort of a virtual machine for text based games. And so it was created to play Zork. And so all of the Zork games use Z Machine. And it just became that for adventure games, it became sort of the gold standard. And so there's probably over a hundred text based adventure games you can play on Z Machine. And so someone would just have to fork Z Machine or even just fork a front end for Z Machine like frots or one of these things and just make it draw the Dall E picture next to it. I think it'd be super fun. 0:24:28.580 --> 0:24:43.220 Definitely there was a Twitter thread where somebody was taking people's profile descriptions on Twitter and feeding it in to Dall E. It was pretty, pretty interesting. What it really? Yeah. Given two sentences of somebody's description. 0:24:44.840 --> 0:24:49.560 So can I know you're saying you don't have access to Dall E. How does someone get access to this? Do you know? 0:24:50.440 --> 0:24:58.120 Oh, there you can. So you click a link to like join their wait list. Basically they're, I think they're just one by one adding people. 0:24:58.680 --> 0:24:59.400 Ah, okay. 0:24:59.960 --> 0:25:06.440 And they also have some restrictions so they don't want you using it for kind of not safe for work type content. 0:25:06.600 --> 0:25:08.600 Oh, sure, yeah, it makes sense. 0:25:10.130 --> 0:25:19.490 But yeah, so it works similar with GPT3. Like I had applied for that and got on the. Got some pretty early access. And that was also a really neat tool to play around with. 0:25:19.810 --> 0:25:44.810 Wow. Super cool. Yeah, so, so yeah, definitely. Folks should try this. You know, sign up for OpenAI. My guess is, let me just try this out really quickly. I want to see if there's a thing where you can say you're a student. Yeah, exactly. Yeah. So you can kind of tell OpenAI, Hey, I'm a student. You know, I just want to try this out and that might bump you up on the list or at least maybe they'll give you a free tier or something like that. But yeah, a ton of cool stuff you could do here. Definitely check it out. 0:25:45.050 --> 0:27:13.840 Yeah. So for my next one, I found this news story inside the longest Atlassian outage of all time. So this was from pragmaticengineer.com this is like this is just a wild story. Yeah. So Atlassian, makers of Jira, which is very central to many people's developer workflows, they had this downtime. So 40 customers lost access to Jira and Atlassian didn't really put out an announcement or say anything. And 40 customers might not sound like a lot, but it turned out that the customers were of the size 50,000 to 800,000 users. These were very large companies and Atlassian wasn't saying anything, but I think they reached out individually to people at these companies, but the companies were getting very Nervous. Then they started talking about it on Twitter. This guy who runs the Pragmatic Engineer started talking about on Twitter. And then Atlassian kind of put out a statement. But if you don't have your ticket tracking system, like, if it's just not available for going on days and days, it's very hard to get your job done right. And there's a lot of important data in there. Like, people were starting to worry, like, what's going on? Like, have we lost all of this information? And it doesn't look good when the company doesn't get back to you at all. Right. So eventually they came out and they kind of said, what happened? Do you have any guesses what happened, Jason? 0:27:14.240 --> 0:27 Was it a DNS issue? 0:27:18.880 --> 0:28:07.960 Sadly, no. So they had, I believe for. They said for legal reasons, which I think means like gpdr. They have, they have a way, a script that removes customers. And there's two ways that it runs. One is like a soft delete, where it just kind of, you know, says, you know, don't. This customer's data is considered deleted, but it's still there. And then the other way, for legal reasons, is it actually deletes all the data from the database for that customer. So they had run this script, which I assume they run on like a nightly basis, to clean up customers. And it's the same script whether it does the hard delete or the soft delete. And it got run when it was supposed to be in soft delete mode, like just hide the customers. It got run in the hard delete mode where it actually blows all the data out of the database. 0:28:08.040 --> 0:28:08.840 Oh, man. 0:28:09.320 --> 0:28:16.840 So that's not good. And then the further thing that happened was it ran with the wrong set of customer IDs. 0:28 --> 0:28:21.320 I was going to say. I mean, why would you even blow away this huge customer? 0:28:22.040 --> 0:29:06.490 Yeah. So, like, apparently not only did they run it wrong in hard delete mode, but they ran it with the wrong customer IDs. So they ended up deleting these 40 customers, probably intending to delete 40 other customers or soft delete them. They actually hard deleted these 40 very large customers. So then they had to come up with what to do from that point. Turns out they, they back up all the data every day. They do backups. Right. But it took them a while to discover that this problem happened. And in the meantime, you know, real customers are doing real things right. So they can't just restore the backup from last night because every customer who's not the 40 customers who are down, like they would all of A sudden lose all those, you know, data. 0:29:06.490 --> 0:29:08.650 Oh, because they're all sharing the same database. 0:29:08.810 --> 0:29:10.090 Yeah, exactly. 0:29:10.170 --> 0:29:10.650 Wow. 0:29:11.210 --> 0:29:36.070 So, yeah, so those were their options, right? Okay, we can restore, but then all these non affected customers will lose data. Or what they ended up doing is like one by one like restoring their data by like selectively inserting rows. Like, I assume they stood up the. They stood up a backup as like another database and like inserted records across like customer by customer, which is why it took them nine days. 0:29:36.150 --> 0:29:41.350 Oh my gosh. I love this part here. It says they can't report the problem because they don't have a JIRA account. 0:29:47.230 --> 0:29:59.190 Oh, it's a crazy story. Like, I feel for them. Like there's been a lot of backlash and people complaining about how horrible this is, but at the same time, like, I just feel bad for these people who had demand this issue or, you. 0:29:59.190 --> 0:30:10.910 Know, I guess, I guess even with gdpr, you can still have like a glacier, like a deep glacier backup, because otherwise they would. The data would literally be gone. 0:30:11.950 --> 0:30:14.990 Oh, yeah, that's true. Yeah, yeah. 0:30:15.790 --> 0:30:29.150 Oh man, that is unbelievable. I mean. Yeah. That if they had literally lost their data because of that would have been really like a disastrous. At least this is. They can recover from. 0:30:30.030 --> 0:30:48.420 Yeah. But I think that they made an error by not talking about this. Like, they kind of left everybody guessing for quite some time until, you know, this came out. Not by them, but by people, you know, being scared and talking about it on hacker news and on Twitter and stuff. 0:30:48.740 --> 0:31:06.280 Right, right. Wow. Yeah. This is an amazing story. I'm so glad you share this with me. This is a wild ride, folks should definitely check this out. That is. That is totally wild. Actually, you know, this reminds me of. Did you hear about Fast? The company Fast? 0:31:06.760 --> 0:31:07.960 Yeah, yeah. 0:31:08.520 --> 0:34:22.530 Oh my God. I mean, that is just unbelievable. So basically. Okay, so, you know, this is a caveat. I mean, maybe we should have had a fifth news story here. But definitely people should do their own research. But what I, what I believe I read was they were, I guess, like, you know, you could always kind of play with statistics. Right? So for example, you could say, well, we have this many visits on our website, but it's not unique visits. So so, you know, then maybe like if somebody or you could say, I have this many calls to my website, but Maybe it takes 10 calls to render the page. Right. So all these are the games you can kind of play where you're saying things that are true, but you're misleading people at the same time. Right. And so I think what was Going on reading is kind of reading between the lines here is, I think Fast was, you know, giving out a bunch of statistics on their growth, which was looking really great, but they weren't using sort of these really core statistics that are hard to fabricate or these core metrics that are hard to fabricate. And so at some point, I think it was a reporter. I don't know if this was leaked or if the reporter reverse engineered it. I think I've heard both stories now, but basically a reporter wrote an article just proving beyond any doubt that, you know, on, on all the core metrics that matter that Fast was an more than an order of magnitude behind what everyone thought, maybe two orders of magnitude or something. And so like, just to like, kind of explain it with an example, you know, they had all this cloud infrastructure. They had a cloud infrastructure organization, but with the amount of traffic they were doing, they could have run it on one desktop. So that's the analogy that I was given. And that by itself is shocking. But, you know, that's not. That's not the most shocking part. The most shocking part was that within three days, the company shut down. And so there wasn't an acquisition. There wasn't like a, hey, we're going to see through this. There wasn't like, we're going to turn the ship around. It wasn't like we work where eventually they got their feet back on the ground. They just shut down. And the only thing I can think of, and you have, obviously there's not a lot of information here. The only thing I can think of is that the investors were just worried about lawsuits and the liability there. And some of these things. A lot of people don't know this, but there are certain things you can do to. That will make you personally liable. Like, I'll give you an example. You know, if you're involved in some kind of like, really overt discrimination on the job, you can actually be held personally liable. So in other words, they can sue you as an individual separate from or in addition to suing your employer. Right. That's an example of a thing where, like, even though you have an llc, you know, under those conditions, the law has been designed so people can actually punch through the LLC and sue you directly. And so, you know, I think that some of these things around, you know, fraud are in that category. And so I think that was really the catalyst that made them just instantly, boom, shut down the whole company. And I mean that. And they were out of money. That definitely didn't help, but they didn't try and look for a buy or anything. They completely shut down. I have friends who worked there and, you know. Yeah, I feel for them. You know, obviously they're super talented. That's a hot market, so. So they'll land on their feet. But it was a huge shock. The, the people who work there had no idea. 0:34:23.090 --> 0:34:40.770 Oh, wow. Yeah, it really went down fast. I think that it's hard if you're looking for new investors. Like, people invest because they don't want to miss out on the new hot thing. Right. And once your reputation has become tarnished, like you're really. You're not attractive. 0:34:41.490 --> 0:35:06.799 Yeah, yeah, totally. I mean, I think. Yeah. I mean, a lot of the engineers ended up going to a company called affairs firm that is, you know, someone in a similar vertical. Now, I think under normal circumstances or ordinary circumstances, you know, there would have been some fire sale acquisition and some employees would have got, you know, some return on their stock and all of that. But. But yeah, this is just a wild story and fortunately, it's extremely rare that something like that happens. 0:35:11.119 --> 0:36:18.580 Sponsor for today's show is Imparticle. At the end of the day, your customer has to be at the center of everything you do. This starts with the right customer data strategy as well as the right foundation to solve the challenges that typically inhibit success, such as data quality, data governance and connectivity. Nparticle is your real time customer data infrastructure that helps accelerate your data strategy by cleansing, visualizing and integrating your customer data from anywhere to anywhere. Ultimately, better data leads to better decisions, better customer experiences, and better outcomes. Some of the best brands in retail, financial services, hospitality, media, travel, gaming and many other industries have chosen Imparticle. Learn more by visiting www.imparticle.com. better data, better decisions, better outcomes. Visit mparticle.com to learn how teams at Postmates, NBCUniversal, Spotify and Airbnb use NPARTICLES Customer Data Infrastructure to accelerate their customer data strategy. 0:36:22.020 --> 0:37:22.610 Cool. So on to Book of the Show. My Book of the show is actually a podcast. Listening to it, neglecting my Audible books and instead doing the podcast thing. It's really fun though. It's the Indie Board Game Designers podcast and it's really fascinating. I mean, first you might even wonder, how can that exist? Like, how can someone make a board game on their own without a big budget or anything? It turns out there's a ton of nice infrastructure there on the supply chain side. And so as long as you're willing to stick with this, canonical things you would find in a board game right like the punch out cardboard pieces and dice. And as long as you don't anything too crazy, you can get pretty much any board game design made relatively cheaply. And so there's a bunch of indie board game designers who do this. There's whole conventions and it's just fascinating. The whole process, I think is really fascinating. 0:37:24.130 --> 0:38:01.490 That's really cool. Yeah. So I picked for my book of the show, it's actually a series, but the series of books called the Laundry Files. And specifically I just finished this one called Quantum of Nightmares. So this is a series of books by Charles Stross and it's hard to describe the genre they're in, but I think if you like science fiction, you'd probably like them. So the Laundry Files is this series. The concept of it is that like the right kind of like computer algorithms can like unlock some sort of demons from some other plane of the universe. 0:38:01.730 --> 0:38:02.610 That's awesome. 0:38:02.690 --> 0:38:51.600 Yeah, but the books are comedies, so it's like a mixture of horror and like Dilbert. So in the very first book I read the Atrocity Archives, like this guy is like an IT worker and he accidentally, you know, is doing some sort of combinatronics and summon some sort of demon. And so it's a bit Men in Black style. Like there's the secret organization who comes and gets him and says like, listen, yes, demons are real and computer science is how you unleash them. And then like. Yeah, so it's like the books often mix these genres together to humorous effect. So in the very first book it's like he is the IT guy at this secret government agency. So he has to make sure the printers work, but also he has to like fight demonstrations and. Yeah, very humorous and strange series of books. 0:38:52.800 --> 0:39:16.380 That is awesome. Yeah, there was, there was a game for the Nintendo DS that was similar. Like you would. That was. It was. Had the same premise. Yeah, I think that's. That sounds awesome. I'm definitely going to give this a read. I love stuff like that. Very cool. Is it. So you said it's three books. Are they like pretty heavy? Like roughly. Like how many pages? Like is this, you know, 20, 30 hours audio or. 0:39:16.860 --> 0:39:22.300 Oh, that's a good question. So the Laundry Files is like a series? I think there's probably like 12 books or something. 0:39:22.620 --> 0:39:23.340 Oh, wow. 0:39:23.820 --> 0:39:31.740 So I would pick the first one, which is called the Atrocity Archives. And the books themselves aren't massive, but there's a number of books. 0:39:32.300 --> 0:39:34.620 Ah, cool. Kind of like Terry Pratchett. 0:39:34.700 --> 0:39:39.740 Yeah, yeah, Definitely some similar vibes. I Would say, yeah, oh, very cool. 0:39:40.120 --> 0:41:57.620 Yeah, that is awesome. Cool. And yeah, and if you want to, to listen to that or Terry Pratchett, you can check both of those out on Audible. Audible is, is a lot of fun. I actually just, I don't know at what point I switched over, but I have the thing now where you pay once a year and you get all your credits up front. And so I just rolled over, So I have 12 credits to spend. And yeah, I still have about three or four books to finish up, so a little bit behind, but. But overall I love it. I mean, especially if you go for a run or go for a hike, you can download the book so you don't have to worry about blowing through all of your, you know, Internet on your cell phone device and everything. So check out Audible. And if you want to check out Audible, you can go to audibletrial.com programming throwdown. That helps us out. If you want to help us out and you already have Audible or you don't want to get Audible, you can check us out on patreon. So that's patreon.com audible programming throwdown. And with that, it's time for Tool of the Show. My tool of the show is Zapier. And so Zapier is a like Internet workflow automation platform. I guess if I had to describe it in one line. What it does is it lets you connect things and automate things that involve common services on the Internet. So for example, you might say, if someone sets up a calendar invite with me, so if I get any new calendar meetings on my Google Calendar, then send me a text message. Right. Or you might say something like, when I email this special address at Zapier, I want Zapier to take the subject line and create a calendar meeting from it. Right. There's all sorts of things you can do. Zapier kind of lets you connect all these things together. I found an open source one called N8N, which I thought was really interesting, I guess. What would that be, Nathan? I don't know, but I actually found Zapier to be pretty generous. They give you, I think it's like a thousand triggers so you can trigger something a thousand times a month before you have to start paying anything. I'm well below that. And overall I thought the experience was great, is really solid and I've used it to streamline a lot of different processes. So I've been really digging that. 0:41:58.020 --> 0:42:06.340 Very cool. Yeah, I've used it for things here and there. It definitely has that kind of Flowchart coding feel to it. But it's cool the things you can hook up to it. 0:42:06.660 --> 0:42:20.330 Yeah, totally. I mean the big thing is it's, it's totally serverless. I mean it's, you know, it's on their network, so you don't need a computer, you know, running Cron, you know, running 247 and all of that. They handle all the infrastructure for you. 0:42:21.760 --> 0:43:35.470 So for my tool, this is a totally bias picked, but I'm going to choose Earthly, which is an open source tool and also a company that employs me. So bias be noted there. But so Earthly is a build tool that tries to bring together these ideas of both building software and continuous integration. So Earth Earthly uses containers, like as in Docker containers to build your software. And it has a format that is kind of a mix of a make file and a docker file. And so you can specify how you would build, you know, your software inside this format. And then, you know, the feature that often brings people to Earthly is then you can run your CI build on your local machine or you can run it in Jenkins or in GitHub Actions or whatever you happen to be using. So yeah, the thing that we're really excited about is just, you know, making builds and CI like, like code and something that you can run on your local environment that you don't have to like put in little commits that says like, oh, hopefully this will get the build working and then push that. And so it's a really cool tool, at least from my perspective. 0:43:36.110 --> 0:43:59.580 And yeah, I'll have to, I'll have to check that out because I. If you look at the eternal Terminal repo, you'll find a ton of these commits that are like, maybe this will work, maybe that'll work, maybe this will work. And they're all commits where I. Using the GitHub website, you know, you can edit single files and. Yeah, yeah, so it's all done through that. And I tried some homebrew thing that was like a local GitHub Actions. 0:43:59.580 --> 0:44:00.220 Oh yeah. 0:44:00.220 --> 0:44:26.170 Where yeah, I tried it and you know, it just didn't work. And I just, I think maybe I needed to be on Linux and I was doing this on my MacBook, but I just, you know, I feel like it's like either works or it doesn't. If it doesn't work, it's like it's not a thing I could easily debug. And so this is a real problem. Yeah, I know I've personally had it. So yeah, I'll have to check that out. So what is, what is Earthly's like, do you sign up or how does it work? Is it a thing you download? 0:44:26.649 --> 0:45:06.290 Yeah. So go to earthly.dev and you can follow the links to download there. You can also find it on GitHub. And so, yeah, it's a command line tool. And yeah, so you write, you kind of write your build in the Earth file format and then you can call earthly from within GitHub Actions or within Jenkins or whatever. And also on your local. So to provide some backstory. So Vlad, who created Earthly, he worked at Google and Google has like internally Blaze, I believe is their build system. And then it was open sourced as Bazel or they made something similar that's open source. I don't know the exact history there. 0:45:06.290 --> 0:45:08.610 Yeah, I think it's like a fork, but yeah, you have the right idea. 0:45:08.850 --> 0:46:08.460 Yeah, so. So Vlad wanted a tool like that, but that was easier to use. Like, Bazel is known to be really amazing if you have like a platform engineering team who can set up like distributed builds and get everything working. And if you use a monorepo and the type of languages that Google uses, it can be really powerful. But for everybody else it's really a big hurdle to overcome. So Vlad built Earthly as kind of a middle ground between something like Bazel and something like GitHub Action. So it allows you to, you know, have caching built in and to kind of do things in parallel. But it also has like a really nice, simple to use format. You know, you don't have to use a monorepo. You can use the tools you already use. If you're using NPM to build something, you can use that. And Earthly, if you're already using Jenkins, you can use Earthly inside Jenkins. Yeah, so that's kind of the backstory behind it. And yeah, you can find it on GitHub Actions or you can just do Brew Install. I think it's Brew Install Earthly for some strange reason. 0:46:09.340 --> 0:46:09.820 Nice. 0:46:10.620 --> 0:46:12.820 But yeah, go to earthly.dev to check. 0:46:12.820 --> 0:46:20.620 So earthly isn't like running your CI, right? You're still. The CI is still running on GitHub Actions. It's just Earthly is facilitating that. 0:46:21.400 --> 0:46:38.280 That's right. Yeah. You would write an Earth file that is similar to a Make file, but with some improvements such that it's containerized and runs in parallel. And then it would still run in GitHub Actions, but it would be running in GitHub Actions via Earthly command line tool. 0:46:38.920 --> 0:47:45.210 Got it, got it. Cool. Yeah, I'll have to check that out. That'll Be nice. So that gets actually a good segue to our topic, which is funding open source projects. And you know, it's one thing that I always wondered, you know, especially when I was in high school and in college, I was wondering, like, these massive projects like Linux, you know, how do you, how does this massive group, you know, community coordinate and how, how do people, you know, just, it costs a lot of resources to run a big project like that. And then also, you know, if we, you know, if these companies decide to be enterprising, you know, how can they make money? And how did people who have sort of put everything in open source, how did they sort of, how did they not just lose all their intellectual property? You know, these things are always really fascinating and I definitely don't have all the answers now, but I've spent many years kind of in this ecosystem. I know you have two, Adam, and we have a pretty good understanding now of how, how that all works. And so I think if you're out there, you know, and you're building kind of tools, you're interested in tools or you're just using a lot of open source, I think that this can answer a lot of those, a lot of questions. 0:47:45.210 --> 0:48:05.130 Yeah, it's a super fascinating area. And like, especially also if you're just thinking like, hey, I'm building an open source tool and I, I would love if this were, you know, something I was paid to do or if I were at least you know, remunerated enough to make it more than just a, you know, something that I have to do or that keeps me busy on the weekends. 0:48:06.170 --> 0:48:53.870 Yeah, that's true. It's like that, it's like that ancient, you know, artist dilemma, like, how can I practice my craft or do that thing that is my sort of core strength that I really love and also be able to sustain money. Yeah. So full disclaimer. You don't have to make money if you have an open source project. I mean, eternal terminal makes $0. And I did put a donations link, but I didn't promote it or anything. And so, yeah, unsurprisingly, Eternal Terminal has never made even a dollar. And that for me is fine. So, you know, we're not saying here that open source projects need to be funded or that they're better, you know, once they're funded or better any, anything. Like there's no value judgment here. What we are going to explain is the ones that are funded, how do they go about doing that? 0:48:55.870 --> 0:48:57.070 Yeah, that makes sense. 0:48:58.110 --> 0:49:57.310 Yeah, so. So I guess, yeah, as I kind of started off here. Donations are really difficult to get right. I think there's this interesting Medium article which we'll link to in the show notes. The title is GitHub Stars won't pay your rent. Which I think is a cute way of saying it, but at a high level. What this person's saying, which I definitely can agree with, is if you put a donation page there, sometimes even if you in the readme, you kind of point up to the donations button and everything. It's generally not enough for people to donate. I think that you need to do something more. I don't think you have to provide anything material, but clearly just telling people to donate on your readme isn't going to be enough. And this person does a much better job than I could. Kind of really diving into details. They spent a ton of time and energy on how to, you know, how to monetize their repository and donations just didn't work for them. But Adam, you found a couple of places where it did work. 0:49:57.870 --> 0:51:50.650 Yeah, so two, two projects that I actually really look up to. Like one is Serenity OS and the other one is the Zig programming language. And so both the people behind these have managed to build up enough support of their open source projects to support themselves full time. And it's kind of interesting the way that they approach that. So Andrew Kelly, who created Zig, you know, he built a community around himself and this language. He blogged about it, he talked about it, he streamed himself building it and he asked for donations when he streamed it. So you know, a little bit his key was almost becoming like a, like an influencer, but for developers, like, here's how I'm building a programming language. And he also talked a lot, you know, to people about how he wanted to work on this for the rest of his life and build this really amazing programming language. And I think that inspired people to say like, you know, I, you know, I'm the type of person who would, I would love to see somebody who can pursue their passion like that. And so he, he was able to leave his job to, to work on Zig full time. And serenityos is a similar project by Andreas Kling who worked at Apple and then left Apple to work on this Serenity os, which is operating system. He's, you know, he was building totally from scratch, streaming himself on YouTube, doing that, and built up enough supporters that it's supporting him. And now for both of these projects, it's now bigger than just the two of them. They have had other people joining and Yeah, I think the big lesson is that you need to, you know, build up a community. You need to use Patreon. You need to tell people, you know, I would like your support. That will help me continue to build this. And I think this is a new idea, right? Like, this isn't. What's the word I want, like. Like, building up a community to support you. Like this in software is kind of an interesting new way to approach things. 0:51:51.450 --> 0:53:27.790 Yeah, totally. I think that. Yeah, I think that, you know, we've seen this in other art forms, right? So you see, like, we go to the. We have a local kind of arts and crafts fair that. That happens once a year, you know, in downtown. And so we'll go there and there'll be a bunch of local artists creating things. And one time we bought a painting from this lady. She gave us her business card. And then we needed a similar kind of thing in a different room. And so these people become part of this local community. They set up this community event. And so they are able to kind of organize and bootstrap, you know, this reputation. Right? So. So we went there because we thought there was. There's a chance there's going to be some really interesting stuff. And so what you're getting out of that is you're getting kind of a relationship. You're seeing the person you're interacting with, and then you get to take a piece of their kind of effort home. And when you see it, you kind of can connect that painting that you see with this person. You kind of imagine them kind of working on it. And so it feels like you really contributed to something real. And so you feel like. Like you actually get something out of it, versus if you just put a donation button, then it's hard for somebody to really feel like they really contributed or they don't know necessarily where their money's going or who's the person working on it, or is this like something that the person has abandoned? And so your money is not actually going to get you anything. Right? And so, yeah, to your point, you know, having that Patreon and having that constant discourse with your community is a collateral that, you know you're in this for the long haul and that they're, you know, really supporting you and your passion. 0:53:28.910 --> 0:53:50.520 And I feel like it. I mean, in some ways, it's the oldest method of funding creative endeavors. Like, I think da Vinci was. You know, there were kings that. That were his patrons that paid him to. To create things. Just the same way that, you know, you can donate to the programming throwdown, Patreon, you know, people supported DaVinci and Michelangelo and et cetera. 0:53:51.720 --> 0:54:01.560 Yeah, yeah, it's a really good analogy. Exactly right. So what is this link? You posted a link here, OSS cache. It's a very short link. I like it. 0:54:02.760 --> 0:54:41.130 Yeah. So working for as I do for a open source dev tool company, I think one of the venture capitalists shared this with us. So this is a list of all of the companies with like a value of over $100 million that are open source. And so the business models have changed over time. Like the very first open source company to get that large was Red Hat. And Red Hat's model was charging for support. So you would pay for support and maintenance to get like rhel, like Red Hat Linux. But the models, I don't know. Should I jump to talking about that article? 0:54:41.290 --> 0:54:42.330 Yeah, let's just do it. 0:54:42.330 --> 0:55:09.860 Yeah, yeah. So Red Hat, that was their model. They became a very large company with the support model. But if you talk to the type of investors who fund open source businesses, they would consider the support model to be sort of a failure. Like only Red Hat really made it work and there was like a whole number of companies that tried and Red Hat also wasn't spectacularly successful. 0:55:10.420 --> 0:56:09.940 Yeah, I remember just riffing on that. I remember when Optimizely went to a support model. So Optimizely used to charge basically I think it was per click or per. It was basically a product. Then you'd pay an incremental fee but you're basically paying for the product. And then I guess there wasn't just enough product there. But for some reason they switched to Optimizely X where it was sort of a support type system where if you had trouble with your how to design your optimization, you would go to them. And that actually to me to double click on what you were saying. That to me was a sign that it wasn't in a good place. And then they, I think Optimizely sold later on they got acquired. I don't remember by who I was. Look it up right now. But to your point, it seems like when companies go the, you know, we'll support this software and that's how we'll all get paid. It seems like that never actually works in practice. This article does a really good job of explaining why. 0:56:10.340 --> 0:56:32.200 Yeah, it's a hard model to make work. Like it is true. Like, okay, I have a product, I open source it, big companies are using it. I'm the best person in the world to offer them support but like that doesn't really scale into becoming a big business. Right. That's just hoping that businesses will pay you because they use something you've made for free. 0:56:32.280 --> 0:56:45.400 Yeah, it's holding them hostage. Right. It's like, you know, like you, at any point you could kind of drop a poison pill and then force those people to get on your support. You know, it's kind of, it's a really weird dynamic economically. 0:56:46.280 --> 0:57:03.710 Yeah. And it's a weird dynamic also just because those big companies can have a lot of power over you. Right. Like I don't know, like they, they could have a, they could have a feature that's being blocked because of your open source product and be very demanding. 0:57:03.710 --> 0:57:38.080 Right, right, that makes sense. So, so episerver is the company that acquired Optimizely, which Episervers, I haven't heard of it. So it's not like they got acquired by Google or something that is a massive like faang company or something. So. So yeah, I think, yeah. And to your point, you know, if you're going to have premium support, you might as well go all the way and just have a separate premium version of your software. Right. Because at the end of the day the support is going to translate into new features that company needs. So why not just make an enterprise version? And I think that model tends to work much better. 0:57:38.880 --> 0:57:46.480 Yeah, I think like that's one of the models you see more commonly today I guess is what is like what they call an open core model. 0:57:46.640 --> 0:57:47.040 Right. 0:57:47.520 --> 0:58:22.250 A core product that is open source and then certain differentiating things that large companies might want would pay for. So I have a friend who works at Mattermost which is like a Slack open source alternative to Slack or Discord. And my understanding is they like it's an open source product but if you want to have like single sign on using like SAML or OpenID or the types of features that you know, a large company demands and will pay for, then those are premium features. So that's like OpenCore as they call that. 0:58:22.970 --> 0:59:26.790 Yeah, exactly. And I think that works pretty well. I think, I mean that's been going all the way back to Visual Studio, right? I mean Visual Studio had the student edition, which I mean actually we should talk about that as well. So in addition to paying for extra functionality, there's sometimes where the, you're paying for the license. So. So I think as a student now you can get, you know, the full Visual Studio, but you're not licensed to run a business that makes certain amount of revenue with that software at that point you have to pay money to get the appropriate license. I know VirtualBox does the same thing where you get, you know, there's feature parity, there's. But it's technically, I guess illegal or I guess civilly like illegal to, to, to run that. To run a business off the VirtualBox without paying for a business license. But. Oh yeah, but Visual Studio used to have like a student edition which was really pared down and so if you wanted the professional edition, then you would have to, you know, pay Microsoft and you get all these extra features. So Even in the 90s, this was a thing. 0:59:27.910 --> 0:59:41.590 And it's interesting because like, you know, what is the difference between something that's open source and something that's free? I mean, they are different. Right? So like, that's a good point, I believe. Right? Visual Studio has free editions, but it's not open source. 0:59:42.790 --> 0:59:51.510 Yep, yep. Yeah, you're totally right. So, yeah, they had the whole, I guess, yeah, that's more like a freemium model than an open core model. So those are two different things. 0:59:52.070 --> 1:00:39.280 And it like my understanding is like for a lot of developer tools, developers have kind of come to expect that things are open source. And especially one reason for that is that maybe if you hit a limitation and you're really locked in to this ecosystem that you might be able to add that feature. Right. By having access to the source, you might be able to add things you need. Also you can kind of inspect it if you're like, hey, this is something that stores the secrets in my organization. Like you might really want it to be something non proprietary that you can look through and make sure it doesn't like curl somebody's password off to some URL or something like that. So there can be security reasons why people might want something to be a very least like source available. Like you could see the source. 1:00:40.400 --> 1:03:03.140 Yeah, that makes sense. Yeah. I mean other things that companies will pay for that individuals wouldn't really get a lot of value out of our, our, as you said, like security and auditing. So there's a good example of this is Sentry. So Sentry, interestingly, I'll tell you, the history of Sentry was started by one. I forgot the person's name, but he was working at a startup and he wanted to catch crashes and then also catch breadcrumbs and look at performance monitoring and all these things. And some of these things, I mean you can kind of piecemeal it together so you can use open telemetry for the performance monitoring. There's like Roll bar and some of these things for catching crashes. Your crash reporter. But there wasn't really one solution that did it all and had a really nice interface. People would have to then put their crashes into Datadog and use datadogs like pattern finding. Right. Because you could imagine you might get back 10,000 crashes. And it's all basically the same issue just with, you know, I expected 32, but I got 16. I expected 30. Or I expected. I expected 32, but I got 17. I expected 32, but I Got 18. It's like, how do you sort of turn that into a pattern that says, you know, this error happened, you know, 17 times. Right. And so they did really nice sort of pattern, I guess, discovery and all of that. Actually, they. There's a whole bunch of cool papers on NLP stuff on how to do that, which they wrote. But so he did an amazing job of that. And then I think, I don't know if the company went under or if this person just left his startup, but he was able to take the IP with him and he just ended up creating that as a service. And to this day, Sentry, even the server, the Sentry server is open source, which is kind of mind blowing. Yeah. You can run a Sentry server yourself with like one Docker command. But were they, you know, like continue to make money is, you know, they have a version of Sentry that's enterprise, that has all the auditing. So if, if someone's like looking through all the errors, you on the back end have a record. Like, you know, Adam looked through all these errors today. You know, Jason went to this project yesterday. And so, you know, companies, you know, that have a lot of liability, you know, that's really important to them. And so even though like literally everything in Sentry is open sourced, you know, they still have a viable business model. 1:03:03.780 --> 1:03:07.140 Wow. I didn't realize the server was open source. That's interesting. 1:03:07.300 --> 1:03:15.620 Yeah. There's a project in their organization, their GitHub organization, called Get Sentry, and it's their entire server. Wow. Yeah. 1:03:16.499 --> 1:03 So after, like, after the whole Red Hat thing, like after the many companies that tried to copy Red Hat with offering support, like one of the other models that really took off is just software as a service. So you see this a lot where in we make a product and it's open source if you run it yourself. Right. But if you want it hosted somewhere and running on a server, you know, and you don't want to have to worry about setting it up, then we'll do that. So you know this is I think like Hashicorp works like this I believe and a lot of database vendors work like this. So you can use. What's an example? 1:03 --> 1:04:06.292 Yeah, what's the Mongo one called? It's like Mongo World or something. I'm going to look. Yeah, like Mongo self hosted MongoDB. 1:04:06.588 --> 1:04:15.080 MongoDB will have their own cloud, I believe that confluent. The Kafka company has their own like host Kafka in the cloud thing that you can use. 1:04:15.480 --> 1:04:28.380 Ah, it's MongoDB Atlas, that's what it's called. And so that you know they hold all the data, they take care of all of it for you, your sandbox. So you obviously can't see other people's data but they take care of the whole platform. 1:04:29.420 --> 1:05:44.830 Yeah. And then the interesting thing with databases in particular like would seem to be a very good fit for this model because you want people to take very good care of your data. But there is a hiccup, right. Which is basically AWS. Right. If you have an open source database, MongoDB for instance, and Amazon can make a hosted version of it, they may actually be better at building a hosted version of it than you are. Or at very least they already have. Most of your customers already have an account there where they don't have an account with your service. So this kind of concern about Amazon led to open source database companies to kind of shift licensing. So I think MariaDB was the first company to do a. I believe they call a source available license. So MariaDB is like a replacement for MySQL and it's a commercial company and you can access the source for it. They have a license that looks much like the Mozilla public license except it says you can't compete with us. You can't make any money doing something using this source that we do with it. So that's basically like not you cloud providers, you can't. This isn't for you, right? 1:05:45.140 --> 1:05:45.460 Yeah. 1:05:46.340 --> 1:05:58.420 And I think this has been a controversial thing in some areas, you know, because people are like, is that really open source? The source is available but you can't do whatever you want with it. It is limited in certain cases. 1:05:58.740 --> 1:06:05.940 Yeah, I mean it's a good question. What was the Amazon one called? Was it called Beanstalk or was that something else? I think it up right now. 1:06:06.260 --> 1:06:07.100 Yeah. I don't know. 1:06:07.100 --> 1:07:00.020 Oh, I think. Okay. You used to have a name. I think they changed the name because they were getting. Part of the problem was AWS basically made it Seem, I think it was called Beanstalk, but they made it seem like this is AWS's take on MongoDB, when in fact it literally was MongoDB. And so now they call it MongoDB on AWS. So they have sort of taken the blinds down on that. But. But yeah, that caused a huge stir, completely disrupted MongoDB's business model. The more extreme version of this is Docker Swarm and Kubernetes, where Kubernetes basically took the, you know, Docker D, the daemon and all the core from Docker and built their own version of Docker Swarm that's just much better than Docker Swarm and just completely undercut Docker's entire business model. 1:07:00.900 --> 1:07:06.660 That's true. Although, like, the thing with Kubernetes is that it is also open source, so. 1:07:07.620 --> 1:07:17.840 That'S a good point. So, yeah, I guess. I guess it's. It is also open source. Oh, I see. But it's. I think it's really just that Google can just afford to hemorrhage money on it and. 1:07:17.840 --> 1:07:19.840 Yeah, exactly. 1:07:20.240 --> 1:07:20.640 Yeah. 1:07:20.640 --> 1:08:22.550 The interesting one to me, and I don't know all the history on it though, is elasticsearch. So riadb, like, they thought of this ahead of time. They said, we're gonna use this license. It's not quite open source. Right. And it basically says, like, not for you, Amazon, elasticsearch, it was an open source, you know, search databasey thing. Like they had an Apache license, I believe, and they started their own hosted cloud for elasticsearch. They're like, we're really good at managing this distributed database, so let us do it. But Amazon also was really good at running elasticsearch. And so elasticsearch changed their license like, I think like a year or two ago after, you know, like years of people contributing to it, thinking it was like true open source, they changed the license to be sort of a source available license to prevent Amazon from using it. But they really upset a lot of people who were like, I contributed to this project, I thought it was open source and you're changing the license on me. And so it's a tricky area. 1:08:23.430 --> 1:09:29.360 Yeah, yeah, it's. It's wild. Yeah. I think you really have to be providing additional value. I think that the license model, it's just very tricky to get right. I mean, I wonder if there's any data on financially how that ended up like, it's counterfactual. Right. We'll never know what would have happened if they hadn't changed the license. But what we could do is we could see, okay, once they change the license, then I guess Amazon would have to stop just having their own fork of elasticsearch. And then how many people went to the elasticsearch? It'd be really interesting to see a post mortem on what the effect was of that. Yeah, one thing Amazon has now is they used to host Neo4J and now they wrote their own called Neptune. They have their own graph database. And that's the other concern is, you know, like you could have a really well crafted license, but then like what is your actual ip, you know, and what, and if Amazon makes their own object database, how can you sort of prove that they took your idea? Right. 1:09 --> 1:09:43.680 Yeah. And yeah, I believe with elasticsearch that they rebranded their elasticsearch type product to have a different name, you know, but under the hood it's probably the same thing. 1:09:43.680 --> 1:09:44.800 Yeah, yeah. 1:09:44.800 --> 1:10:03.230 But so not to say it's all doom and gloom though, I think databases are the truth, trickiest area. But if you look at like WordPress is probably like one of the big successes in my mind. Like I think like all of the Internet runs on WordPress if you look close enough. 1:10:03.230 --> 1:10:04.070 Yeah, that's right. 1:10:04.470 --> 1:10:15.430 And it's completely open source and all they do is offer hosting for WordPress and people, you know, are happy to use WordPress.com because, hey, they're good at hosting WordPress. 1:10:16.070 --> 1:12:22.140 Yeah, WordPress is a great example. I'll, I'll give a little shout out to my wife. My wife has a site called, what is it called? I'm totally failing here. It's called Lone Star Family Fun. And so my wife has a site where she posts like fun things to do around Texas for kids. Right. So for families. And, and so I was helping her set it up years ago and I knew that, you know, WordPress is going to be the thing because she didn't have to do anything that complicated. And WordPress super popular. But when you search for WordPress you originally get bluehost.com and some other ones. And so this is similar to what we were saying with Amazon where someone is hosting WordPress and so you could say they're kind of undercutting WordPress. And we had, we ran Bluehost for less than a month and then we ended up switching to WordPress proper. And it was ultimately because of, of bad service. So, so this is like, this is where, you know, what is it called? Like customer service is key. So WordPress, even though it's either open core or entirely open source and anyone can run their own WordPress hosting site relatively easily. They have just phenomenal customer service and they've built a moat through interacting with a lot of people who want to start websites. So for example, in my case I was there helping my wife out, but we'd go to bluehost and they're asking all these really technical questions, questions and what's your DNS entry and your C name record? And I was able to do all this, but a normal person would just, would have a lot of trouble there, right? Someone who's not really tech savvy. You know, WordPress has all of those battle scars from putting up so many sites, right? And so they, the WordPress site creator, like almost knows what you want and it's like an eight ball, magic eight ball. It narrows down what you want and then gives it to you. So all of that is to say that you could easily build, you know, an open core, open source project that is, and monetize that project through just, you know, relentless, you know, care and love for your customers. 1:12:23.659 --> 1:13:52.950 Yeah, totally. I feel like it's interesting if I were building an open source product project, like I like to make like command line tools and stuff, right? So I could start making that. If I were following some of these things we discussed, right? If I might want to build up a community so like get on Twitch or something and like show people how I build it, build up a following around that, make it open source. But then like, it's hard past that getting supporters. Like, how would you get people to pay for a developer command line tool? Well, people don't really pay for developer command line tools, right? Like that's not. Maybe some. But the next thing I would probably try is like, oh, are there like software as a service components that I would add? Right? Like, is there, okay, it's a command line tool, but maybe there's something I can do in the cloud that kind of alleviates some of the burden. Like with Earthly, it's a build tool and so you can use it in Jenkins. But our ultimate plan is to have our own, like kind of build servers in the sky. And because of the way we've structured things, we will be able to build people's software much faster than, you know, circleci or Jenkins might be able to do. And I think that if you want to build a business around open source, you kind of have to think about that, like, okay, I want to build something really cool that's open source for people to use. And then if I want to take it to the next stage, like what do businesses you know, what are they willing to pay for? What are the problems they have? 1:13:53.910 --> 1:16:14.260 Yep. Yeah, that makes sense. I think too, with open source, I got a lot of pushback when I added instrumentation to a terminal terminal, you know, people were very sensitive to that. And to this day, you know, every now and then we'll get an issue, you know, now it's like every time someone. This is kind of like a bit of a tangent, but anytime I see an issue that's related to documentation or something like that, I always tell the person like, file a pr and even if it's redundant documentation, it means that person didn't find that item where, you know, we put it, right? So, so file a PR and whatever you write, we'll put it into the site. And so now most people are pretty copacetic with it. They can turn it off, but there's a lot of pushback with adding instrumentation and there's no way to hide it. It's an open source project, so everyone can go to the code and see, oh, you know, when there's a crash, you know, it goes and sends a crash to some web endpoint somewhere. So. So I got a lot of pushback for that, but I'm really glad that I did because even though I'm not making any money from Internal Terminal, it still matters to me, the developer experience. And so I still strive to make that better with a little time I have that I can spend on it. And so in the case of Eternal, there are clear things that we can do to make it better. And I could come up with like a top 10 list of things that I would do to improve it. And so you need to do that, collect that data and focus on giving people what they want and what value matters to, to them. And you'll hear a lot of things we talk about, talked about already. You know, there are things that don't really matter to the creator very often, like auditing. Very few people who make open source tool care how many times someone goes to this part of the tool. But companies care, right? And so when you're creating value for somebody, that might mean building things that aren't that interesting to you, but are really important for that company and allow you to be funded to build everything else. And this is similar to going back to your example, Michelangelo. People at the time did a lot of religious and cultural icons and these things because that's what brought in the bills, right? And a lot of these people were really religious, so they're not cynics or anything. But that's what brought in the bills. And then they could do their artwork and their craft and do both of those at the same time. 1:16:15.140 --> 1:16:38.860 Yeah, I was just thinking like if you wanted to make. Yeah, like what? Imagine you got a lot of requests for Eternal Terminal that they wanted like some way to have a central server so that they could both, you know, so they could punch out through firewalls by, you know, reaching out instead of in. Then that might be like a commercial, you know, add on like. Oh, you'll somehow reflect the connections off some central server. 1:16:39.410 --> 1:18:23.270 Oh yeah, yeah. Like, like what is that called? You know, when they do the WebRTC and if you can't do a punch through a turn, like a turn server that like reflects the information. Yeah, I think, you know, for companies using a terminal, I think a lot of them, you know, want auditing and want the ability to say, okay, who logged into this machine and for how long? And that's definitely a business need. Another example here of someone doing this right is, and we had them on the show is Vercell. You know, so Vercel makes Next JS and they also make a hosting system called Vercel. And so you can do like next Next up or something like that. And it will by default pick Vercel, although it's open source, so you could pick other things and it would take your site and publish it to Vercel. So aws and so Amazon's going to come off looking like a jerk in this podcast. But it is what it is. So Amazon made their own version of Vercel called Amplify and the user experience is just terrible. Like it doesn't work. I ran into so many errors. I would go through GitHub issues and find like issues with the same error and so many thumbs ups and no progress. And so they just didn't have that really smooth developer experience. And so even though, you know, there's a David and Goliath example, you know, even though Vercel is, I want to say maybe 200 engineers, maybe that's being generous, they're able to provide just a really nice experience. And so, you know, as an end user, I tried Amplify and went back to Vercel, like almost right away because Vercel's never not worked, even as I've updated the project. 1:18:24.070 --> 1:19:35.740 Yeah, that's a super interesting example. And I think, yeah, if you're building an open source project because like you want it to exist in the world and then like, okay, I can put more resources into it if I Get funded. I think you approach it differently than an organization that's like, oh yeah, another thing to run in our cloud. Yeah, like I think you're much more invested in the process. Yeah, there's a cool tool called Buff. It's like with one F buf. It's like a GRPC and proto buff Linter. And so they have, it's open source, their command line tool, but then they have some additional features where like you can document things or I forget what their, I forget what their feature ads are, but they're all like kind of hosted on their website. Right. So there's kind of like a. Oh like a schema registry for your protobuf files that you can host with them. But like I talked to some of the developers there and they just care like really a lot about proto buffs and the people have been using them wrong. And you know, here's problems that people hit with grpc. I think that really the real secret to funding an open source project is like that it's something that you're really impassioned about and you want to talk to people about it and you want to make it better. 1:19:37.020 --> 1:21:29.440 Yeah, that makes sense. And I guess so maybe we'll end it by just talking about, we'll take turns talking about how you can get your project funded or what it takes. And I think we talked very mechanically here. But from my perspective, a lot of people who build things are people who love systems and love things. And so it's not intuitive to then sort of go out and solicit. It's not intuitive. Sort of be like a product advocate and go out and solicit advice and build the product that other people want. Right. And so most products, like look at Red Hat for example, it started as like a hobby project where this person was building something themselves. And so then at some point you have to sort of look through, look behind the mirror and build the product for everybody else. That's a hard transition and you see it everywhere. I know musicians, I have friends who are musicians, but they're only making the music for themselves. And, and so the, you know, it just, it's kind of like a hard thing to explain for music. It's like, it's much more clear in software, but it ends up being, you know, my friend's music ends up being like super technical in certain areas that most people can't really appreciate. And it's sort of like climbing on top of challenges that they faced in their mastery of their instrument that like no one else can really Relate to. Right. And so the same kind of thing where, you know, you're going to have to really be good at starting on your own or with a small group and building something that's going to take a long time, many hours by yourself in your basement, but then also have those skills to go out, ask the audience, solicit feedback, and build an ecosystem. So to have a really successful financial open source project, you have to do both of those things, either as a partnership or by yourself. 1:21:31.600 --> 1:22:42.820 Yeah, I totally agree. And yeah, the key in my thinking, if you're having an open source project, I built something and I think it's kind of cool and I throw it up for myself. That's great. If you really want to build a lot of traction, probably it's just important that you are solving a problem that other people have. And oftentimes, you know, that's just the case. Like, you know, I have a problem and I build a tool for it and other people have it. But the only real way you can find that out is probably by talking to people. You know, like, just like ask, you know, other people who you think might use this project if it would be helpful. Like, you know, earthly. You know, I kind of mentioned that problem of like having the build fail in CI and trying to have to poke with it. You know, there's many solutions to that problem. But I think what's important is that, like, usually when I talk to people and I have that complaint, like, they immediately understand, like, oh, yeah, that's a pain. And so I think you want something like that. You want to have a problem, like a clear problem statement. You're like, a lot of people have this problem. And even if I'm not going to make a business out of this, you know, I just know if I build this open source project that it will serve a lot of people's interest. 1:22:44.900 --> 1:23:02.410 Totally makes sense. I think it's a phenomenal way to end it. So, Adam, thanks so much for coming on the show, being a guest host. We've had a lot of guests. This is our first time with a guest host. I think it was awesome to have you on board, a really great episode. And thank you again for putting in the time and coming on the show. 1:23:03.200 --> 1:23:09.760 Yeah, thanks for having me. I apologize. I got a bit of a cough and I took a lot of cough medicine. So hopefully everything I've said is coherent. 1:23:09.840 --> 1:23:49.360 It sounds totally fine. A lot of folks don't know that, you know, for a duo episode, you know, an episode that isn't. And even for the interview episodes. We do a lot of prep work. There's a lot that goes behind the scenes on really any podcast and ours as well. So it's a huge effort. Thanks again, Adam for coming on and a phenomenal episode for folks out there. You know, we have a couple more interviews lined up, but we will probably do a couple of duo episodes as well over the next couple couple of months. And thanks so much for listening for supporting the show and if you want to catch Adam's show, this show is called the co Recursive Podcast. It's phenomenal. I recently interviewed. We talked about the one we that you mentioned before we started recording. 1:23:49.440 --> 1:23:49.880 Yeah. 1:23:49.880 --> 1:23:50.960 Is that live? Okay. 1:23:50.960 --> 1:23:54.320 Recently it'll be. It'll be May. May 1st or 2nd. 1:23:54.720 --> 1:24:06.300 Ah, cool. So. So you'll hear an interview between Adam and a gentleman who worked on the Mars Rover, which is super, super cool. So thanks again, Adam. Thanks folks out there for listening and we will catch you on a couple weeks. 1:24:19.580 --> 1:24:24.260 Music by Eric Barndaler Programming throwdown. 1:24:24.260 --> 1:24:39.430 Is distributed under a Creative Commons attribution Share alike 2.0 license. You're free to share, copy, distribute, transmit the work, to remix, adapt the work, but you must provide an attribution to Patrick and I and share alike in kind.