WEBVTT 0:00:15.134 --> 0:01:13.437 Programming Throwdown Episode 176: ML Ops at SwampUp. Take it away, Jason. All right folks, so this is going to be a bit of a different episode than folks are used to. I'm actually at the SwampUp conference, which is hosted by JFrog, and you know, a lot of people actually probably the number one email or message that we get is, 'Can you explain DevOps? ML Ops? Can I get a job in these areas?' It seems like a really good—people see it as a really good gateway to, you know, getting into tech. And we get tons of requests for it. So I had this amazing opportunity to talk to four, you know, different people with very different backgrounds who all kind of ended up in this discipline and can share their stories with us and also explain the technology, explain what DevOps is, explain what ML Ops is. And so, you know, we're kicking this off with James Morse from Cisco, so thanks for coming on the show, James. 0:01:15.209 --> 0:01:24.520 No, thanks for having me. Cool. So why don't you kind of tell us—actually, let's start off with what do you do at Cisco? What is your title and what are you responsible for? 0:01:24.520 --> 0:02:02.374 So probably a really high title. I'm technically DevOps Engineer, categorized right now under Self Software Engineer, which is perfect segue because of the blurred lines it's just everybody's, you know, constant battle like you're referring to. But yeah, so DevOps Engineer for a team currently called Enterprise DevOps as a Service. So we actually provide DevOps services to other teams within Cisco, try to ease that segue for them whether they aren't quite on a DevOps path or really just need specific services that fall under that umbrella and just supporting those systems and helping those teams. 0:02:03.184 --> 0:02:08.770 That makes sense. So give us a little bit of your background, like how did you get into where you are now? 0:02:08.804 --> 0:03:35.040 Sure. Yeah, so I went to Guilford Technical Community College, which is in North Carolina, and got a degree in both Computer Information Systems—sort of general IT degree, all kinds of stuff under that umbrella: server administration, Windows Server, Linux systems—and then also decided on the parallel path for getting the Networking Technologies degree, which coincidentally was most focused on Cisco hardware. So routing, switching, kind of think of the path for getting like a CCNA certification. Now what is CCNA? Cisco Certified Network Associate. They've kind of floated around either associate or admin—there's a few different—and there's CCNP for professional, different layers, and you know, kind of starts at the bottom. And I think now they've split it into CCIE, but I've long since been in that, so I'll embarrass myself as an actual Cisco void out. But yeah, it's the entry level at the time for me, which I did get that degree in the past was again your basic like setting up, you know, routers, which is under Cisco's umbrella, so makes you learn how to, you know, do subnetting by hand and do it literally on a Sharpie and a dry erase board. Oh wow, yeah, yeah, I was pretty intimidated, but you learn a lot, and it's a lot. It still helps me today, you know, configuring systems and I'm just having that, you know, understanding of, you know, IP communications and setting up different subnets and gateways, and so cool. 0:03:35.040 --> 0:03:46.172 And so at some point you were like a system administrator and then you transitioned to DevOps. So what is the difference there? Like what's the—yeah? 0:03:46.172 --> 0:05:44.419 So when I first started, you know, I graduated around 2009, and then quickly was placed at a company doing—title was VoIP support something along those lines. And it ran the gamut. It was pretty much somewhere between support technician, you know, taking, you know, first-level calls, but also because it was a smaller company doing what traditional sysadmin work was then. So that was something, you know, around the systems were all Linux-based; they were CentOS at the time, and you know on top of that you had different applications installed for, you know, running various things, mostly Voice over IP software. So all kinds of system administration at that time like mostly in that company it was—they were wanting somebody with the Linux background, so Linux sysadmin typically, and probably still to some degree today, sysadmin work is usually split somewhere between like you're specializing maybe in Linux, Unix, all those types of systems, to the, you know, more Windows side, Microsoft side. Microsoft Server, those types of things. Obviously now it kind of runs the gamut because you have cloud services and things like that. So you could still technically maybe be system admin or, you know, some type of Cloud Engineer and not really be, you know, but you're DevOps or DevOps Engineer. The transition there was—I think about a year into that company was when I started hearing the buzzwords about cloud. Like the CEO I worked under was very much trying to, you know, keep his finger on the pulse of things and like to be, you know, very bleeding edge as far as any kind of new trends and try to, you know, capitalize on that help people that were trying to maybe get to the cloud. What did the cloud even mean at the time? Was was sort of, you know, almost the DevOps of now. Like people who are in the industry hearing 'cloud' even now—maybe you know, you might hit somebody who's not in the industry and kind of get what that means. But I rarely run into somebody who's not in the industry who's just like, 'Oh, I know what DevOps is.' Right? Um so that was sort of the DevOps term then. It was the buzzword. So that very quickly led into— 0:05:45.140 --> 0:06:14.402 Some blurred lines of, you know, what I would consider CI/CD Continuous Integration and delivery or deployment. And that wasn't still true like what I would call DevOps, but it was starting to go the route of more automation. So less like, 'Okay, I need to deploy something that might involve, you know, opening up something like my IT was like WinSCP and then copying some files over at the right time. Okay, we're gonna deploy on this, you know.' 0:06:15.972 --> 0:07:19.641 Sunday way off some weird hour, 11 PM, and then copy these files over, maybe restart some services. Hopefully you don't have a lot of those if you're a smaller company, maybe you have to do that, you know, in some for loop for, you know, hours if you do have a lot. So that transitioned to things like, you know, a lot more scripted stuff. Things like Ansible were starting to come up where you can just, you know, write some playbooks, run those playbooks and those things would happen automatically. And so that sort of blurred the line for me at least some getting into DevOps where it was more, 'Okay, you've got your code and hopefully a good Git and something, you know, most famously like GitHub, and maybe I have a branch something like production or main, something along those lines. I'm working in my feature branch and I get, you know, merged into dev, and that dev gets eventually maybe gets, you know, reviewed—hopefully gets reviewed by, you know, one or two people—and so on. So like, you know, maybe is slated to be merged for production.' And then in my idea into true DevOps, there's at 0:07:21.565 --> 0:08:17.522 Least at some point where you're going to merge that into, you know, your production branch, whatever that's called. And a lot of things are going to start kicking off automatically. So instead of this guy who's just waiting for whatever time and he's just copying frantically logging into systems and clicking and dragging if it's, you know, UI based, you're running for loops and, you know, something like a Linux CLI or terminal, it's going to should be a lot more automated and maybe have some checks and balances. So, you know, if this thing fails, you know, don't do this, and so, you know, kick off an email or maybe, you know, a lot of people use Slack or, you know, things like we use—being at Cisco, we use Webex—and maybe a post into there that something succeeded or it failed. But there shouldn't be a whole lot of manual if not any in a really ideal situation manual intervention or, you know, manual—some people have deemed clickOps where you're just, you know, in there manually fixing things or changing things that makes 0:08:17.809 --> 0:08:39.140 Sense. So would it be fair to say like system administration is—is it fair to say that it's the part that is left over because it's either not worth automating or you just haven't automated it yet? And DevOps is this automatic tool chain that covers the rest? Is that like the right way to frame it? 0:08:39.140 --> 0:09:30.321 Yeah. So there's and that's why where a lot of people you know kind of get in this philosophical argument about, you know, there's so many things that kind of came about around the same time. So when I first started, you could, you know, all the things now that sort of fit together—definitely DevOps, CI/CD, even Agile development weren't barely even being talked about. I mean again, I'm having some light buzzwords like cloud; some of those may have been being pursued from an idea but not with those phrase, you know, that phrasing of those terms. And literally what and within my career watch it actually evolve into those things where people are using those terms daily. So a lot of those things in my mind get blurred because they sort of all came out at the same time although technically they're completely separate. You could be doing Agile, for example, but not actually be using any DevOps services. 0:09:30.709 --> 0:11:24.020 Imagine that world I've never seen it in person. I've seen, you know, maybe DevOps not being used ideally or having some pieces missing that would make it better. Um, similar to CI/CD, you could be doing technically doing like Infrastructure as Code, like something like Ansible, and maybe even doing it with CI/CD or like the Git flow that I was talking about, but not really still being doing all the other parts that come with DevOps. Um, yeah, they don't have to go together, but sort of the complete package in my mind is doing something like using an Agile development and then kind of pairing that with CI/CD and DevOps workflows. But in general, the idea is of course that you're using like the most common tools for DevOps would be something like GitHub—GitHub Actions, right? So now I've that merge I was talking about that's getting kicked off maybe by an Actions workflow, and that Actions workflow just means like, for example, if I'm using one of the examples earlier, you know SwampUp, I think in the one of the trains they were using—it's pretty common Node.js, NPM. So that NPM build might be kicked off actually by Action, so somebody sitting there actually running all these NPM commands and then, you know, pushing that somewhere, you know maybe like JFrog Artifactory, and then some system then again pulling that down to be in production. You could do all that manually and still using those tools, so DevOps isn't just using those tools, right? And CI/CD isn't just having that it flows sort of the—again, like there's this chart that usually comes out or visualization rather where it almost looks like an infinity symbol where you know as you have that deployment, they you want that immediate feedback. So as soon as you've got a deployment and you're already sort of kind of on the path to the next release because it's sort of this, you know, smooth transition, right? So you can have lots of minor iterations and sort of break. 0:11:24.020 --> 0:13:14.759 The barrier that existed when I started with traditional C7. I rarely talked to any developers, you know, unless there was a problem, right? Systems down or whatever that was. Sort of, you know, the typical Mo versus—oh, I just need maybe some change or some help on, you know, get up Actions. There's a there's a problem. I need new permissions or something. But if they're given all the right permissions and they've got, you know, management and checks and balances on things like, you know, PR reviews and things to get into that automated production workflow, then that's where that DevOps term in my mind kind of comes out because the developer side—the development side and operations Ops are sort of blended. Because as a developer, even though maybe I don't know too much about like the kind of stuff I used to do Lennox administration and Lennox OS and like how these files get where they need to be, do they have executable bits? Like I'm not worried about all that. I just know that if I merge this PR, you know, domain production whatever it is, that all these things kick off, and if there's a problem I don't understand, maybe I call, you know, DevOps support or, you know, whoever it is. Maybe it's even just a technical lead or somebody that knows like, oh, that error just means this. You know, start a new feature branch and fix this and try it this way and that. Yeah, that kind of takes that out of the aspect at least in that flow. It doesn't mean that nobody's setting up those systems that are running it because that's kind of where my current role, you know, as this comes in is like we're, you know, deploying things. Like we're here at JFrog SwampUp, my main role is managing deploying and maintaining Artifactory itself, and that's still in some cases on a server. We both have a server and the cloud and all kinds of different deployments those still have to get deployed by somebody. But once all those things are kind of interconnected, they allow that continuous workflow for that developer to not have to worry about those things. Yeah. 0:13:15.467 --> 0:14:42.660 That totally makes sense. I remember like—I've seen this through my own career. I mean, I've mostly been kind of on the research side, but I have my wax wings have put me a little bit too close to the sun every now and then. I have been burned by the production gods every now and then. As we and yeah, in the early days they would—earlier days of my career—they would build a new build every week, and it would be a manual process. And there was an IRC channel, and you would just get pinged like, 'Hey, you know, you made this change, and this file, you know, doesn't compile. You go take care of it.' And then it kind of as you said, it's become more and more and more automated to where most companies are just pushing new versions all the time, and they're even automating the failure. So it's like the contingencies are all automated. And so now what you're doing is is just orchestrating this self-healing machine, which is really impressive to watch and to be a part of. So now it's as you said, it's more like you get a Slack notification—'Hey, you know, this file that you changed that you know, you is actually being used in this other system you might not even known about, and it broke.' And so you need to revert it. That kind of thing, but it's very automated now. It's really cool. 0:14:42.660 --> 0:16:29.378 Absolutely. I think automation is a huge part of it for sure. Um, and then it's also the sort of the interlinking between those automations because automations, you know, predated DevOps and the terminology there, but I think the true thing is, you know, kind of going back to that infinity loop diagram or visualization or whatever you might call it that a lot of people bring up, and I've seen it a couple times even here this year. And it's not last year. I was lucky we got a small book last year, and similar that it's that how they're all intertwined now when they're all playing together well and then they've been configured, right? And you're taking advantage of all the modern features. You really shouldn't have to be doing much unless there's again there's a problem. A lot of cases now, a lot of what has been showcased in both last year's top of and this year, the big focus on security. So there's the term now DevSecOps and several similar type terms. But the idea that somewhere in that flow that we were just talking about something should be scanning, right? So some failure may not actually be that somebody technically did something wrong; they just did something they shouldn't have. You know, maybe there was a secret in something, maybe there was an old version used for something going back to that NPM, maybe they used a version that's got vulnerability in it. So you want to see that fail. So somewhere in that pipeline, you either have a plug-in or some version of, you know, if it's good of Actions—maybe there's an Actions for, you know, like JFrog, but if we're here they have their own called Frogbot that inner, you know, has a lot of integration with Artifactory and X-Ray where, you know, it's making sure those vulnerabilities, you know, don't exist. And you can even have it, you know, block the pipeline or maybe just email somebody if it's, you know, the type where maybe it's just a development flow, you don't want it to block, but you do want somebody to know, 'Hey, this should not go to production.' 0:16:30.171 --> 0:16:48.980 That makes sense. What about like—you know, you have this really expansive background in DevOps, and so how do you see MLOps from your perspective? You know, what is the sort of delta there? Um, what what is the part like what's the leap from one to the other? 0:16:48.980 --> 0:18:30.899 It's a really interesting one because I have—I've been learning a lot. There's been a lot of focus on that with this SmallOfUps. I've learned a lot. The keynote was very good in that. I think they did a good job with their diagrams on literally showing those two differences. So for my background being very since I was more sensitive like a lot of people, it's interesting especially with the term. So people come in more from the development side—I want to see more operations. So they get into DevOps from that obviously. I would the other boat which is people with SysAdmin backgrounds or some version of operations getting in and then you know the development side kind of comes in there, so not having much background with the ML side or AI. I can at least say that what I've learned already and even within today's talks is but there's gonna be a lot different workflows. So there's a lot more parts and pieces with everything. I was just talking about—you're not just going to have like, 'Oh, got my get repo, I've got my code, I push it here to run some commands and build stuff,' and eventually as long as it gets you know enough actions or if you're using Jenkins and you've got the right Jenkins plugins things going to get my app deployed. Not to say that's simple and to downplay some of the more advanced apps when you're talking MLOps, there's a lot more at play. Like am I using a public model from something like Hugging Face? Am I, you know, something being developed in-house completely um from scratch and have the resources for that? Obviously, there's so much out there that's a lot less common but still plenty of people especially depending on how sensitive the work is going. So there's so many more moving pieces, but I think the idea is still the same. You're just going to see especially as JFrog is already integrating with some of that. 0:18:32.532 --> 0:19:24.000 That it shouldn't be as awkward as it was in its infancy where a lot of that's going to be customized stuff like now. It should be again a very similar workflow, but my short answer would certainly be there's gonna be a lot more pieces there that you're gonna have to integrate, like store and even just from a storage perspective, like the models are a lot larger than they just pulling maybe some Python libraries, right? A lot of most people have worked with Python. They've run an import doesn't crash your machine, doesn't fill up their hard drive, right? Doesn't take many models to you know fill up at least, you know, a modern slim book, you know that where, right? It might just have, you know, 500 gig or something on your hard drive. Maybe three or four models could easily fill it if not just one large one. Yeah. So now you're thinking, 'Okay, we're gonna have these properly set up to be stored correctly and optimized.' So there's a lot more at play at the very least than just a standard DevOps work that. 0:19:24.422 --> 0:19:53.245 Makes sense. And so we have a lot of folks listening who are you're just starting their careers. Um, you know what advice would you give for people? Let's say, you know, folks are in high school or maybe folks are thinking of going back to college and they're deciding between that and a coding boot camp or learning on the fly. Um, you know what advice would you give for young folks who are just getting started if their goal is to get into a DevOps career path? 0:19:53.295 --> 0:20:26.320 I've seen that question a lot alive even a couple friends personally have asked that questions. I've given it a lot of thought. I think, you know, depending on what your background is, it's gonna really sort of change that answer at least to some degree. Um, but I think the one common thing is certainly to just don't take for granted the wealth of information that's out there, whether it's, you know, just free resources like YouTube. There's tons of, you know, great videos that teach you all the different pieces of things like GitHub, GitHub Actions, Jenkins. Um, you 0:20:28.665 --> 0:22:12.987 know just all the different, you know, you would—I wouldn't necessarily get too caught up in, 'Well, I need to master something like a specific language,' whether it's Python or whatever. But it's going to take at least the common core understandings of some programming language and having, you know, the basics of that because no matter what you're doing even on the side that I am or, you know, maintaining something like Artifactory, you're going to have some language whether it's for scripting or for automation. Um, so having some good basics of that and then also just again taking advantage of all the information that's out there on things like YouTube and Udemy um that are very affordable, very accessible. You know, when I started, you would go find the closest relevant O'Reilly book, right? And yeah, have it by your desk. It wasn't very common to see just a few O'Reilly books or a whole shelf of them, you know, in whatever team's room that you worked in. Um, and you go grab one off the shelf. Not to say that it wasn't available, but that was still what sort of the habit like. Yeah, instead of searching to retrieve of maybe like various posts and things like Stack Exchange that, you know, at least early on it may have been more difficult than now, especially with things like ChatGPT or anything GenAI. Um, so don't take for granted the amount that is out there. You know, just use that to your advantage. Um, and at least learn those most common tools, you know, GitHub and some type of version something like Actions or Jenkins um that helps you automate those things and just start playing around a lot of those things, right? Now are um free for open source and like, you know, students and things like that, so it should be able to at least break in at, you know, minimal cost or, you know, even free in some. 0:22:13.493 --> 0:22:25.660 Cases. Yeah, totally. Yeah. Um yeah, I use GitHub Actions extensively for a lot of my open source stuff and they haven't charged me yet unless there's some mounting bill somewhere, right? Yeah, unless they get a collections call tomorrow, we'll see, but yeah. 0:22:25.744 --> 0:22:27.330 So that's the great thing like. 0:22:27.330 --> 0:22:27.972 When you're learning, 0:22:27.972 --> 0:22:55.292 Like you feel like don't be frustrated when you break things. Part of it is like break it. See how you can break it, you know? Break it different ways and learn how those, you know, broken things are solved and and how you can, you know, figure out change ways to keep the automation going even if it hits, you know, something. It's not critical, maybe it was meant to, you, you know, continue on whatever. And that's sort of the idea. Like don't don't get frustrated as things failing. Like figure out and use that as a learning opportunity. 0:22:55.799 --> 0:23:44.247 Basically, yeah, definitely. Yeah, this is great. So I think, you know, one common thread that we've talked about on the show that we're just double-clicking on here is, is, you know, build cool stuff. That's basically the short of it. If you build awesome stuff, you're going to have to maintain it, I guess. That's one thing with DevOps is, um, I guess you'll have to build something for other people or build something where, where you know, you need to have a process. But even if you don't, I mean, you're still going to get a great experience as a developer and that will catapult you into into, uh, you know, DevOps career somewhere. So yeah, absolutely cool. Hey James, I know you have to rush. Thank you so much for taking the time to talk to us. I really appreciate it. And if I get any questions for you from the audience, I'll shoot you an email. Yeah. 0:23:44.415 --> 0:23:47.487 Always have any help. And yeah, it's great being here and appreciate. 0:23:47.790 --> 0:23:48.668 It happening. Cool. 0:23:50.271 --> 0:25:48.580 Thanks a lot. Hey everybody. So we are here at Swamp Up with Luke Marsden, who's the CEO of Helix ML. Thanks for coming on the show, Luke. It's great to be here. Cool. So um why don't you give people a little bit of a background into, you know, how you kind of arrived into starting Helix ML and what's your kind of backstory? Yeah, absolutely. So um I'm a startup guy. I've been doing startups my whole life. This is startup number three. Um back when I was 15, I started a web hosting company. Wow. And I then went and did Computer Science at Oxford, and out of the back of that, um that experience, I was really inspired to try and solve some of the practical problems we had in the web hosting company, and I did that by building a distributed web cluster. Um and then that evolved into we ended up pivoting that business into solving storage for Docker because when Docker exploded, we were like we already have all of this tech for dealing with stateful containers because we were using FreeBSD jails, and so we just applied that technology to um uh to Docker. Well, let's dive into a little bit. So we have a lot of high school folks listening in. So 15 years old start a company. How does that work? How do you go to somebody and say, 'You know, I will build this website for you,' and you should write me a check? Did your did your parents help you with that or did you like how do you do that as well? I mean, I had a co-founder who I met online on IRC, and um that's uh like the old school way that we used to communicate back into. I used to love IRC. It's great. I mean, I guess now it would be Discord, right? Yes, that was the Discord of our era. Yeah, exactly. Um and um yeah, we just decided together to put together a web hosting company, and the amazing thing about that was that you could—we were just able to put it online, put the website up there, um and start telling people. 0:25:48.580 --> 0:27:46.660 About it, and people showed up. Um and one really fun thing happened that that helped us a lot with that business was um I've forgotten what it's called, but there's like this old Perl-based blogging framework, and the guy who created that blogging framework found our web hosting service, and then he left his uh company and then promoted and then he said, 'I'm really happy with my with my hosting service,' in front of like all of his audience. And so oh, we got a nice boost in traffic from that. That's amazing. Yeah, yeah, it was very lucky. I don't know if I've ever told this on the show, but but uh I did a lot in high school as well, and one of the things I built was a little isometric game engine. Oh yeah. And at the time there weren't a lot of game engines. This is this is uh like 1997 or something, and so there wasn't a lot of Java really anything. Yeah. Um and so it got picked up in a book and it got popular, and I wonder if I wonder if people can just build things and get them noticed, or if that time has passed. I feel like now you probably need to a little bit more promotion. There's not there's just so much content out there now. I mean, I I think that's true, but um I think if you have a passion for solving a problem, then um give it a go. And if anything, there's probably more ways of getting the word out these days. I mean, we didn't have YouTube back then. Like we didn't have um Discord communities like I think that's true. Um uh and more people are online now than we were then. So yeah, another thing is like there's the the Vanguard has moved. So in other words, if you want to make a Java game engine—I'm sorry, like it's just there's too many of them. Yeah. But but there's a Hugging Face LLM leaderboard. Yes. And if you know and there's a variety of them, so you might end up, you know, with with uh, you know, some passion. 0:27:46.660 --> 0:29:00.417 And some talent. You might win like the the uh, you know, Greek LLM leaderboard. You could reach the top or something. So absolutely. So shall I continue my story? Um so yeah, I mean we we did the um the storage for—we pivoted that that business uh hybrid cluster into ClusterHQ, which was solving storage for Docker. And then we got involved in the very early Docker and Kubernetes days back before um people really knew how to make um containers work with like databases and other stateful services. Um so at that point, um yeah, we raised $15 million dollars just by walking up and down Sand Hill Road and saying 'Docker' and storage in the same sentence.' Now what is Sand Hill Road? Oh, so it's um it's a road uh in Palo Alto where a lot of uh the VCs um live or work. Yeah. So uh yeah, it was a large concentration of venture capitalists. 0:29:02.678 --> 0:29:13.460 it was it was interesting and i think i learned a lot from that experience um i think the biggest thing i learned was that um 0:29:17.300 --> 0:31:15.957 even if you have like a large series a you need to be really really thoughtful about um not growing the company too quickly before you really truly have product market fit um and so uh yeah that was that was one of the lessons learned like we grew the team quite quickly and uh uh and and it was challenging but but then beyond um uh that first company uh i then had another go um we had a uh we built um we we we started out with with this idea of um versioning for uh development environments okay so um um a lot of my career i've spent like trying to find commercial applications of zfs which is a very clever bit of file system technology that came out of some microsystems and then got ported to linux yep um and and so the that attempt to commercializing zfs was well if you've got a development environment and you manage to reproduce an interesting bug let's say on your laptop um then shouldn't you be able to um not just do a git commit of the code at the point at which you can find the bug um but also uh take a snapshot of the local development databases that you have running and so that you could maybe attach a runnable snapshot of that thing to a github issue and then another developer could just pull it down and like reproduce the bug immediately rather than like having to click around in the ui to right get the database into a certain state um turns out that uh ai and machine learning had a much bigger data versioning problem than devops did and software engineering and so we ended up pivoting to building out this end-to-end mlops platform but with that same idea of versioning your workspace and so what data scientists like ai and ml people often do like when they're developing ideas is they use jupyter notebooks but 0:31:16.075 --> 0:33:09.780 when you're using a Jupyter Notebook, it's very hard to keep track of your work very accurately. Um uh and even like the order in which you run cells can affect the output and things like that. But what we did was to add this um snapshotting of your state uh before and after you did a run. So if you did anything in the Jupyter Notebook that would like train a model, for example, um then we'd snapshot before and snapshot after, and then we'd also automatically build up this provenance graph so you could say like, 'Oh, I created this model from this data,' but this data was transformed from this other data using this process. Um And so you can kind of recursively build up this um uh this tree structure of kind of how you got to that point. Um So yeah, that that was uh that company was called DotScience. Um and um yeah, uh that was I love that name. Is it like the configs were in the DotScience folder? Your home directory is that where the name came from? It actually got really confusing because people would always put like a period and then science, right? But it was actually dot. So it was oh, that was a lesson in naming. Yeah. Department of Transportation Science. Yeah, I could see other ways that that could get misconstrued. Yeah. So a lot of people might not know this, but ZFS and uh Btrfs—yeah, root FS—um they have this amazing property called copy-on-write. And the way it works is uh you've probably done this even for school projects, and you might say like I have a document. Um I mean nowadays probably everything's in the cloud, so it's kind of transparent, but you might say like I have i have a set of uh artwork that I'm working on, some digital art, and uh you know I'm iterating on it, but I don't want to lose my history. And it doesn't really make sense to create a Git repository. I might not know how to do that. The simplest thing would be just create 0:33:09.780 --> 0:35:09.270 a folder for each day, yeah? And just have my uh you know Blender files or my Photoshop files just, you know, copied in each folder. And then that starts to become really expensive because you might have all these other assets and they're just all getting copied every single day, end up using a folder disk space. Do that exactly. ZFS has a genius idea. It's basically reference counting. So when you copy a file, ZFS doesn't actually copy it; it just creates a shared pointer between those two files. Um Now as soon as you change even one bit of that of either of those files, then ZFS has to make a copy. Um But if you don't do that, you can have a hundred, a thousand, million copies of the file and it's not going to increase your storage cost linearly like a regular file system. And so yeah, I've always been fascinated by that. Um I really was intent to install Btrfs on my latest uh computer that I got about a week ago, um but it wasn't supported by GRUB or some other issues, and so it's it's to your point like it's still not not it's still not totally, you know, smooth yet. Yeah. It's been around for a long time unfortunately. Yeah. Well, I mean Ubuntu embraced ZFS on Linux fortunately, and so you can install Ubuntu with a ZFS root. Um Right? So um yeah, I mean encourage anyone who's brave enough to give that a go, and you can do all these cool things. But I mean we run like uh my newest company, Helix ML. Uh we run um all of the infrastructure for that uh in a data center in my basement, really. Yeah, because uh we're bootstrapping the business so we didn't want to um to incur a ton of cloud costs. So I mean it's not a Fiber Leak, and um yeah, we use ZFS on Linux for all of the production storage for that, and then um actually rsync.net. This is an interesting piece. 0:35:09.625 --> 0:37:08.580 Of ZFS trivia: rsync.net has support for them. They're a backup provider, but they have support for like ZFS send into rsync.net, and then they provide that as a service. So what that does is it means that you can take a snapshot every day of your production system, and it's an atomic reliable snapshot that doesn't take up any extra disk space, like you were saying. And then you can just send the difference between that snapshot and the previous day snapshot over to rsync.net, and it will automatically keep up today. So anyway, we've ended up falling down a file system rabbit hole. No, that's fine. Okay, one last file system question because it's just burning in my mind: what is the difference between ZFS and Btrfs? Are they totally different things, or are they two people trying to do the same thing? Well, I like how you describe it as Btrfs. I think that's what it is, right? Or no, I think that's a joke. Oh really? I thought that was the official. I think it's called ButterFS. Um oh, you are totally calling it. I think you fell for it. You know, I have to admit, I am. I've said this on Show Local Times. I am known to be the most gullible person. You know, different people have their have their superhero strengths and weaknesses. Like I fall for everything all the time, so I'm not surprised. Well, no, I mean, I don't mean it in a negative way. I just think it's—I can imagine someone on the internet calling it Btrfs as a joke because it's like Bitrod, and I think that actually—I looked it up. It's ButterFS. Yes, that is the official name. Oh, or BtreeFS, which probably makes the most sense, right? Yeah. Well, I think Btrfs is actually quite apt because it's—yeah, I mean that project has had been plagued with data loss issues, and I've never really trusted it to actually be reliable. 0:37:08.580 --> 0:37:19.140 whereas zfs was engineered at some microsystems like very nicely um and i was grateful when it i was grateful when i stopped having to run solaris to use it 0:37:21.460 --> 0:39:16.740 But yeah, anyway, well that is fascinating. There were licensing issues, right? But I guess it's all cleared up. Oh yes. And I think the—yeah, I'm glad that Mark Shuttleworth Canonical kind of took a stand on that and he said like we're gonna go ahead with this. Our lawyers have cleared it, like really cleared the way for a lot of other people to say like okay, if it's good enough for Ubuntu, then we can—we can use it. Nice. Um that is great. That is Ubuntu's like real big contribution is just getting everybody to have faith in the whole ecosystem. I mean, so okay, we talked in previous episodes about DevOps. You know, we had a dedicated show about it. We've interviewed some folks about DevOps. For folks who are listening, if you missed the DevOps episode, hit the pause button on this episode, go back, listen to the DevOps episode because what we're going to talk about now is the delta between DevOps and MLOps. MLOps is much newer. You know, we haven't spent a whole bunch of time on it on the show, but as folks know, AI is becoming really important, and so MLOps itself is also becoming important by proxy. So what is MLOps, and how does it differ from DevOps? So if you think of three disciplines—software engineering, just like how you write and develop software, how you test it, how you version it—and you think about DevOps, which is how you do CI/CD for deployment, how you operate that, how you do immutable infrastructure in the cloud, and things like that. And then you layer in this third discipline, which is AI/ML, which is like this world of 0:39:17.380 --> 0:41:16.259 using data to generate models that pull the patterns out of that data and are able to make predictions, right? MLOps is the intersection of those three disciplines. So it's the intersection of software, DevOps, and AI/ML. That makes sense. And so if somebody you know wants to get started in the field of MLOps, you know how can they—you know how do they not get overwhelmed with all three? Because all three of those are—you could spend a lifetime. They're all crafts of their own. I think your data science and data engineering is a craft. Software engineering is a craft. You know, AI and making the loss go down, it's a craft. I've spent many years on. So how do people get in that intersection? Is it the kind of job where it's really more of a mid-senior level, or can folks kind of get into that, and what would that look like? I would say that in order to become an MLOps practitioner, you only need a little bit of all three. So for example, if on the software side you learn a bit of Python and you're comfortable with Git; on the DevOps side you get comfortable with Docker and containerizing things and maybe deploying Docker Compose, maybe push out into Kubernetes. And although I'd say Kubernetes is optional, and then on the AI/ML side, if you just like train a linear regression model or something in PyTorch or like XGBoost, I mean that—that's enough to get you started. And then you can start looking at tools like MLflow, which allow you to keep track of model artifacts and track runs. That was something we were big on in my second company, DotScience. 0:41:16.259 --> 0:43:13.940 This idea of run tracking, and—and yeah, like deploy one of these models into production using Git, Docker, PyTorch, or whatever, and now you're an MLOps engineer. And then you can take it from there, and there's like lots more sophistication in how you scale the systems and so on. But basically that's what you need. So it might sound intimidating because it's like three different disciplines, but yeah, if you just take a little bit of each one, then you can get up and running. And I would shout out to the MLOps community. So mlops.community is actually a community that we started at the end of DotScience. Actually, like right when the pandemic started, our sales pipeline dried up, and we were like what are we going to do? So let's start an open—let's start an open community around. Right? It's like I'd always wanted to do that. We bootstrapped that community from nothing with another tech community called Bristech in Bristol where I live in England, and then my colleague Demetrius took the MLOps Community and ran with it. And we're now like over 20,000 people on Slack. Wow, we've got a meetup in San Francisco this Thursday which we're hosting, and there's meetups all over the planet. So it's amazing how these things—Wow, that is remarkable. But there's tons of really great material on MLOps. On the MLOps Community, there's a good YouTube, and yeah, I'd recommend that as a resource. Yeah, that is great. I love how like you took such a kinesthetic approach to it. I'm right on the same page. I think—I'll read the book when you know I've kind of have something half built and it doesn't work, and it's like okay, now it's time to read the manual. I know some people are the opposite. I used to report to my manager, used to be Peter Norvig at Google, and he would—he would read the book first. Yeah, it's like oh, we're going 0:43:13.940 --> 0:44:06.891 to use Python. Okay, I'm going to just start reading the Python manual from page one. Yeah, my wife is like that. Yeah, yeah. I mean, but I think that for me like what works is starting with some kind of problem, and what I've learned over time is actually even better than that. And I'm—this is where I'm not there yet, but I'm trying is actually starting with a customer. Yes. You know, I think starting with a person who has a problem, but you know before that, I think using yourself and saying okay, what is a problem in the real world for me? Yeah, starting with that and working backwards to, you know, what AI thing do I need to build, and then what language do I need to build the AI thing. Yeah, and go from there. 0:44:09.304 --> 0:46:08.100 uh yeah after dot science i did consulting for a few years um and worked with clients all over the world which was great i really recommend it actually like being a one man or one person consulting company can be really amazing um but but then what happened i was watching this kind of open source ai space um uh kind of towards the middle to end of last year like august september uh 2023 um and i saw these two really interesting things happening um the first one was that um mistral 7b came out um and now suddenly like you could have like a good quality llm like a chat gpt level ish llm that you could run locally on your own machine and the and the other piece was um that it became possible to both run but also fine-tune those models on consumer hardware so you could now like have an almost chat gpt level um model that you could fine-tune on your own private data on like a single 3090 like the kind of gaming um gpu that you might have in your in your home pc now like let's dive into that a little bit for folks so you know a lot of people have heard the word fine-tuning yeah but like how do people actually do that here's i'll say what i think it is and you can correct me and fill in the gaps so when they trained mistral they had some pytorch or tensorflow or mxnet whatever it is they had some code to train mistral they give people that code and so you can basically run that code on the current model and basically continue training on your own data is that pretty much how it works exactly so fine-tuning is just more training okay um so you take the weights of an existing model um and you train it more on training data that is your own private training data um it is a little bit more complicated than that because there's a technique that's often used called low rank adaptation 0:46:08.660 --> 0:48:07.765 um which i don't fully understand the math but it's some sort of matrix decomposition where you end up just having to train like a much smaller um set of weights uh than the whole set of weights and that's really cool because it makes it tractable to train or to find to do more training on this model um but without needing like huge memory requirements and that's what i mean by it became possible to like fine-tune mistral yourself on like a single gpu um and that depends on that kind of lower so is it holding the original model in the cpu memory and then the gpu memory has the low rank version or what's uh that's a really good question i mean i think everything fits in gpu memory um but i think it only needs to do back propagation on the smaller matrix oh that makes sense yeah so it's able to like uh do that with fewer resources and also just takes less time right that makes sense yeah because for folks who don't know the um the way the way back propagation works and we talked about this a little bit in the ai episodes but you keep this gradient matrix so effectively doubles the amount of memory you need because for every matrix you need this sort of shadow right yeah and if and you might need to up the resolution too like maybe the model can run an 8-bit but for training it has to be 16-bit and so so that now you're talking about a 4x multiplier and so if you can just do inference on the big model it can sit in 8-bit yeah and then do the training on the smaller 32-bit or 16-bit model yeah yeah um that is really cool um yeah yeah so um so i noticed these these two really interesting things happening in in the world and um i turned to my friend and co-founder kai in bristol and i said it's time to have another go like let's so was he with you at that okay yeah we kind of got the band back together um so so for helix um yeah we um we 0:48:08.305 --> 0:49:55.060 saw this this opportunity and we went in and we we then like spent three months furiously hacking together the stack i mean it the we we joked that it took us 10 years to know what to build in 10 days yeah that's how it goes but um but yeah we we put together this this stack that allowed you to um deploy uh these open source models like mrl7b and also fine-tune them um and to make that easy for people with like a nice web interface where you just drag and drop in some pdfs or some documents um and then we did this interesting piece around the fine-tuning where um we would use we would take the source documents chunk them down into little pieces and then we would um use and use an llm to generate training data from those source documents because what you want to do when you're fine-tuning is is train on data that's similar to the kind of questions that a user would ask of it assuming it's an instruct style model like a question answering model um so you can't just train it on like the raw text because then it will just be good at completing the raw text right but what you need is you need to train it on things that are like the questions that users are going to ask so we actually use another llm to transform the source documents into questions and answers about the source document and then we use those and question answer pairs to fine-tune the model um but our most popular blog post was how we got fine-tuning mrl7b to not suck and um that goes into a bit more detail so maybe this is a dumb question but but you know these models are known to be trained on enormous data sets yeah and i think there's one called the pile it's like 10 terabytes of text or something yeah some enormous 0:49:56.744 --> 0:51:54.100 amount how do you how does your fine-tuning have any statistical significance when you compare it to just this enormous data set like how are you able to to to move the model move the needle right on the model yeah so i think it has to do with like the the learning rate that you choose um okay when you're you're doing the fine-tuning um uh and what what has been found kind of empirically is that uh even just using a very small number of samples in fine-tuning with like i guess a relatively high learning rate so it makes them makes the model change quite a lot but not too much because if you go too far then it just goes like haywire and starts spouting garbage so you've got to find this like middle ground um but even with a small number of samples you can get the model to start generating things that are similar to like the the the fine-tuning data set or start generating responses in a different style or with a different structure uh really quite quickly and then it's yeah it's about finding that trade-off you don't want to over bake the thing so that it just kind of memorizes that um and forgets all of its prior like learning from uh from the pre-training um makes sense but yeah we we like tuned those parameters and we found something that worked um but then the interesting thing was i guess commercially from a business perspective is that we went into the market in december last year with two hypotheses the first hypothesis was um let's uh people will care about running models on their own infrastructure people will care about local lms and the second hypothesis was um people will care about fine-tuning and in particular fine-tuning for knowledge which was that like that piece that we that we developed uh and what we heard back from the market um was resounding yes to running models locally so we launched on december 21 um uh on december 24 uh this company shows up on our discord 0:51:54.100 --> 0:53:54.020 and they're like we're based in germany we're really interested in self-hosting these models we're not ml experts but we see the opportunity and by january 1 they'd integrated helix already into their stack wow like this is incredible this is like and and then they were going after like big enterprise customers um with and in in partnership with us and so that that was like really exciting it's like the the universe was telling us like there is an appetite for running llms locally and building um kind of the capabilities to to make that easy to do um now uh in terms of the second or the first well the second hypothesis about fine-tuning we actually got a big meh from the market like we put all this effort into fine-tune this fine-tuning stuff um everyone just said like oh we just want to do rag like retrieval augmented generation yeah interesting why do you think that is in hindsight i think because fine-tuning for knowledge is slower than rag um and the results are not as good so like it was kind of an experiment to see like oh well maybe people will care about it um but um but yeah i mean and so what we did was we extended the stack so we added rag to it and the other thing we did was we added api calling um which is where you can give the model um an open api which is confusing because it's not open ai it's open api used to be called swagger a swagger spec yeah it's easier to just describe it as a swagger spec um so you give the model a swagger spec and then you give it a query from the user and you basically say like a bit like how function calling works you you you say to the model like um you've got uh these three apis you can call and please clap so the first so it comes in three parts the api calling the first part is um classify the user's question um and then tell you whether um uh the the user's query requires a 0:53:54.500 --> 0:55:54.260 um an api call uh yes or no and if so which one and then the second step is like constructing the api call so assuming that you want to make an api call based on the user's question like can i rent a um a crane that can handle three tons in hamburg next thursday like construct an api call that will query the product catalog uh with the correct question and then um the third part is taking the response from the api so the system will execute the api call for you and then taking the response from that api and summarizing it back to the user so from the user's perspective they're just saying like hey have you got a crane and the model quickly says yes i've got one i've got these three available they cost this much um but what's actually happening underneath is that classification uh api call construction and summarization steps um to rest so that api calling feature has been really commercially successful for us um and tons of people are interested in that um and so i'll just shout out quickly to the fact that we did our 1.0 launch last week um so yeah if you're interested in running models locally uh we now run um on windows mac and linux you can run it alongside olama we have a nice application editor so you can like click buttons to set these things up and add knowledge um with rag um uh add api integrations and so on um so if people don't have let's say a gpu could they still run helix on let's say an ec2 instance yes and you can even just run it on cpu because olama runs on cpu oh okay however if you have a um a mac like one of the more recent m1 m2 m3 max um olama also works with the gpus on those machines so you actually get really quite good performance on a mac um on a cpu linux or windows machine it will be a bit slower uh but if you've got like a gaming rig at home running windows then you can install wsl2 like windows subsystem for linux and um 0:55:54.820 --> 0:57:53.380 And Docker, and then the whole thing does work. Yeah, it's pretty cool. That is awesome. So yeah, or you can dual boot and have a Linux machine with ZFS on it. Yes, that is another option that's really cool. So how does that work? It's like a desktop or a library, I guess. I'm trying to figure out here. So in the case of OpenAI, they give you a key. Yeah, and you're calling into their server with their key, and that's how they do billing and all of that. In your case, how does that work? Is it—I guess also a key, but you run—you get the key from Helix.ML. So we are Docker Desktop license. Like if you have less than $10 million of annual revenue, then you can use us for free. Oh, perfect! Just download it on the website like and it's an install script, so you just like run our installer that creates a Docker Compose file. You run `docker compose up`, and then you look at localhost in the browser, and like the whole thing is running there. But that means of course you can then also deploy that if you're a company. You can deploy that on your own internal infrastructure, and that's where kind of the DevOps piece comes in. We also support Kubernetes, so you can run the whole stack on Kubernetes. And yeah, we might use Kubernetes with a bunch of GPUs in production for our own service. Very cool. Yeah, this is fascinating. Okay, so maybe we'll wrap up with what's one last piece of advice for let's say someone's listening right now and they are completely infatuated with this. They want to get into MLOps, and we talked a little bit about you know kind of things to study and all that, but in terms of maybe more of a like a mindset or kind of lessons you've learned on the soft skills side, what are some advice that you can give to folks out there who want to get into this field? Yeah. I mean, I would recommend joining the MLOps community so you've got other communities to talk to about it. 0:57:53.380 --> 0:59:51.140 have peers to to talk to about it um and then um yeah i would i would say start playing with the technology yeah like i said earlier if you want to get into ml ops like play with uh python git docker pytorch that kind of thing um if you want to get into this newer field of llm ops um this uh like how you manage rag and um api integrations kind of on the other side of the llm uh api boundary if that makes sense um then um yeah download olama and spin up an llm locally if you like like play with helix and and set up like an api integration and then that'll set you up well to um to be able to go into maybe like a job interview or something and say like i've got this experience or um or to build a site project yeah totally amazing advice i think uh we covered um we actually covered docker in the kubernetes episode which is uh you know we should have in hindsight maybe had a dedicated docker episode uh but check out the kubernetes episode if you haven't already uh we talk about minikube and there's probably better things out there that episode is a little dated now that's okay minikube's solid is is this still the thing or there's kind as well that's right yeah i used kind recently um but uh yeah get get set up on your machine and uh check out helix ml totally free uh for folks out there um if you're gonna install it on your work computer and you work for like google or some company has a lot of revenue you probably need to check with your boss first or just come talk to us on discord yeah or just talk yeah exactly if you're a hobbyist uh just install it and get started and try things out yeah awesome cool all right thank you luke thank you so much for your time it's been awesome thanks for having me on cool all right so as part of this three-parter you know we uh we just had uh luke on 0:59:51.140 --> 1:01:49.307 to talk about um you know what is ml ops and kind of get us started on that and explain his story and now i'm really excited we have uval fernbach here who is the cto of jfrog ml and he's going to explain to us more about the whole kind of supply chain of ai software so thanks so much for coming on the show uval thank you thank you and really nice to be here cool um yeah it's uh it's a pleasure to be here yeah and just a recap we are at the swamp up conference which is i think hosted by jfrog yeah it's hosted by jfrog and it's actually my first swamp up so uh cool excited to be here that is awesome um so before we dive into ml ops why don't you tell us a little bit about your story what kind of led you to jfrog sure um so previously before joining jfrog actually only two months ago i was part of quark quark is an ml ops platform and i was one of the co-founders and cto for quark that was acquired by jfrog uh end of jfrog uh end of june um and now basically part of jfrog ml so before being part of quark and founding quark i was working for amazon web services for five years i was part of the aws machinery and service team basically working with the aws customers on their challenges around machinery um so i've seen hundreds of customers hundreds of companies um trying to solve that trying to understand how they can start iterating on machinery building models experimenting and eventually impacting the business based on machine models and ai applications um and that's basically one of the main reasons why together with my other co-founders we decided to fund quark and help those companies to actually achieve that to make sure that they impact their business and build models not just for the research or for the development 1:01:49.814 --> 1:03:49.220 but actually um making that supply chain work and affecting their production as well cool so you know i think one really unique story i'd love to dive into is you're at aws which is a very large organization right um how roughly how many people are there how many engineers at aws maybe so tens of thousands i've been there for five years and in that time it changed from you know a few thousands to probably tens of thousands uh by the time i i um i actually um founded quark so it changed quite a lot i'm not sure what are the numbers right now but yeah it's why by the way it was amazing to see how a company that was relatively small at least versus amazon actually grew and became such an amazing business yeah definitely yeah i feel like the documentation the quality of service of aws i haven't used other ones recently but i remember um this is years and years ago looking at all the different options and just seeing just just a higher level of quality for aws like things just tended to work and the documentation was really well done and that that there's an extraordinary amount of effort that and and user study and things that go into that yeah yeah i must say that i i learned a lot during that time because i've seen you know the company that basically founded the cloud or the surface company that created the cloud that understood that their customers are not it their customers are developers right that need to build software and they build that as a self-service basically you know uh product-led development before anyone talked about it and um i think that one of the reasons why eventually we found the quark is that we saw that coming through there are so many services and you know it became really really difficult for companies to actually utilize the different services and not just utilize different services but the challenges of growth you cannot just use the c2 and s3 1:03:49.220 --> 1:05:46.899 anymore you need like different high level um solutions to make your your product work and and i believe that it's amazing what aws uh built um but nowadays i think that in many cases using those building blocks directly and trying to build on top of that is actually too much for many of the companies and actually the amount of investment that you need to do to actually do that the amount of of engineering work that you need to do to build for example you know an ml solution it's just too much for many of the companies and yeah it's really difficult to do that yeah that makes sense i kind of experienced this firsthand where you can stand up things uh with the web ui you know and and so it's like i want an ec2 instance i want the database but then at some point you need to programmatically do that it's like oh i need a beta database and the beta database needs to have all the same things that you just you just forget what you clicked on a month ago so you need this infrastructure as code and then that's a rabbit hole and it's hard to hire people who have that talent and so it becomes difficult yeah yeah it's it's it's actually amazing to see how companies have grown to a place where they understand that it's not single point solutions you need to create platforms so for example going to to the softly so software supply chain understanding how a software supply chain looks like it's not you know just a ci cd it's your source code and how you manage your binaries and then how the security infrastructure looks like on top and then why software engineering is different than data science and how it's all connected to the same supply chain and and eventually this challenge is not just you know spin up up instances and databases and and writing code in git it's way more than that it's having a system that can actually work in scale and allow your company to be efficient both in 1:05:47.459 --> 1:07:45.540 you know once you have like two three five ten developers but the same efficiency once you have thousands of developers and you want to actually make sure that you deliver on time and not just deliver over time but you delivered a trust for the products products that you can trust both in in terms of the features but also in terms of the security um and make sure that they they work in production exactly the way that they should based on like the real product spec for example yeah that makes sense so you went from you know giant company to clock i'm assuming you know maybe a handful of people when it started yeah sure yeah and so when it started you know it was four people but uh of course we grew all the time yeah yeah so was there sort of a shock uh you know the moment you went to clock and you're like wait there's no it department there's just me you know like like how did you navigate that because i noticed i know a lot of people who go from big to small company and there is a risk like there's some people where it just doesn't work out and i mean that's okay they go back to facebook or whatever but but uh you know what was that experience like for you you know there's all risk and i think that founding a company you know being one of the co-founder or even one of the first employees is not for everyone and and that's great because i think that the challenges that you have doing that are different challenges different experiences than working with you know a giant company like amazon um so i personally love it i i think it's it's an amazing experience i love the fact that it was on me like if i had a problem with my computer i needed to fix it i needed to talk with you know the the the person that we bought the computer from i need so yeah you know it's part of like the the experience of founding company being part of a small organization a startup uh so i really love it but 1:07:45.540 --> 1:08:47.267 of course it's not for everyone and i think that one of the challenges is that you need to make sure that you focus on the right things the right challenges because there are many things to do but your main goal is always to build a successful company and make sure that your customers actually you know get the benefits of your product the customers are actually happy with the solution that they get and want to to use you and of course recommend you to their friends and colleagues yeah that makes sense so maybe uh before we jump into the mlop stuff what was quark like what what did quark do and what does quark continue to do through the acquisition so um as i said quark was founded four years ago and we started from day one uh to focus on this challenge called mlms so basically allowing companies to start from the development to the production we follow the need to manage multiple solutions multiple platforms and even like 1:08:48.786 --> 1:10:44.076 utilizing uh or actually handshaking the code between different stakeholders during that process so from day one our main focus was production how to make sure that models will work in production and of course going uh left or going back from production on how you can make sure that the supply chain actually work from the development to the production um and and smoothly as possible so we started with that and i think that one of the challenges of mlops is that it's not just around the models it's around the data as well so you need to both have that solution for data that allow to again have data for production have data for inference but also data for training and then the way to manage that data those features um the same way as you do that for models and of course nowadays the same way that you do it for gen ai applications lms things that are a bit different by nature but eventually are based on the same like infrastructure and the same needs in terms of their um basic needs yeah yeah that makes sense i think a unique challenge about ai is the problem of leaking the label so for example let's say you fine-tune a model on a series of questions and answers and then use those same questions and answers to evaluate the model well it's going to be pretty pretty good so yeah and so that might sound like obvious but it's actually really difficult to not accidentally leak the label it's so difficult to to keep sacrosanct like your your evaluation set and keep it from accidentally especially when you're doing aggregations at so many levels from accidentally sort of poisoning the well i i fully agree and i think that like having the proper data platform for ai and machinery applications is actually crucial 1:10:44.340 --> 1:12:42.960 because it's not just making sure that you manage the data sets right and understand what's the difference between your training data set and evaluation data set and i agree that's crucial because that's the only way for you to to understand the metric of a model it's also understanding that the data that you build the model on is actually available during prediction during inference so and by the way this is i think one of the reasons that many ml projects fail is because their scientists are doing research on the data they get the data from you know the data warehouse they train the model but eventually that data doesn't exist during prediction it doesn't exist during inference and many projects fail because they created an amazing model but they don't have this data when the application needs it when the application calls the model the application doesn't know uh i don't know the story the story about that specific persona during that time and cannot get that because maybe it wasn't calculated yet and and one of the things that we started with in quack is to create a feature platform that allows to create both an offline uh um feature store and an online feature store that are basically based on the same calculation the only feature store allowed to get a low latency current value of your features while the offline features are allowed to get all that training data that was calculated exactly the same so you know that there is no um basically drift between your offline or training data and your inference data it's exactly the same it was calculated the same and the data that you were trained on is actually available to your meet for us yeah i mean that you know the story i always tell folks was uh there was this product called google plus uh you know it died uh along with a whole bunch of other google products but we were doing the uh friend suggests that a bunch of ml stuff for google plus a long time ago and there was an issue 1:12:43.062 --> 1:14:40.580 where the day of week feature was zero based uh at training and one based that's serving and so sunday didn't exist and saturday i guess also didn't exist or i don't remember exactly what happened with saturday i think it was a one hot encoding yeah so sunday and saturday both didn't exist and it caused all kinds of chaos and it was extremely difficult to find because you just would give bad suggestions on the weekend and and there's no compiler error there's there's no uh easy way to find they're written in two different languages you know the training and the serving system so uh so yeah i think there was a product feast which was a very early attempt at this um but um yeah how has that evolved uh is it to the point now where you can write it once and it works on both so um so you know know i again i believe that the feature store is a crucial part of an ml platform and and fist by the way is an open source that still exists but um i think that one of the main challenges with such solutions like fist like um the the cloud vendor solutions for feature stories that it's not just about the storage it's about the data pipeline as well you need to have a data pipeline that eventually can connect both the offline and the online on a single process so you write your code once in a in the same language and that code will be used to create the data for both and again it's really connected to what you just said because in the past many models were for example written in python but then the production implementation was let's say in c or java or whatever and you needed to recreate the features in a different language and there is no actual way to compare the two and understand that the features were created the same uh so so many companies had the same challenge that you just talked about of having that you know that's the difference between the production and the training um and and again it's it's the same challenge with a feature store that are only the storage level like fist and 1:14:40.580 --> 1:16:40.172 like others because you need to have that same data processing layer that will be used to create like both features and i must say that uh part of it is that python now actually grew enough to be good enough for production because it's not like in many cases it's not really python production you build a model in python but then eventually it's a c object that's running in production uh although the code is in python so you can actually use the same code and and get pretty good performance so uh that's of course part of the technology uh enhancement that happened during those years oh very cool so you know a lot of projects start with jupiter notebook which is kind of itself kind of this extension or or uh uh descendant of mathematica right mathematical notebook which is just beautiful like you know interactive thing and it just becomes really hard then to to to productionize that because it's not dot py files it's all these cells and and uh so how do you recommend for folks to to go from that notebook to something that can run at scale so so it's a great question because um i need to talk about supply chain and that question eventually uh pivot me back to supply chain and i i love notebooks i think that notebooks are great for some things for example if i want to visualize data if i want to have an interactive environment notebooks are amazing for that uh by the way ideas are amazing for other things for example debugging code ideas are way better for that sure notebooks there are advantages for each one of them but both are not built for production like eventually when you want to build for production you need to have a proper ci you need to have a proper supply chain that move that that that code to be an artifact and from being an artifact eventually deploy that and of course making sure that you have a lineage between the deployment the artifact the source code all that should be connected in a way so what i've seen is 1:16:40.999 --> 1:18:36.660 that notebooks are great are amazing but eventually once companies um understand or graduate to the place that they want to be to production they need to add more structure to the way they build the models and for example one of the things that we've done in quack and now with jfo ml is is is to have an opinionated way of how to build models in terms of structure that allow every new data scientist to look at the model that they never looked on before and understand okay this is the training of the model this is the code for the inference this is the code that fetched the data from the feature store like immediately understand the structure of a model immediately understand how that model was built and be able to help or or you know train a model that they haven't seen before just by looking at the code and looking at that structure so i believe that notebook evites it's part of the research but in some phase you need to move from the notebook to have that code manage uh for example in in a source uh uh version solution so let's say a git uh solution in a way that can actually be automated so you need to have some kind of structure of what's the train job looks like what the prediction looks like what's the depends we well the depends is which packages you need to run that model because packages you know especially in python a new version come out can come out and break everything so you need to actually you know freeze those the pants is to make sure that you have that model actually reproducible and not just train once on a notebook and someone's computer and we'll never walk again yeah yeah definitely yeah and you have to be able to go back as well like uh um we did this these really hacky things but there's got to be a principled way like once you've you you have it in source control and it's in production you're always going to want to do more data science because you 1:18:36.660 --> 1:18:42.260 are going to change the product. The product changes without you because new customers adopt and some. 1:18:43.940 --> 1:20:40.624 churn and everything. Yeah, and so you know being able to run a Jupyter Notebook on your production code and then make changes—that whole integration is extremely complicated. Yeah, so valuable. Yeah. You know models degrade over time. That's that's I think the way model looks like because data changes over time. So part of having the structure, so by the way one of the things for example that we do with J4ML now is that whenever I build the model, I automatically basically copy or or freeze the source code that I use to build that model. I freeze the dependencies that we use to build that model. I create the model without in fact the trained model, but I can reproduce that model at any point in time. I know which configuration I use. I know what was the source code I used. I know what is the data set that I used. And I think this is a practice that companies must utilize. Practices must make sure that they have the ability to reproduce a model because they will need to train that model again. They will need to fine-tune it. They will need to understand if even without model monitoring and if even without, you know, really looking at the data and seeing the drift—it probably happens. And if it happens, it means that they will need to retrain that and they will need to have the structure that allow them to retrain that even if the data sign that the build that is not available anymore, it's not in the company or you know just your team grew and you have more projects. Yeah, that makes sense. So on episode 158 we actually had Bill Manning from JFrog come on the show and talk about software supply chains. And so folks should definitely listen to that episode if you haven't already. So what's the delta between that and AI supply chains? How does AI kind of make that problem different? So, 1:20:42.312 --> 1:22:39.627 first of all, the process itself looks different. Like an ML project starts from research. It starts from experiments. You need to track those experiments. Those are phases that do not exist with software. Usually when you build software, it happens because of two reasons: first, you have a new feature, and second, you have a bug. So those are the reasons why you start working on a software project. But with ML, it's also because your data have changed. It's also because you have more features and you want to make that model better. So there are many reasons why to work on an ML project, and it always starts with some kind of research experiments. In those phases looks entirely different in terms of ALOps vs MLOps. Same for the deployed model because the monitoring that you do for software is only the infrastructure monitoring—maybe logs analysis, maybe understanding if there are bugs and what are the latencies of different processes, and those kinds of things. But with models, you actually need to monitor the data as well. You need to monitor the model and make sure that this model actually gives the right impact on the business, make sure that this model doesn't degrade over time. So the process itself looks a bit differently, but eventually what happens in the software supply chain process is that you have code, then you create artifacts, and then you deploy those artifacts, and those parts are the same with AI with ML. Maybe some configurations are different. Maybe for example nowadays with AI you have also prompts as part of your artifact. Maybe you have an external model like I don't know ChatGPT or something like that, and there is a new version, and it's not something that you manage, but it's still an API that you call. It's still application that you need to manage. So eventually it's the same artifacts being the same same idea but different processes, and even the security posture, even the understanding if a model is is trusted or not—it's trustworthy. That's, 1:22:40.217 --> 1:24:39.299 pretty much the same as any other kind of software. So if you build software and you scan those software and you scan the dependencies and you scan your runtime environment and you want to make sure that the same software that you build in runtime don't have vulnerabilities and you scan it from the source code up to the right time and understand how your security posture looks like, it should look the same with Machine Learning because eventually those are packages, those are artifacts, and third-party APIs that need to be secured and need to make sure that you manage that life cycle in a way that you trust. Yeah, that makes sense. This was awesome. Thank you so much for your time. This was really great. One last thing: do you have for students who want to get into this field? What's some advice you could give them? So first of all, it's amazing field, so do it. It's great advice, but second is that I think nowadays with all the GenAI models and LLMs, the entrance point is actually easier than the past. Like in the past, you needed to understand the statistics around the data and you need to add these data to start training models, and there was quite a lot of understanding, quite a lot of work you needed to do to actually have a model ready. And nowadays you can start an understanding that you know just by using for example ChatGPT. Yeah, I think that as an example my father only now understand what they do because they understand he saw how model looks like and what are the effect of models. So I believe that this is a great starting point for every student that wants to start learning about Machine Learning, about AI. You know, play with those tools, understand your specific prompt change change the way the model reacts, and eventually those are models that may be more complicated than the models that you know we've done five years ago that we use for specific, you know, point solutions, but eventually those are models, 1:24:39.299 --> 1:24:58.407 that use the same technologies behind the scenes. And understanding how those models behave will give you quite a lot of knowledge about how what the development looks like and how you can utilize Machine Learning for every kind of task. Very cool. Thank you so much Yuval. I really appreciate your time. Thank you very much. 1:25:00.398 --> 1:25:12.340 All right everyone. So we have been talking in this episode about MLOps, and I'm really lucky that we have Steven Shin on the phone who's done a lot of really interesting work with 1:25:14.134 --> 1:25:35.346 this, and we're going to focus on LLMs and Graph RAG as kind of two case studies of you know getting ML and AI, you know, implemented and then you know what that whole process is like and then getting all of that into the hands of customers. So welcome Steven. Thanks for coming on the show now. Very 1:25:35.751 --> 1:25:48.020 glad to be here and excited that I'm speaking at SwapUp actually on on the same topic. So I'm going to be talking a bit about Knowledge Graphs plus LLMs and in particular how you can apply them to your DevOps pipeline. 1:25:48.812 --> 1:25:58.347 Cool. So before we jump into all of that, why don't you give us a quick background? What was your path that kind of led you to where you are now, and what are you doing right? 1:25:59.157 --> 1:26:45.040 now. Yeah, so maybe we'll start with where I am and go backwards. So I'm VP of Developer Relations at Neo4j. Neo4j is a graph database company but also does a lot with generative AI and Machine Learning and is building out architectures for Graph RAG that a lot of enterprises are using. Before this, I was working at JFrog for basically the same role, VP of Developer Relations. And I did this Oracle slightly different role, but basically the same role. I was running the developer marketing team, and the way I the way I kind of got into Developer Relations in general because I was also a developer advocate for a bunch of years is I 1:26:47.892 --> 1:26:48.580 made the mistake. 1:26:51.165 --> 1:26:51.891 Okay. Of writing. 1:26:52.617 --> 1:26:53.342 A book. Oh. 1:26:53.427 --> 1:27:29.725 Okay. Um, now I'm not discouraging anybody from writing books. It's a wonderful way to increase your reputation, to like teach and explain things which you're passionate about to a larger audience. Um, and when I wrote the book, I was um fortunate enough to have some great co-authors who I collaborated with on the title. Um it's a lot of work. You're basically giving away six months to a year of your life where you have no weekends, no evenings, especially if you have a day job that you also have to keep going. Um and then when you finally finish the book, the publisher says, 'Oh.' 1:27:31.311 --> 1:27:36.180 That's a wonderful book. We've released it on, you know, Amazon Books, etc., but— 1:27:37.875 --> 1:28:09.567 Can you help us promote it? Oh, you become the promotion arm for the book. Like if you're not submitting to conferences and talking on a topic, if you're not on social media, if you're not out there being a vocal advocate for the technologies you care about, and—you know obviously—pointing out there might be a good book that folks can read for more information, then you're not doing your job. Now that first book I wrote? Oh my god, it must have been, I'm dating myself like 15 years ago. 1:28:09.752 --> 1:28:21.220 So you knew a technology and the publisher approached you and said, 'You know, Steven, you are really gifted in this technology. Why don't you write a book on it?' That's kind of how that went down. 1:28:21.261 --> 1:29:12.443 Yeah. And actually, when I was in college, I assumed it was the opposite. I assumed like you wrote a book and you went to publishers and you said, 'Look at this great book that I wrote. Would you want to put this on your brand and publish it?' But no. No, publishers, especially tech publishers, work exactly the opposite way. They have a roadmap. These are the titles we want to publish; these are the topics that matter to us. Occasionally, you can influence that, but not until you're an established author. You have really good relationships with the editorial group and they then say, 'Okay, for these books, we want to author for these titles. We want to author who would be a good candidate? Like who has the expertise, the who can authoritatively write a book and help us market it later on?' And— 1:29:12.510 --> 1:29:18.467 So you had the footprint then that they were able to find you? So you had done some kind of promotion to get to that? 1:29:18.957 --> 1:29:46.379 Point. Yeah. So I mean, it was for Java Effects technology, and I got in really early, like beta days. I was already building applications with it. I had a lot of good connections in the Java Effects community, and I got invited with somebody else who'd written another book with the same publisher. And now this was the first real Java Effects book. Now fast forward 15 years later, and we got asked to update basically the same 1:29:48.066 --> 1:30:09.109 title for Java 21 and 23 with all the latest features and capabilities. So it's, oh my gosh, become for this small niche market, Java Effects, which I don't actually do professionally anymore, but I still keep up on it, and I act very close with the community. This has basically become the authoritative guide for client developers in Java. Wow, this— 1:30:09.852 --> 1:30:40.244 I'm totally dating myself here, but the last time I professionally wrote Java was this thing called Google Web Toolkit. Oh yeah, and you would create JavaScript with Java. This is one of these like in hindsight, in my opinion, really bad ideas. I mean, maybe not, but—and I was writing, I think in Java 6 or something, and it was compiled to JavaScript, and all. It's kind of a it's a really strange. I mean, that was my first introduction to Java. 1:30:40.244 --> 1:31:24.996 Yeah, it's very strange. GT was kind of an interesting approach. Yeah. Um, I think we've come a long way there with JavaScript frameworks which do all the heavy lifting. So basically you can have a very feature-rich application without a heavyweight back end. Yeah. So I think that's become the modern development framework. GT was an attempt to do that—have the heavy front ends, but then have you write in entirely in Java and then deploy all that JavaScript magically and like have the web application just appear. It's technically difficult to do that perfectly, and therefore it couldn't keep up with like modern JavaScript frameworks. Yeah, I mean. 1:31:25.334 --> 1:31:38.980 And then on the other end, it really got squeezed by Node because the advantage of GWT was for example you could write validators in Java and validate the client side and the server side, but now you can do that with JavaScript. 1:31:39.070 --> 1:31:52.755 Yeah, and again for reference, the Google team accepted and endorsed the use of both GWIT and GWT for the same acronym, so they would use both. Ah, yeah, they never agreed upon what it was supposed to make sense. 1:31:53.009 --> 1:32:30.572 Cool. So okay, fast-forwarding all the way to current time. So you are actually, before we dive into LLMs, let's talk about this a little bit so you know what is a Developer Relations Advocate and someone who leads a team of advocates. You know what that job is because we're going to dive into a lot of technical content here, and when people think of public relations, they think of speeches and writing speeches for candidates—these kinds of things. But developer relations, you know, you're actually building a lot. So why don't you kind of give people a little bit of scaffold there? 1:32:30.960 --> 1:32:35.770 Yeah, okay. So for those folks who don't know the job or role of a developer advocate. 1:32:37.930 --> 1:33:11.865 A developer advocate is basically somebody who they're advocating for developers both in the product. So if you work for a company like saying, 'Hey, we have these users and these—they're trying to do things with their product.' Like, let's actually help them out and build features which are going to be beneficial to them, but then also educating developers about new ways of doing things, new techniques, and kind of upskilling, helping them upskill. And the way I would describe a developer advocate—like the most simple description is a developer advocate is a 1:33:13.435 --> 1:33:14.260 geek with social skills. 1:33:16.101 --> 1:33:41.540 All right. So you have to be technical. You have to be able to write code, be up with the latest technologies and trends, and constantly learning—like picking up new technologies. But you also need to be able to present, able to do interviews like this, to kind of be very fluent, so strong English skills, like strong presentation skills. Those are all really important. 1:33:43.641 --> 1:35:38.644 You can actually come up to be a Developer Advocate from either a highly technical role—like a lot of developer advocates start their career as programmers and architects, and like they kind of get tired of just building things. Like they want to kind of move up and actually be the change agent for industry and for folks who are adopting technology. So it's a great career path past. Like what do you—what do you do when you're tired of just being the most technical person at your company? On the other hand, you can also become a Developer Advocate by having great social skills and great kind of English language presentation skills. One of the folks who I just hired, their name is Naya Macklin. I'm also a speaker at SwampUp; got accepted before even joining my company. Company, and they came up through a background of journalism and politics, kind of being involved in political campaigns, being involved in all of that—writing, outreach, campaigning—and wanted to move to a more technical role, wanted to move to something which was more technical. So they went to a bootcamp, kind of learned Python, JavaScript, all the basic skills, you know, all the way up to building web applications, like deploying technologies. They were Developer Advocate at Couchbase, now a Developer Advocate at Neo4j. And very early in their career—like I would say that—this isn't to be a Developer Advocate. It's not something that you have to be old and gray to be a good Developer Advocate. Some of the best developer advocates actually—most of the best developer advocates mirror the audience. So if you're talking to a technical audience, you want to be able to just talk to them as peers. If you're going to a university and speaking to them, you want to be able to talk as a recent graduate kind of, like being very close to the audience. I think that makes it more credible and makes you more effective in the role. 1:35:39.589 --> 1:36:25.620 Yeah, that makes sense. Very cool. Yeah, so folks out there, this is one of many really interesting professions that we're going to learn about here in these sessions. So if you have any questions about this or any other professions, don't hesitate. You shoot emails to us, post in the Discord; feel free to keep that conversation going. There's a lot of really interesting activity on Discord, and feel free to join and be a part of that. Okay, so let's dive into LLM. So LLM stands for Large Language Model. I think now they're starting to call them Foundation Models because you have vision and all these other modalities. But you know what is a large language model? How would you describe that? 1:36:27.143 --> 1:36:51.460 To somebody? Yes. I think that this in the past has kind of required a technical explanation of like how you train the models and then how you can actually build the models to learn by feeding them extremely large data sets and then having them kind of iteratively complete the next word or kind of the next idea in a chain of commands. Now, actually, 1:36:53.367 --> 1:36:57.265 it's much easier to explain this now because everyone's 1:36:57.569 --> 1:36:58.885 That's true. 1:36:58.885 --> 1:37:24.019 Yeah. So if you're using ChatGPT, if you're using Copilot, if you're using like any of these tools which kind of give you a language interface to talk to and interact with your code, with like the web, with an enterprise data set, then you are using an LLM behind the scenes, and you know you can see that. 1:37:25.530 --> 1:38:04.815 It's a very effective tool for a lot of tasks which are for humans time-intensive and require a lot of knowledge entry, which require a lot of knowledge gaining. So it's great for summarizing information, great for writing emails, excellent. Don't recommend this at home for writing research papers, research, but like the class papers, like things which the professors say, 'Oh, research this subject and then write it to a three-page paper,' like it's yeah, amazing at that. But of course, completely against most schools' rules. That's right, don't violate. 1:38:04.815 --> 1:38:20.627 Your school policy? It's really good at portmanteaus. You know, you can say, 'Give me a portmanteau of these two concepts,' and you will come up with some incredibly brilliant names of companies. You know, it's really good at. 1:38:20.931 --> 1:38:30.027 That now I would say what LMS are poor or they don't have a strong ability for is in general they don't reason. 1:38:33.891 --> 1:39:51.719 Like we do. Yeah. So if the source material, if the context is rich enough to kind of piece together and give clues on what the answer is, it can both kind of pull from that body of knowledge but then it can also synthesize information which maybe is not clear from the very large. Like they feed these systems with hundreds of billions of words and like huge unstructured document sets, and so it does kind of these amazing leaps which seem like reasoning, but they're not actually reasoning as humans think and reason. And also at a related thing which they're poor at is math, right? So in general, like it's not you know they work as simple calculators and can answer basic math questions because again, that's all exists in the source material, or they can specifically train the models and add things for common questions which get asked. So they do a good job of calculating and returning your results via going to an agent or some other system which is specialized, but in general, they're not designed for math. And if you give them a complex problem like I was playing around with one of these kind of online LM games where you're supposed to trick and hack the LM. Oh. 1:39:51.719 --> 1:39:52.420 That's the thing. 1:39:52.420 --> 1:40:18.260 And it was kind of fun. Like they set it up as the LM thought it was like a wizard and like it was protecting some secrets, and you were supposed to convince it to give you the secrets. And basically, they had multiple levels for difficulty where they add additional prompts which would prevent you from doing attacks which will allow you to circumvent the LM. 1:40:19.765 --> 1:40:48.570 But basically the way to hack it is you did a combination of reasoning and hard math problems. So you'd ask it to, for example, do like a rough RSA algorithm or something complex, moderately complex, and it basically you ask it to give you an answer, apply a math algorithm to it, and then the system which is checking the answer to make sure it's not revealing secrets now can't check the answer properly because you've encrypted it. Ah. 1:40:49.144 --> 1:40:50.089 I see, but what? 1:40:50.477 --> 1:41:07.032 You look what you learn when you keep giving it harder and harder and harder math challenges is it actually falls apart. Like for example, like rotation ciphers, it'll consistently get the first few letters right. It gets worse and worse and worse as it goes along because it's lazy and it. 1:41:07.808 --> 1:41:20.025 Doesn't really care about. And then when you I noticed if you tell it it's wrong, it'll get better. It's almost kind of like reinforcing an animal or something. It's like, 'Oh, I'm so sorry.' You and it would get a little bit better, but they're never. 1:41:20.667 --> 1:41:46.637 Yeah, and then the ultimate answer was to actually give it like small snippets of code to generate or compile because again that the web is full of so much code, and those systems are also tuned for doing a certain amount of code challenges and problems. So the hardest reasoning problem you can give an LM actually is to do coding. So you combine that with like asking for some information or like trying to hide some information, you can actually trick it to do quite amazing. 1:41:47.009 --> 1:42:23.931 Things. Oh, interesting. So okay, so you know I think that yeah when people think of LMs, a lot of people have used ChatGPT, Perplexity, these other things, and that's pretty straightforward from like a product standpoint use case where you go to chatgpt.com and you're literally just a blank screen with a text box. You type what you want, and so it's as simple as product you can make, and it's because their technology is really impressive and they just want to focus on that. What are some other places, you know, maybe less obvious where LMs are being used in the real world? 1:42:25.011 --> 1:42:39.962 Yeah, so I was mentioning things which LMs aren't good at. And an additional one which LMs are not good at is um since they're trained on mostly public data, they really know nothing about enterprise systems, right? Nothing about AskIt. 1:42:39.962 --> 1:42:42.089 About your email things in your email, for example. 1:42:43.000 --> 1:44:01.860 Yeah, yeah. And like similarly, if you're if you're in a company which does um, you know, supply chain management of like parts and things for aircraft, you can't ask it about like what's the part number or what's the what—what parts do you need for a certain maintenance operation on a plane? Because that's just not something that's generally available to LMs to be trained on. So um one of the techniques you can do with LMs is you can do something called Retrieval Augmented Generation, where you feed in a body of this additional knowledge, additional information from could be from a from a database, from a bunch of documents from some other source which is not public. Um you embed it in a vector database, and um then when you query the LM, you first query the vector database—the vector store. You ask it for information which relates to this. You pass that on to the LM as context. And now the LM, which is very good at answering abstract questions, now becomes an expert in this new data set and this new knowledge set, but with kind of the same limitations where it's not good at reasoning. It's not good at all a bunch of things, and this produces another problem. 1:44:03.949 --> 1:44:56.616 Which is um so it's getting back to the aircraft example. Let's say let's say I'm a technician. I need to perform maintenance on the fuselage of an aircraft, and I want to know like for this particular aircraft model what part number do I need to do this repair which I'm trying to execute. Um now if it's not exactly in the source material—if like maybe the relationship between like the maintenance operations and the parts isn't clearly spelled out, the LM won't say, 'Oh, I don't I don't have that information; like you should maybe talk to somebody who's done this before.' What this is going to say is it's going to say, 'Oh well, if you're doing this sort of operation, there's a number of different things which you could use for this, and I'll give you a list of different parts which are totally plausible, and I'll tell you any of these would work.' And just go ahead and oh man, and I think you're 1:44:57.038 --> 1:45:00.580 explaining a lot about like bowing doors falling off airplanes. 1:45:02.500 --> 1:45:03.974 I hope they're not using 1:45:03.974 --> 1:45:15.432 the lms for the maintenance on bowing planes but as you can see that it's for enterprise use you can actually get pretty far with an lm but it has problems with what we're currently 1:45:17.060 --> 1:45:53.451 referring to as hallucinations um we're basically the model will extrapolate information which may or may not be true and um depending upon how good the source information is um how good the encoding is you can you can get to um some customer support systems they get to maybe you know 60 70 accuracy or accepted answers when um a customer support representative looks at this and says is this the right answer to to give a customer but the question is when when you get to that accuracy let's say you get to 80 or 90 accuracy is that good 1:45:55.375 --> 1:47:55.221 enough for aircraft maintenance for supply chain management for fraud detection like there's a whole bunch of critical use cases and like like is is that good enough is that accuracy good enough and then the second question is if when it's wrong how do i know it's wrong how do i how do i check the system how do i explain how it got the results so um this this is actually the the subject of my talk which is there is there's another way which you can um encode um enterprise information which has been in use for a while it's called um knowledge graphs or property graphs basically the same thing um and a lot of expert use systems will use knowledge graphs as the system of truth for capturing data kind of building expert systems but the typical problem with those is to build an expert system on top of a knowledge graph you need an interface so you need to build like an application like you have to have a bunch of like queries and drop down menus and things for people to find information so wouldn't it be nice if you could ask the lm but then have it give you information from a knowledge graph instead of just randomly pulling information out of a vector store and so this technique for pairing knowledge graphs and um lms together is called graph rack it's a really effective way of improving the accuracy of your results um there was a study recently done i believe by by gartner where they showed a 54 increase in accuracy just by switching to knowledge graphs versus like traditional vector stores oh okay um it makes it easier to explain the results because part of the knowledge graph gets passed into the lms context and knowledge graphs unlike um vector databases actually you can you can reason about them you can see what are the nodes or the relationships you can actually start to understand why the the lm was giving a correct or incorrect answer and then you know go back to the source data 1:47:55.407 --> 1:48:05.734 And fix it? So is the LM translating your English request into a Knowledge Graph query? Is that how it's working, or is it just more fundamentally integrated with the Knowledge Graph? 1:48:06.291 --> 1:49:14.517 Yeah, so it's doing a couple things. So um when you take the source material and you put it inside of a graph database that also supports vector search, it's doing a standard vector encoding using word vectors of the embeddings. It's also using an LM to create a graph. And you can—you can hand-make a Knowledge Graph; you can also tweak the knowledge the resulting Knowledge Graph. But a really quick way of doing this is to actually use an LM to generate the Knowledge Graph off the source material as well. And once you have a Knowledge Graph plus the vector database and linkages between them, now you can basically feed it into the vector store, the question gets back some embeddings, see what nodes they're associated with, and pull the related nodes, and then feed that all those context into the LM. And so basically it's a better vector search because it's vector search which includes Knowledge Graphs and real data coming from the 1:49:15.479 --> 1:49:38.682 LM? Yeah, that is super cool. Okay, so um to have folks deployed GraphRAG in different industries and tell us maybe like a good story/bad story kind of thing, you know what's a what's something maybe hilarious that came out of it, um you know, and then what's also something where kind of at scale you had kind of an aha moment or eureka moment. 1:49:39.475 --> 1:51:38.020 Yeah, so I think a lot of our customers are using Graph RAG and graph databases specifically to solve the problem of getting higher accuracy on RAG. Some of our customers are using it for, in particular, customer service systems. That's one of the common scenarios. A second one in general is recommendation engines. Oh, it makes sense. So you're trying to give back like better search results, like better recommendations to end users. And actually, we had an interesting use case where this is a different use of LLMs. It still generally falls under Graph RAG, but it's more of like a research system than it is a query system, where one of our customers maintains oil fields and kind of that oil infrastructure. And you know, making sure that the supply chain is uninterrupted is important for them. But there are so many data points in terms of conditions and maintenance issues and weather conditions that it becomes very hard to even understand root cause analysis on why things are getting delayed or slowed down. So what they did is they fed huge amounts of data into an LLM using a graph, a graph representation of this using a graph store, and then had the LLM start to reason and give some potential answers about where the issues were with, you know, outages or supply chain issues or things in there, and got some interesting insights. Now it's a humongous model, very slow to load up the massive quantities of data and things which they did, but from a research standpoint, they got some really valuable insights which would have required like a humongous amount. 1:51:38.020 --> 1:52:34.080 of human manpower and research to actually go through the data and build those insights. So I think there's a variety of different use cases for it, and I'd say the biggest challenge—and these are all successful, production-ready systems I talked about—but the biggest challenge in the kind of the white elephant to the room related to LLMs and general GenAI architectures is it's all startups and very little in production. So like when you actually get to the gallon to it and you're like, okay, well how many people are using LangChain or a Llama or these models in production? And they're like, oh, we have a really promising system and like we're getting the accuracy up and like you know we're almost there. But I think a lot of folks are almost there and are kind of looking for the technology to mature. 1:52:35.599 --> 1:53:46.879 Yeah, and also for the cost to be reasonable as well. I think that it doesn't doing research and doing development with LLMs makes sense because the technology is not that expensive to prototype with when we were doing at scale large volumes of data—a huge amount of resources and processing and GPUs which you need to then execute the building of knowledge graphs or in general building of RAG and search on RAG, that can be quite expensive. So I think also enterprise use cases are a great place where the benefit-cost trade-off makes a lot of sense. If you're just doing like this for a general consumer system, you don't want everyone just firing off queries randomly unless you have a huge amount of capital like OpenAI, right? But if you're in a corporate system where it's customers doing queries and they're solving business problems, then it makes a lot of sense that I can pay the token fees and everything, but then it's saving me and my customers time and money because we're able to get answers faster and easier than we would if we were going through a human workforce. That makes 1:53:47.655 --> 1:54:30.096 sense. And so just kind of one last question on the theme of MLOps. You know, if someone wants to go into the MLOps profession, what is your best advice for them? You know, let's say they are finishing high school and they have a choice between getting a four-year degree going into a boot camp—should they try to get a degree really specific on MLOps or just a general computer science or even a general math degree? How do you feel like folks should navigate that? 1:54:30.923 --> 1:55:08.115 Yeah. No, I think that it's there's a lot of great options for technology degrees. Despite what folks say, including ShowMeBenheim in the keynote, stay at swamp up. Developer jobs are at risk. Like I think that AI technology continues to build and create new opportunities and new technical challenges that you need really smart people to solve. And my best advice for folks would be the real skill set you need to learn in college is how to solve hard 1:55:09.769 --> 1:55:20.339 problems. Yeah. And so if you feel challenged, if you're in a degree program or if you're taking something where you feel like you're learning new material, you're able to 1:55:22.019 --> 1:55:36.364 you know use your expertise to solve problems to to reason about things to to make a difference that skill set the the ability to understand pick up a new challenge reason about it lms don't reason humans do right and and 1:55:36.499 --> 1:57:35.535 actually come up with some creative solutions that's what's going to be the valuable skill set going forward and you know maybe maybe we're not going to be like you know sitting there and like like coding basic algorithms for for doing you know list sorting and all that stuff in the future i mean hopefully people aren't doing that other than like bergamy one-on-one because machines are better at that and they're better at optimizing those things than we are right but hopefully we're the ones who are actually taking the real world problems figuring out how to solve the hard problems and actually ml ops is a great example of this because um figuring out how to observe how to secure and how to make sure that machine learning models are being taken from the developer all the way through to production is a hard problem and a machine isn't going to solve this for us this is something which you need people who understand the problem they understand the space they can do it actually even my talk here at swamp up is about using one of the tools in that tool chain artifactory which you can use as a model repository pulling information out of it feeding it into a knowledge graph using the neo4j knowledge graph builder and in a in a couple hours on a on a weekend or an afternoon you can basically have your your own lm with enterprise information from your machine learning pipeline and then start asking questions like oh which which dependencies um have an mit license on them or like what's what's what library what's the latest version of this library like you can start asking your own little system these questions cool so i think that it's an exciting time for folks who are in technology professions because if you're able to learn to reason and to create and understand um kind of new new problems and new challenges then you'll have um pretty much work no matter what field or um degree program you go into 1:57:36.244 --> 1:57:50.993 that is great advice steven yeah i think uh spot on uh totally agree um cool it's awesome having you here um at at a swamp up here in austin and uh i look forward to chatting with you later on thank you so much for uh entertaining the folks yeah 1:57:51.415 --> 1:57:56.545 no thanks for having me on the show and i'd love to join the um discussion on discord as well cool that'd 1:57:56.663 --> 1:57:58.340 be great we'll look forward to it 1:58:03.380 --> 1:58:14.980 uh 1:58:33.380 --> 1:58:36.060 And share alike in kind.