1 00:00:00,151 --> 00:00:02,083 Sean Haley, how are you today? 2 00:00:02,083 --> 00:00:03,453 Good to see you. 3 00:00:03,453 --> 00:00:05,655 I really appreciate you guys joining me, man. 4 00:00:05,655 --> 00:00:12,260 I enjoyed your Michael Scott paper quite a bit and we're going to talk about it today. 5 00:00:12,260 --> 00:00:14,602 But before we do, let's get you guys introduced. 6 00:00:14,602 --> 00:00:20,985 So you're both academics at University of, is it Oklahoma? 7 00:00:21,766 --> 00:00:22,786 University of Oklahoma. 8 00:00:22,786 --> 00:00:24,027 In my notes, have Arkansas. 9 00:00:24,027 --> 00:00:25,988 That's wrong. 10 00:00:26,068 --> 00:00:29,450 Why don't, Hailey, we start with you, introduce yourself and then we'll... 11 00:00:29,450 --> 00:00:30,811 Hand it over to Sean. 12 00:00:31,017 --> 00:00:31,387 Sure. 13 00:00:31,387 --> 00:00:33,128 So I'm Haley Stilwell. 14 00:00:33,128 --> 00:00:37,950 I am an associate professor of law at the University of Oklahoma College of Law. 15 00:00:37,950 --> 00:00:40,211 I teach evidence and criminal law. 16 00:00:40,211 --> 00:00:45,813 I write in those fields as well, but I also have a particular interest in exploring AI in my scholarship. 17 00:00:45,813 --> 00:00:49,015 And that's kind of where Sean and I's relationship began. 18 00:00:49,015 --> 00:00:49,255 Yeah. 19 00:00:49,255 --> 00:00:50,025 And I'm Sean Harrington. 20 00:00:50,025 --> 00:00:57,178 I'm the director of technology innovation here at OU Law, which is kind of a nebulous title, you know, at different institutions that means different things. 21 00:00:57,178 --> 00:00:59,711 Here what it means is I teach AI in the practice of law. 22 00:00:59,711 --> 00:01:06,231 And I also run what's called their digital initiative, which gets tech training to all the students, essentially. 23 00:01:06,326 --> 00:01:09,869 And I met someone from your law school. 24 00:01:09,869 --> 00:01:10,440 Yeah. 25 00:01:10,440 --> 00:01:11,930 Was it Kenton? 26 00:01:12,792 --> 00:01:13,572 Yeah. 27 00:01:13,572 --> 00:01:19,107 Um, we were, he happened to be attending a session that I was helping, helping facilitate and sat at my table. 28 00:01:19,107 --> 00:01:23,601 So it was pretty cool when I saw your paper and was like, yeah. 29 00:01:23,601 --> 00:01:26,404 And connected the dots and, here we are today. 30 00:01:27,626 --> 00:01:28,459 So 31 00:01:28,459 --> 00:01:29,190 to your podcast. 32 00:01:29,190 --> 00:01:31,432 And so he was like, my gosh, you got to go on that. 33 00:01:31,432 --> 00:01:33,373 I was like, Okay, yeah, I'm in. 34 00:01:33,434 --> 00:01:34,014 Good. 35 00:01:34,014 --> 00:01:34,374 Good. 36 00:01:34,374 --> 00:01:34,594 Yeah. 37 00:01:34,594 --> 00:01:44,574 We started off, you know, we started off primarily in the knowledge management and innovation realm at large law firms as our audience. 38 00:01:44,574 --> 00:01:46,174 And it's really evolved from that. 39 00:01:46,174 --> 00:01:49,954 I mean, we have, I've got people from the marketing function. 40 00:01:49,954 --> 00:01:52,985 I've got attorneys, I've got managing partners, I have academics. 41 00:01:52,985 --> 00:01:59,721 So, you know, the innovation topic is very timely in the legal world. 42 00:01:59,721 --> 00:02:00,477 right. 43 00:02:00,477 --> 00:02:03,718 Like the timing was, the timing was very fortunate for us. 44 00:02:03,718 --> 00:02:07,940 So I started right around the time chat GPT was released. 45 00:02:07,940 --> 00:02:08,740 Yeah. 46 00:02:08,740 --> 00:02:10,260 So, I mean, just pure luck. 47 00:02:10,260 --> 00:02:13,181 Like I didn't see that and go, we need a podcast. 48 00:02:13,181 --> 00:02:18,754 was just, we had been selling into not the world of knowledge management for many, many years. 49 00:02:18,754 --> 00:02:26,164 And then we saw this interesting dynamic where all saw these innovation roles start to pop up and 50 00:02:26,164 --> 00:02:26,958 Right. 51 00:02:27,244 --> 00:02:31,688 started having conversations with, what's the difference between knowledge management and innovation? 52 00:02:31,688 --> 00:02:35,612 And those conversations were very enlightening. 53 00:02:35,612 --> 00:02:37,372 And I thought, you know what? 54 00:02:37,473 --> 00:02:40,115 I should record these and make a podcast. 55 00:02:40,115 --> 00:02:54,757 So I recorded three episodes, didn't tell anyone about it, posted them, you know, just on the usual platforms, Spotify, Apple podcasts, like three weeks before, Ilta Ilta con where 56 00:02:54,757 --> 00:02:55,338 we met. 57 00:02:55,338 --> 00:03:00,578 Kenton and I had like 10 people come up to me in the booth and go, Hey, I your podcast. 58 00:03:00,578 --> 00:03:01,018 It's great. 59 00:03:01,018 --> 00:03:03,398 I was like, how the hell did you find it? 60 00:03:03,398 --> 00:03:12,958 And I guess people just, I don't know if they keep Google alerts on certain keywords, but, um, that was almost a hundred episodes. 61 00:03:12,958 --> 00:03:15,458 We're going to have our hundredth episode here in a few weeks. 62 00:03:15,458 --> 00:03:18,115 And, uh, yeah, it's been, it's been a lot of fun. 63 00:03:18,115 --> 00:03:18,495 awesome. 64 00:03:18,495 --> 00:03:27,835 mean, the thing is you go to something like Ilta and you just have, you know, 100 of those types of conversations and you run into Damien real or whatever, you know, and you have a 65 00:03:27,835 --> 00:03:32,075 conversation like, man, I wish I had somebody to save this, you know, other people could repurpose this. 66 00:03:32,075 --> 00:03:37,175 I want to have so I have this little, uh, have you seen the limitless AI pendant? 67 00:03:37,334 --> 00:03:42,692 No, but I've used, I've actually used the, think it's called plod, which I think it's similar. 68 00:03:43,437 --> 00:03:43,849 Yeah. 69 00:03:43,849 --> 00:03:46,802 And so he just sticks right here and then you have that knowledge base for forever. 70 00:03:46,802 --> 00:03:50,816 And I'm like, man, anytime I want to go to one of those conferences, I'm to be wearing that thing. 71 00:03:50,816 --> 00:03:59,666 Yeah, you know, the whole privacy thing around that, like, do you have to tell everybody like, hey, I'm recording that, you know, I was, yeah. 72 00:03:59,666 --> 00:04:05,546 of don't have like the social etiquette for it because obviously, you know, people knew that you were recording them, they would speak to you very differently. 73 00:04:05,546 --> 00:04:13,146 But you want the other conversation because the utility of retaining that information and having it in a data set that you could manipulate is like so high. 74 00:04:13,146 --> 00:04:20,886 It's just like, we're gonna have to figure out something like a little green, you know, light on my lapel that says Sean's always recording or something. 75 00:04:20,886 --> 00:04:22,406 think I turn it off. 76 00:04:22,406 --> 00:04:25,146 Nobody's want Nobody's gonna want to talk to me anyways. 77 00:04:25,146 --> 00:04:26,582 That's only gonna make it worse. 78 00:04:26,582 --> 00:04:29,211 in the legal crowd, right? 79 00:04:29,211 --> 00:04:31,918 I mean, they're very privacy sensitive. 80 00:04:31,918 --> 00:04:37,265 I had a faculty member here that I was just explaining what the thing did. 81 00:04:37,265 --> 00:04:38,096 It was in its dock. 82 00:04:38,096 --> 00:04:39,149 I was like, it's not recording. 83 00:04:39,149 --> 00:04:40,551 I was explaining what it was. 84 00:04:40,551 --> 00:04:44,115 And she made up an excuse to get up and leave my office almost immediately. 85 00:04:44,383 --> 00:04:45,003 That's funny. 86 00:04:45,003 --> 00:04:45,423 Yeah. 87 00:04:45,423 --> 00:04:54,763 I mean, we, um, we use a platform called fireflies for our meetings and 100 % of our, of our client base is law, our law firms, large law firms. 88 00:04:55,183 --> 00:04:59,983 And, um, surprisingly we don't get a lot of pushback on that. 89 00:05:00,063 --> 00:05:09,803 And, you know, but I do think that people manage their candor when they see and, and rightly so I get it. 90 00:05:09,803 --> 00:05:11,319 Um, but 91 00:05:11,355 --> 00:05:14,071 Anyway, let's talk about your paper. 92 00:05:14,071 --> 00:05:15,706 So it really caught my eye. 93 00:05:15,706 --> 00:05:19,112 It was, and the reason is because I'm a big office fan. 94 00:05:19,484 --> 00:05:20,464 Love it. 95 00:05:20,464 --> 00:05:22,324 So the title was all Haley. 96 00:05:22,324 --> 00:05:24,264 She came up with that. 97 00:05:25,924 --> 00:05:27,264 Oh, yeah. 98 00:05:27,264 --> 00:05:27,535 Yeah. 99 00:05:27,535 --> 00:05:29,077 they're great. 100 00:05:29,077 --> 00:05:35,343 So if I remember correctly, the title was Michael Scott is not a juror. 101 00:05:36,244 --> 00:05:36,766 Okay. 102 00:05:36,766 --> 00:05:40,654 And, um, give us a little bit of background on the paper. 103 00:05:40,654 --> 00:05:45,203 Uh, I don't know whoever wants to start and then we can jump into that a little bit. 104 00:05:45,321 --> 00:05:46,491 Yeah, you started, it your idea. 105 00:05:46,491 --> 00:05:51,744 Yeah, so the origin story kind of comes from my just academic research in general. 106 00:05:51,744 --> 00:06:01,828 There's this standard that's often utilized em in courts called the reasonable juror standard where courts are basically trying to determine what a perfectly behaving 107 00:06:01,828 --> 00:06:07,210 impartial juror that follows all the court's directions, how they would think and understand evidence. 108 00:06:07,210 --> 00:06:14,387 And it comes into play a lot when em courts are assessing whether the admission or exclusion of evidence was 109 00:06:14,387 --> 00:06:16,939 in violation of a rule or the constitution. 110 00:06:16,939 --> 00:06:26,874 And so I was bothered by some of the court's reasonable jury determinations because it just didn't track at all with how I thought real people would think and understand about 111 00:06:26,874 --> 00:06:27,985 certain evidence. 112 00:06:27,985 --> 00:06:38,390 So that prompted me to want to create a mock jury study to compare what the court has determined reasonable jurors would think and compare it to what real people would think 113 00:06:38,390 --> 00:06:43,301 and kind of highlight where the court has got it wrong and in turn results in 114 00:06:43,301 --> 00:06:45,922 probable constitutional violations. 115 00:06:45,922 --> 00:06:57,508 And so with that project, what I wanted to then do is find a way to empirically ground the reasonable juror where possible to bring the reasonable juror and real jurors closer 116 00:06:57,508 --> 00:07:00,389 together when courts were making those determinations. 117 00:07:00,389 --> 00:07:06,682 And the easiest way to do that is with mock jury data like we did in this study, but it takes way too much time. 118 00:07:06,682 --> 00:07:08,867 It's way too expensive to do. 119 00:07:08,867 --> 00:07:20,567 And so when uh Sean and I were kind of talking about it, we got the idea, what if we could create a mock jury bot that could output mock jury information instantaneously to help 120 00:07:20,567 --> 00:07:27,852 inform courts when they're making these determinations to protect criminal defendants and all people that are appealing their cases. 121 00:07:27,852 --> 00:07:38,704 And so we thought it would be pretty easy, just use existing platforms to feed them, you know, what this, the demographic profile of a juror and spit out information and 122 00:07:38,704 --> 00:07:41,156 Hopefully it would track what real jurors would do. 123 00:07:41,156 --> 00:07:43,587 And that is just not even close to what happened. 124 00:07:43,587 --> 00:07:44,958 And that's where this paper came. 125 00:07:44,958 --> 00:07:45,308 Yeah. 126 00:07:45,308 --> 00:07:50,642 And like the other thing is, you know, we looked up a bunch of studies that were trying to do this and it's like sample size, 112 people. 127 00:07:50,642 --> 00:07:56,566 So there just wasn't a lot of research and uh actual surveys of humans done on this stuff. 128 00:07:56,566 --> 00:08:00,669 So, you know, you quickly realize that it's really expensive to survey people. 129 00:08:00,669 --> 00:08:05,773 And so there just wasn't like a good academic research in this area. 130 00:08:05,773 --> 00:08:06,649 So we're like, well, 131 00:08:06,649 --> 00:08:10,635 That's even another reason then to integrate AI into this thing. 132 00:08:10,801 --> 00:08:11,111 Yeah. 133 00:08:11,111 --> 00:08:20,087 And you know, I, we talked to, we get a lot of attorneys on the show, some practicing, some not the ones who practice. 134 00:08:20,087 --> 00:08:28,072 I always like to ask, you know, what are the valuable use cases with AI trial prep is near the top of the list. 135 00:08:28,072 --> 00:08:31,034 And the data doesn't show this. 136 00:08:31,034 --> 00:08:33,396 really it's kind of drafting due diligence. 137 00:08:33,396 --> 00:08:36,923 you see as kind of the most common, document review. 138 00:08:36,923 --> 00:08:38,559 Yeah, document review would seem like fun. 139 00:08:38,559 --> 00:08:41,859 Yeah, it seemed to be the most common use cases cited. 140 00:08:41,879 --> 00:08:49,650 But when I hear about in terms of value, we had a guy on the podcast, name's Stephen Embry. 141 00:08:49,650 --> 00:08:50,750 You familiar with? 142 00:08:50,750 --> 00:08:51,650 Yeah. 143 00:08:51,810 --> 00:08:58,650 So Steve is a former, he was a 30 year attorney at I believe Finnegan, which is a, they're a client of ours. 144 00:08:58,650 --> 00:09:00,930 They're an intellectual property firm. 145 00:09:01,270 --> 00:09:04,914 And he was telling me how 146 00:09:05,111 --> 00:09:19,716 either him or his colleagues would use it for trial preparation in basically having AI challenge the evidence in some way, like ask all of the questions that opposing counsel 147 00:09:19,716 --> 00:09:22,977 might present in a courtroom. 148 00:09:22,977 --> 00:09:30,249 he said, you know, he talked about how, how useful that was, which I found very intriguing and makes perfect sense. 149 00:09:30,249 --> 00:09:32,760 Like I'm a daily user of AI. 150 00:09:32,860 --> 00:09:33,877 super familiar with. 151 00:09:33,877 --> 00:09:34,647 drive here for you. 152 00:09:34,647 --> 00:09:36,118 said, pretend. 153 00:09:36,118 --> 00:09:37,298 I said, here's our paper. 154 00:09:37,298 --> 00:09:40,469 Pretend that you're Ted and ask me some stuff. 155 00:09:40,469 --> 00:09:45,541 I mean, it really lowers the temperature coming into the environment because you're like, okay, I'm feeling pretty confident. 156 00:09:45,541 --> 00:09:49,172 You know, I've kind of been challenged on things that where I wasn't super confident. 157 00:09:49,172 --> 00:09:50,132 I looked a few things up. 158 00:09:50,132 --> 00:09:51,983 So it's like, you feel ready to roll. 159 00:09:51,983 --> 00:09:55,230 So I could definitely see that having a lot of value for attorneys. 160 00:09:55,230 --> 00:09:58,441 Yeah, I mean, it's almost like a practice run that you get. 161 00:09:58,441 --> 00:10:03,943 So when I heard the study, the name was catchy, so that drew me in initially. 162 00:10:03,943 --> 00:10:11,946 But then when I read what you guys were actually doing, I was like, wow, this seems like a perfect use case for AI. 163 00:10:11,946 --> 00:10:18,468 And it turns out it wasn't that perfect using the frontier models. 164 00:10:18,468 --> 00:10:21,699 Tell us about what your findings were initially. 165 00:10:22,083 --> 00:10:28,305 Well, the first thing we tried to do was just let the frontier models try to create a jury. 166 00:10:28,305 --> 00:10:34,016 So we said, create for us a jury pool that is similar to what a federal jury pool would be. 167 00:10:34,016 --> 00:10:36,438 And that's where Michael Scott emerged. 168 00:10:36,438 --> 00:10:39,100 It was really hilarious. 169 00:10:39,100 --> 00:10:41,411 They would output the demographics of the jurors. 170 00:10:41,411 --> 00:10:48,433 So it was a white man in his mid-40s who is the manager of a mid-sized paper firm in Scranton, Pennsylvania. 171 00:10:48,433 --> 00:10:52,145 which you and I would obviously know is Michael Scott. 172 00:10:52,145 --> 00:10:57,867 Michael Scott is not a real person, let alone a real juror in the federal jury pool, right? 173 00:10:57,867 --> 00:11:06,069 We also had a lot of other interesting combinations of, there was a 90 year old woman who was a part-time botanist, part-time DJ. 174 00:11:06,069 --> 00:11:07,437 I love that one. 175 00:11:07,437 --> 00:11:10,311 We had an abolitionist podcaster. 176 00:11:10,311 --> 00:11:15,824 So it seemed like when these platforms were left to their own devices, they were generating 177 00:11:15,824 --> 00:11:25,106 jurors that were more for show kind of eye catching types of backgrounds that really didn't reflect what we needed for our purposes. 178 00:11:25,106 --> 00:11:29,717 What real people on a jury would actually look like demographically. 179 00:11:29,717 --> 00:11:30,478 Yeah. 180 00:11:30,478 --> 00:11:39,560 And you can tell that, know, there a kid in, you know, Washington is using them right now to study who's 12 years old and maybe using it for creative writing. 181 00:11:39,560 --> 00:11:44,079 So, you know, there's a big range of why people are using these tools and they have the dial. 182 00:11:44,079 --> 00:11:51,150 on certain types of representation, which could be very useful, obviously, in a creative writing context, but in ours, that was, you know, catastrophic, because it was wasn't 183 00:11:51,150 --> 00:11:52,451 representing reality. 184 00:11:52,896 --> 00:11:55,888 Well, and look, I love Michael Scott. 185 00:11:55,888 --> 00:11:59,200 He'd be the last person I'd ever choose to be on a jury. 186 00:11:59,200 --> 00:12:01,301 Not, not known for good judgment. 187 00:12:01,301 --> 00:12:04,543 Um, well, and that's interesting. 188 00:12:04,543 --> 00:12:19,501 So I have had a lot of debate myself and I've seen a lot of debate in cyberspace, LinkedIn and Reddit about the frontier models coming and essentially crushing the legal specific 189 00:12:19,501 --> 00:12:27,750 tools just based on just how quickly the trajectory that they're on in terms of capabilities and 190 00:12:27,750 --> 00:12:38,941 I think your findings will first tell us what you found as potential root cause as to why this experiment was not successful with the Frontier models. 191 00:12:38,941 --> 00:12:39,461 Yeah. 192 00:12:39,461 --> 00:12:45,143 So first of all, you know, they have lengthy system prompts, which people, you know, leak on GitHub all the time. 193 00:12:45,143 --> 00:12:50,608 And you can see kind of why they're training these models to do things like be a good study tool for a student. 194 00:12:50,608 --> 00:12:57,852 And then we also just saw that just rampant bias on certain topics that just was completely at odds with reality. 195 00:12:57,852 --> 00:13:07,117 And so when we were looking, you when we actually finally ran our actual mock juror surveys, what we discovered is that people are like fairly moderate on average. 196 00:13:07,117 --> 00:13:10,320 And these would be kind of extreme versions of people. 197 00:13:10,320 --> 00:13:17,727 And again, you know, I think it's just because they're, you know, kind of meant to do everything instead of replicate juror juror scenarios. 198 00:13:17,727 --> 00:13:18,787 What else? 199 00:13:19,108 --> 00:13:24,132 Well, I think to covering kind of the second part of our experiment is important to explain that as well. 200 00:13:24,132 --> 00:13:33,380 After we left the platforms to their own devices, that's where the mock jury data came in to where, OK, we were going to tell the platforms now. 201 00:13:33,518 --> 00:13:35,409 here are the demographics of the jurors. 202 00:13:35,409 --> 00:13:37,099 You're not gonna pick them anymore. 203 00:13:37,099 --> 00:13:38,349 No more Michael Scott's. 204 00:13:38,349 --> 00:13:38,699 So. 205 00:13:38,699 --> 00:13:44,451 part coded in all of the demographics with a lengthy system prompt ah using Lang chain if anyone cares. 206 00:13:44,451 --> 00:13:54,354 Yeah, so we use basically the demographics from the real humans and told the platforms generate those exact humans and predict how incriminating this piece of evidence was. 207 00:13:54,354 --> 00:13:58,395 So we had an apples to apples comparison, how accurate were these platforms? 208 00:13:58,395 --> 00:13:59,956 And they weren't very accurate. 209 00:13:59,956 --> 00:14:01,368 Not only were they not accurate, 210 00:14:01,368 --> 00:14:03,340 but they were biased in different ways. 211 00:14:03,340 --> 00:14:16,733 ChatGPT was actually not very biased, then Jim and I thought the evidence was a little more incriminating on average, and then Claude was a little underrating in terms of 212 00:14:16,733 --> 00:14:17,364 incrimination. 213 00:14:17,364 --> 00:14:23,951 That was consistent basically across every type of demographic, so we saw those consistent skews in every single one. 214 00:14:23,951 --> 00:14:26,012 you know, X type of person, was deterministic. 215 00:14:26,012 --> 00:14:32,196 So if you're X type of person, then you necessarily would believe why you would, you know, if you're a certain type person, maybe you don't like the police. 216 00:14:32,196 --> 00:14:37,419 And so you would always find them, you know, guilty, you know, those types of things just happened over and over and over again. 217 00:14:37,419 --> 00:14:41,162 And we knew from our real world human data that that's not how real people behave at all. 218 00:14:41,162 --> 00:14:43,964 You could be a type of person and think something completely different. 219 00:14:43,964 --> 00:14:48,727 And so that, you know, just the replication of stereotypes was rampant through them. 220 00:14:48,727 --> 00:14:51,515 And this was something that as we were doing it, 221 00:14:51,515 --> 00:14:56,108 as somebody who sits in these things all day effectively, I was like, yeah, they're definitely gonna do that. 222 00:14:56,169 --> 00:14:59,801 But that was the most interesting thing to Haley, understandably, I think. 223 00:14:59,801 --> 00:15:04,505 And then when I presented this topic at a couple of different conferences, that was the thing that everybody latched onto. 224 00:15:04,505 --> 00:15:09,979 Like really, these are just wildly over-representing stereotypes and replicating stereotypes. 225 00:15:09,979 --> 00:15:12,360 That's catastrophic for the law in many ways, obviously. 226 00:15:12,360 --> 00:15:14,973 So that was something where I was like, people are interested in this. 227 00:15:14,973 --> 00:15:15,677 And so we... 228 00:15:15,677 --> 00:15:22,277 Spent a long time in the paper talking about that because it was something that maybe doesn't occur to a lot of people like the system prompts. 229 00:15:22,337 --> 00:15:25,597 If you're not, maybe Ted, you know what a system prompt is. 230 00:15:25,597 --> 00:15:34,377 But I think the average chat GPT user doesn't realize before you enter anything into the system, there's a five and 10 and 20 page potentially system prompt that has all of these 231 00:15:34,377 --> 00:15:37,377 instructions, do this, don't do that type stuff. 232 00:15:37,517 --> 00:15:41,737 so those types of things, I think maybe don't occur to the average lawyer. 233 00:15:41,737 --> 00:15:45,417 And they're definitely happening on the legal tools as well. 234 00:15:45,841 --> 00:15:47,739 You know, Westlaw has a lengthy system prompt. 235 00:15:47,739 --> 00:15:51,655 Lexus says they all have lengthy system prompts that tell it what it can and cannot do. 236 00:15:51,655 --> 00:15:52,195 Interesting. 237 00:15:52,195 --> 00:16:03,144 So is this kind of the Swiss army knife scenario where, know, they've got a lot of tools, including a little, little scissors there, but if you ever had to cut a thick piece of 238 00:16:03,144 --> 00:16:05,325 canvas with it, it's just not a fit. 239 00:16:05,325 --> 00:16:07,997 So it's good at doing a lot of things. 240 00:16:07,997 --> 00:16:08,588 Okay. 241 00:16:08,588 --> 00:16:11,410 But not really anything specialized. 242 00:16:11,410 --> 00:16:13,671 Is that really the dynamic at play? 243 00:16:13,870 --> 00:16:17,612 Yeah, and that's how tell the students, know, lawyers are very worried about hallucinations. 244 00:16:17,612 --> 00:16:18,972 In large language models. 245 00:16:18,972 --> 00:16:25,715 I saw students hallucinations at this point in time are basically solved if you're using Vlex's Vincent AI for using Lexus AI fusing Westlaw. 246 00:16:25,715 --> 00:16:27,476 It has a hyperlink to the case. 247 00:16:27,476 --> 00:16:30,477 It's your duty as an attorney to click that and go read the case. 248 00:16:30,477 --> 00:16:31,698 That's why they pay you the big bucks. 249 00:16:31,698 --> 00:16:33,899 But if you're using chat GBT, that's not the case. 250 00:16:33,899 --> 00:16:38,059 And so I think, you know, it is, think the Swiss Army Knife is a really good call. 251 00:16:38,059 --> 00:16:41,099 You're using a Swiss Army Knife when you should be using a skillsaw or something. 252 00:16:41,099 --> 00:16:46,159 And that skillsaw is Lexus, know, Vlex, whatever it is, which is a law specific tool. 253 00:16:46,159 --> 00:16:54,859 And here, what our results really showed is that these platforms were not good at replicating the noise in the real human responses. 254 00:16:54,859 --> 00:17:00,419 You know, it really seemed like demographics ended up being deterministic of what 255 00:17:00,455 --> 00:17:08,084 how incriminating they thought each evidence was, but we saw in our actual real human data that there was a lot more variation among demographics. 256 00:17:08,084 --> 00:17:18,587 so really at a high level, these platforms are just not equipped to replicate human judgment the way it actually comes out with real humans and their varying backgrounds. 257 00:17:18,587 --> 00:17:18,867 Yeah. 258 00:17:18,867 --> 00:17:22,250 And you know, I mean, I've actually noticed some interesting dynamics. 259 00:17:22,250 --> 00:17:26,773 The safety and alignment protocols are implemented very differently. 260 00:17:26,773 --> 00:17:35,568 Um, you know, there's four models that I use primarily, which are in chat, GBT, Claude, Gemini and Grok. 261 00:17:35,568 --> 00:17:40,912 And really I use them in that order in terms of frequency. 262 00:17:40,912 --> 00:17:45,685 but I ran into a really interesting little exercise where 263 00:17:45,823 --> 00:17:50,395 I saw there was a graphic on LinkedIn and it was a caricature of two people that I know. 264 00:17:50,395 --> 00:17:52,436 Um, in fact, they're both podcasters. 265 00:17:52,436 --> 00:17:55,217 was Richard Trowman's and sac of Bramowitz. 266 00:17:55,217 --> 00:18:02,380 And they had this picture and I was like, the guy who's supposed to be Richard Trowman's looks just like this actor, but I can't think of who it is. 267 00:18:02,380 --> 00:18:12,164 So I took a quick screenshot and put it into, I think I did Claude first and was just like, what famous actor does this per does the person given the thumbs up look like? 268 00:18:12,204 --> 00:18:13,458 And it said, 269 00:18:13,458 --> 00:18:14,228 I'm sorry. 270 00:18:14,228 --> 00:18:17,593 I actually have the, let's see if I can find it real quick. 271 00:18:17,593 --> 00:18:19,854 what it said, cause it was really interesting. 272 00:18:19,854 --> 00:18:28,569 it basically refused to do it and it did so on the basis of it was trying to be, here we go. 273 00:18:28,569 --> 00:18:29,189 Let's see. 274 00:18:29,189 --> 00:18:35,262 It says, I see the image shows two people in what appears to be a promotional blah, blah. 275 00:18:35,262 --> 00:18:35,965 However, 276 00:18:35,965 --> 00:18:48,165 I can't identify who specific people in images look like or compare them to famous individuals as this could involve making assumptions about people's identity based on 277 00:18:48,165 --> 00:18:49,385 their appearance. 278 00:18:49,405 --> 00:18:52,545 And I was like, all right, that's interesting. 279 00:18:52,545 --> 00:18:54,125 Let me go to ChatGPT. 280 00:18:54,125 --> 00:18:56,525 It's a little goofy, but okay, whatever. 281 00:18:56,525 --> 00:19:03,085 They're probably trying to solve some sort of a bias. 282 00:19:03,085 --> 00:19:05,361 or privacy thing or like you said, bias. 283 00:19:05,361 --> 00:19:08,939 It doesn't want to say that I look like somebody stereotypically a thing, whatever. 284 00:19:08,939 --> 00:19:09,510 Yeah. 285 00:19:09,510 --> 00:19:10,852 So I went to ChatGBT. 286 00:19:10,852 --> 00:19:12,905 ChatGBT just flat out refused. 287 00:19:12,905 --> 00:19:15,188 It just said, I cannot help you with this request. 288 00:19:15,188 --> 00:19:16,891 Gemini did it. 289 00:19:16,891 --> 00:19:20,207 I still don't know who the actor was, um unfortunately. 290 00:19:20,207 --> 00:19:22,027 So that's surprising when we were running. 291 00:19:22,027 --> 00:19:23,529 we're talking about refusal rates. 292 00:19:23,529 --> 00:19:26,822 When we were running ours, Jim and I had the highest refusal rate, right? 293 00:19:26,822 --> 00:19:27,703 Jim and I had it. 294 00:19:27,703 --> 00:19:32,717 And even for the nerds in the audience, even through the API, it would say, sorry, I can't do that. 295 00:19:32,717 --> 00:19:34,479 I can't, I can't engage in legal advice. 296 00:19:34,479 --> 00:19:40,724 Now subsequently, you know, and this is all obviously moving, moving terrain under our feet at all times. 297 00:19:40,724 --> 00:19:42,766 So you can do it one day and it'll refuse to do it the next day. 298 00:19:42,766 --> 00:19:45,869 We'll do it because they've dialed something back or down. 299 00:19:45,869 --> 00:19:47,864 But that was um for us, it was 300 00:19:47,864 --> 00:19:50,426 Gemini and Claude were the two highest refusal rates. 301 00:19:50,426 --> 00:19:52,097 ChatGPT was a little bit better. 302 00:19:52,097 --> 00:19:57,561 And even through the API, I just like, I'm not going to engage in, you know, essentially legal advice. 303 00:19:57,561 --> 00:20:00,273 And you know, what you realize is there was a time there. 304 00:20:00,273 --> 00:20:09,449 And I think that that time has probably passed where, know, if the state of California can sue legal zoom for forms and that's considered unauthorized practice of law, these are 305 00:20:09,449 --> 00:20:12,802 pretty uncomfortable with, you know, anything kind of approximating legal advice. 306 00:20:12,802 --> 00:20:14,783 Now, now they give legal advice, no problem. 307 00:20:14,783 --> 00:20:16,290 They'll issue a bunch of caveats. 308 00:20:16,290 --> 00:20:18,775 but they will give you lots of legal advice. 309 00:20:18,775 --> 00:20:24,327 I don't know what movement behind the scenes has happened, but they all kind of collectively said, we can give some legal advice. 310 00:20:24,327 --> 00:20:25,100 That's what it seems like. 311 00:20:25,100 --> 00:20:26,220 You should try your picture out again. 312 00:20:26,220 --> 00:20:26,821 Yeah. 313 00:20:26,821 --> 00:20:27,633 now it'll do it. 314 00:20:27,633 --> 00:20:28,079 Yeah. 315 00:20:28,079 --> 00:20:28,990 that's a good point. 316 00:20:28,990 --> 00:20:32,063 This was, this was probably a couple of months ago. 317 00:20:32,063 --> 00:20:36,308 So, yeah, there's not a date on this, but Grok did it, of course, which you would expect. 318 00:20:36,308 --> 00:20:39,282 Grok, not a lot of guard rails with Grok. 319 00:20:39,282 --> 00:20:45,067 And, you know, did, did you guys try Grok at all in this experiment? 320 00:20:45,192 --> 00:20:47,442 No, so we just stuck with those three. 321 00:20:47,442 --> 00:20:52,057 I just for a dumb reason, I just didn't have a GROCK subscription at that point so I can get to the API. 322 00:20:52,057 --> 00:20:53,188 So I didn't, I do now. 323 00:20:53,188 --> 00:20:55,370 And there's some stuff where GROCK is really useful. 324 00:20:55,370 --> 00:20:56,640 know, first of all, it's very fast. 325 00:20:56,640 --> 00:20:57,831 just in the chat interface. 326 00:20:57,831 --> 00:21:00,394 And then number two, it performs really, really well. 327 00:21:00,394 --> 00:21:09,761 think, you know, Elon Musk has kind of been in the political news so much that they don't report on his model the same way, but that it's a really interesting and powerful model. 328 00:21:09,761 --> 00:21:11,822 And the other thing is like, if you need 329 00:21:11,860 --> 00:21:19,204 You know, if you're on your phone and you need kind of a survey of kind of what is going on in social media, it's really useful because as Twitter data. 330 00:21:19,947 --> 00:21:22,087 I've done stock stuff for my wife with it. 331 00:21:22,087 --> 00:21:27,113 yeah, I was curious if there were any different results with Grok. 332 00:21:27,113 --> 00:21:32,297 Well, what about, um okay, so you tried the Frontier models, weren't successful. 333 00:21:32,297 --> 00:21:34,789 What was the next step? 334 00:21:34,789 --> 00:21:37,561 It sounds like you might've tried some open source models. 335 00:21:37,703 --> 00:21:39,445 Yeah, so I went on and eat. 336 00:21:39,445 --> 00:21:47,115 So I went on the leaderboard and kind of tried to find something that we could run locally so we could save token costs, of course, and was trying to look at, you know, less 337 00:21:47,115 --> 00:21:49,097 censored models was the idea. 338 00:21:49,097 --> 00:21:59,036 And actually, a handful of the models, even one of the Mistral models that was a French company, their Frontier Lab, and even one of those wouldn't do kind of law, advice, 339 00:21:59,036 --> 00:22:02,495 practice of law, replicate human experimentation type things. 340 00:22:02,495 --> 00:22:08,587 I was able to find a less censored version of Mistral that would do kind of whatever I told it to do, which was very useful. 341 00:22:08,587 --> 00:22:10,228 It was very useful for what we're doing. 342 00:22:10,228 --> 00:22:21,501 And so then we decided to try to take all of our real human responses, turn that into a JSON training set, and then fine tune that less censored model as just kind of like, will 343 00:22:21,501 --> 00:22:22,881 this do anything? 344 00:22:22,881 --> 00:22:26,332 And it ended up being much better than any of the other models that we did. 345 00:22:26,332 --> 00:22:28,621 So that was pretty interesting. 346 00:22:28,621 --> 00:22:29,981 And it was cheap too. 347 00:22:29,981 --> 00:22:34,661 So I remember I used to fine tune models kind of early in the chat, GPC revolution. 348 00:22:34,661 --> 00:22:36,001 And it was expensive. 349 00:22:36,001 --> 00:22:41,261 I mean, it would be too even for a small model, like a 24 billion parameter model would be hundreds of dollars. 350 00:22:41,261 --> 00:22:46,121 When I did this one, it was, I think it was $14 or something. 351 00:22:47,941 --> 00:22:49,321 I use a service. 352 00:22:49,321 --> 00:22:50,741 Well, I just use tokens for mine. 353 00:22:50,741 --> 00:22:52,121 So I did it with two different packages. 354 00:22:52,121 --> 00:22:54,061 There's an open source package called oxalotl. 355 00:22:54,061 --> 00:22:58,436 And then there's a service called open pipe, which is no, I guess no code. 356 00:22:58,452 --> 00:23:04,338 As long as you have the JSON training set, you can just throw it in there and you know, do whatever model and it'll also host it for inference, which is pretty cool. 357 00:23:04,338 --> 00:23:08,421 But initially I just did it at home on my home computer with axolotl. 358 00:23:08,883 --> 00:23:10,964 Open source, fine tuning package. 359 00:23:11,387 --> 00:23:12,969 did you try llama? 360 00:23:13,102 --> 00:23:14,822 No, we so we didn't do the llama. 361 00:23:14,822 --> 00:23:18,602 Oh, so we tested the llama model, but it ended up being even worse. 362 00:23:18,602 --> 00:23:24,382 And so I couldn't even I couldn't even fine tune it enough to get it to be anywhere near human level. 363 00:23:24,382 --> 00:23:29,353 And so I was like, all right, we have to find something less censored than llama, all the meta models. 364 00:23:29,353 --> 00:23:37,413 And then after the fine tuning, were, what, was the, how was the testing after, after you fine tuned? 365 00:23:37,838 --> 00:23:42,518 So we essentially just did the same thing that we did to the real humans and scaled it. 366 00:23:42,518 --> 00:23:45,358 I think we did it up to, was it 5,000? 367 00:23:45,358 --> 00:23:45,678 Yeah. 368 00:23:45,678 --> 00:23:46,478 It was 5,000. 369 00:23:46,478 --> 00:23:50,018 So instead of, we only had 1200 actual juror responses. 370 00:23:50,018 --> 00:23:59,698 So we scaled it up to 5,000 real human responses where we would say, you know, go read this, first take on this, you know, this demographic or take on a random demographic from 371 00:23:59,698 --> 00:24:07,158 the range present in the real jurors, then read the thing, then respond to the thing as if you are from, you know, said demographic. 372 00:24:07,247 --> 00:24:08,747 And then we just scaled it. 373 00:24:08,747 --> 00:24:14,655 it's just like, just ran my computer for like an evening, basically, just let it grind through 5,000 iterations. 374 00:24:14,655 --> 00:24:18,178 And then we took that and measured it against the real human responses. 375 00:24:18,395 --> 00:24:20,192 and how did they compare? 376 00:24:20,192 --> 00:24:27,512 Much better, much better, which it was one of those things where I wasn't sure if just fine tuning was going to be enough. 377 00:24:27,512 --> 00:24:29,672 It felt like something that was more. 378 00:24:29,672 --> 00:24:36,372 You might have to have like some kind of a genetic workflow and you know, frankly, we can't afford to pre train a model of our own. 379 00:24:36,372 --> 00:24:39,732 So I, I wasn't sure if just fine tuning would be enough. 380 00:24:39,732 --> 00:24:48,959 And then the other problem is when you're fine tuning a model, even a small model, like 24 billion parameters, we had 1200 Jason. 381 00:24:48,959 --> 00:24:52,639 data points, know, JSON rows or data points to use. 382 00:24:52,719 --> 00:24:58,099 Most people do like 50,000 and we'll see benefits up to 100,000 fine tuning. 383 00:24:58,099 --> 00:25:04,259 So we just thought it was possibly just way too small of a training set to have any, you know, measurable impact on it. 384 00:25:04,259 --> 00:25:10,999 And the fact that at 1200 it did was like, Whoa, Whoa, you know, this is something that again, it was fairly cheap. 385 00:25:10,999 --> 00:25:11,899 It could do it for free. 386 00:25:11,899 --> 00:25:15,959 If you had the right hardware, like the Mac studio that we're running all this stuff on, we can do it on that. 387 00:25:16,093 --> 00:25:17,925 So it was really interesting to go. 388 00:25:17,925 --> 00:25:22,947 was like one of those breaks, know, one of the million aha moments I seem to have every month with this technology. 389 00:25:22,947 --> 00:25:24,993 It was like, whoa, we can actually do this. 390 00:25:24,993 --> 00:25:26,646 You know, it's like what it felt like. 391 00:25:26,646 --> 00:25:31,206 And when you say much better, like give me a sense of magnitude. 392 00:25:31,206 --> 00:25:32,926 Like how, how did it do? 393 00:25:32,926 --> 00:25:46,466 I don't know if this was a percentage that you use to measure alignment, but like give me a sense of, it doesn't have to be the exact number, but was it two, two X three X better 394 00:25:46,466 --> 00:25:48,206 50 % better. 395 00:25:48,655 --> 00:25:55,106 It was, geez, I don't want to state it, because I had it right here, but my computer locked. 396 00:25:55,106 --> 00:25:59,408 It was, I wrote it down just so I could specifically say this. 397 00:25:59,408 --> 00:26:02,394 I remember reading it was significant. 398 00:26:02,394 --> 00:26:12,013 95 % accuracy and geez, like essentially the margin of error was half, half of what it was with the model, the frontier models. 399 00:26:12,013 --> 00:26:17,839 So it was, you know, twice as good effectively as chat GPT trying to do this on its own. 400 00:26:17,839 --> 00:26:24,385 And so, and again, like, you know, there was a lot of technical things that I didn't do that I would do differently. 401 00:26:24,385 --> 00:26:27,188 Now I would use a different style of fine tuning if I was going to do it. 402 00:26:27,188 --> 00:26:28,330 We would have, you know, 403 00:26:28,330 --> 00:26:29,950 even scale with synthetic data. 404 00:26:29,950 --> 00:26:36,870 So that was something that I thought about, but she didn't want to, uh, cause a lot of people had just have a reflexive bad reaction to synthetic data. 405 00:26:36,870 --> 00:26:44,830 But had we taken that and then scaled it up to, know, 50,000, you know, take 12, take 1200 and then use that to scale up to 50,000. 406 00:26:44,830 --> 00:26:46,530 And then they use that to train a model. 407 00:26:46,530 --> 00:26:49,210 It probably would have been substantially better even then. 408 00:26:49,210 --> 00:26:56,710 So, uh, you know, just for a small investment and a small amount of training data, we got pretty big wins. 409 00:26:56,710 --> 00:26:58,077 And that was like the real like, 410 00:26:58,077 --> 00:26:59,601 breakthrough, I guess, of the paper. 411 00:26:59,601 --> 00:27:01,435 Especially as compared to the platform. 412 00:27:01,435 --> 00:27:07,962 Yeah, and definitely substantially better if you just use like the API to Gemini or Claude or Google or OpenAI. 413 00:27:07,962 --> 00:27:08,313 Yeah. 414 00:27:08,313 --> 00:27:10,835 And what conclusions can we draw from this? 415 00:27:10,835 --> 00:27:24,621 So back to my earlier comment about there's a, it is a constant point of discussion in the legal tech community about are these legal specific solutions going to get steamrolled by 416 00:27:24,621 --> 00:27:26,322 the frontier models? 417 00:27:26,362 --> 00:27:33,916 And your study makes me think maybe, maybe not, because if the frontier models are 418 00:27:33,916 --> 00:27:41,270 have controls, we'll call them, in place that make them a Swiss army knife where they're good for a lot of things. 419 00:27:41,270 --> 00:27:46,953 So they're a mile wide and an inch deep, as opposed to being an inch wide and a mile deep. 420 00:27:47,134 --> 00:27:57,720 It seems like, I don't know if it's a fair conclusion or not, but your study for niche legal use cases, the frontier models might not be the best solution. 421 00:27:57,720 --> 00:28:00,341 I don't know, Haley, what's your thought on that? 422 00:28:00,562 --> 00:28:03,143 think that's exactly right, at least as of now. 423 00:28:03,143 --> 00:28:14,755 I think that our project really proves the need to have these models specifically trained on human data, human judgment in the realm of evaluating evidence in these legal 424 00:28:14,755 --> 00:28:15,415 scenarios. 425 00:28:15,415 --> 00:28:23,829 I think it could be huge in terms of, as you mentioned earlier, trial prep, things that lots of trial boutiques would really latch onto and utilize. 426 00:28:23,829 --> 00:28:25,699 But right now they're just not there. 427 00:28:25,699 --> 00:28:29,310 And I think it is because of that Swiss army knife nature. 428 00:28:29,478 --> 00:28:33,250 But I do think our fine tuning model shows that it is possible. 429 00:28:33,250 --> 00:28:44,038 So with lots more data and lots more training, there could be an essentially mock jury bot at some point in the future that could be reliable for trial prep and strategy and a lot 430 00:28:44,038 --> 00:28:44,789 of other things. 431 00:28:44,789 --> 00:28:48,792 Yeah, I know, right now the big the frontier models obviously have the data problem. 432 00:28:48,792 --> 00:28:55,457 There's only four good places you can get, you know, US legal data, Vlex, now Clio, guess, Vlex still, Westlaw, Lexus, Bloomberg. 433 00:28:55,457 --> 00:28:57,096 Now those are the only data sets. 434 00:28:57,096 --> 00:29:04,192 and they haven't purchased one of those yet, which probably indicates, which, you when I hear people talk about it, it's that they just don't think it's a big enough market right 435 00:29:04,192 --> 00:29:04,401 now. 436 00:29:04,401 --> 00:29:06,254 They're going after literally everyone. 437 00:29:06,254 --> 00:29:14,400 But if at some point they do purchase one of those data sets and have that on the backend, then that's really gonna blow up the business model of like a Lexus or Westlaw or Vlex, 438 00:29:14,400 --> 00:29:22,352 unless they're offering something else in addition to it, some sort of agentic workflow, some sort of tool that plugs into your firm, which I think is. 439 00:29:22,352 --> 00:29:27,216 why probably the Clio Vlex acquisition is really interesting to me right now. 440 00:29:27,216 --> 00:29:31,170 But right now, the frontier models have a data problem and they're just not focused on legal. 441 00:29:31,170 --> 00:29:38,045 They're too big for legal, which is crazy to think about with the 1.5 billion or whatever that's going into legal investment this year. 442 00:29:38,045 --> 00:29:41,659 It's like, no, we're still considered small dollars to them, I think. 443 00:29:41,659 --> 00:29:46,170 Yeah, well, and honestly, and the industry is extremely fragmented. 444 00:29:46,170 --> 00:29:51,452 And if you add up the entire Amlaw 100 revenue, it's $140 billion. 445 00:29:51,452 --> 00:29:55,694 Like that's, that would be like a fortune 100 company. 446 00:29:55,694 --> 00:29:58,689 So that's the entire, that's the top 100. 447 00:29:58,835 --> 00:30:05,598 So it is, it is a relatively small industry, when compared to others, but you know, 448 00:30:05,778 --> 00:30:09,040 There was an AI company that did try to buy one of the information providers. 449 00:30:09,040 --> 00:30:12,450 Harvey made a play for Vlex, I think as part of their series. 450 00:30:12,450 --> 00:30:17,865 They were trying to raise, yeah, five, 600 million to buy Vlex and that didn't happen. 451 00:30:17,865 --> 00:30:18,590 And then... 452 00:30:18,590 --> 00:30:26,445 I know some people who work at like, you know, the open legal data sets that are out there, some of the more free ones, and they've been approached by the likes of open AI for 453 00:30:26,445 --> 00:30:26,995 purchasing. 454 00:30:26,995 --> 00:30:31,006 So they don't have a complete set, but they have a really good set that could be really useful. 455 00:30:31,006 --> 00:30:38,058 You know, the benefit of having the Lexus and Westlaw is you've got all the secondary materials, which are beautiful for training, uh even on, you know, very niche kind of 456 00:30:38,058 --> 00:30:39,378 specific areas of the law. 457 00:30:39,378 --> 00:30:44,140 You've got this really good rich data that's curated by attorneys that have worked in the field for a billion years and whatever. 458 00:30:44,140 --> 00:30:44,831 So. 459 00:30:44,831 --> 00:30:45,942 That is the benefit of those ones. 460 00:30:45,942 --> 00:30:50,213 You Felix went the really interesting opposite route, which is we don't have the secondary materials. 461 00:30:50,213 --> 00:30:53,889 We're just going to make it all a Gentic and we're going to craft you an on-demand treatise. 462 00:30:53,889 --> 00:30:59,763 So I think, you know, unless they can get ahold of Westlar or Lexis, I can't imagine Westlar wants to sell ever. 463 00:30:59,763 --> 00:31:01,624 You know, their data is there. 464 00:31:01,624 --> 00:31:05,387 That is their gold mine that they're sitting on, you know, Thompson writers. 465 00:31:05,387 --> 00:31:08,509 But if they can get ahold of Lexis, that would be some pretty interesting content. 466 00:31:08,509 --> 00:31:11,961 And, know, Harvey now has the API to Lexis. 467 00:31:11,961 --> 00:31:13,252 So I don't know. 468 00:31:13,252 --> 00:31:13,582 see. 469 00:31:13,582 --> 00:31:14,305 We'll see. 470 00:31:14,305 --> 00:31:18,674 Yeah, I've heard that integration is very surface level. 471 00:31:18,674 --> 00:31:19,961 um 472 00:31:19,961 --> 00:31:20,932 struck me as strange. 473 00:31:20,932 --> 00:31:26,681 think a lot of techie people, I think it struck us as strange because it's like, know, what you want is good structured data. 474 00:31:26,681 --> 00:31:33,520 You know, it's like saying, here's this big gulp and here's one those little coffee straws to drink out of it, you know, with the API. 475 00:31:33,520 --> 00:31:37,154 So it's like, ah maybe they'll fix that in the future. 476 00:31:37,154 --> 00:31:37,667 I don't know. 477 00:31:37,667 --> 00:31:42,469 Yeah, it's and you know, this brings up another interesting question and something I talk about a lot. 478 00:31:42,469 --> 00:31:57,285 So I'm of the opinion that in the future, let's say five years from now, once we have a enter the tech enabled legal service delivery era, um, I'm a big believer that law firms 479 00:31:57,285 --> 00:32:02,138 are going to have to leverage their data and in order to differentiate, right? 480 00:32:02,138 --> 00:32:04,329 Buying an off the shelf tool is not going to 481 00:32:04,329 --> 00:32:08,160 to differentiate you because your competitor can go buy the same tool. 482 00:32:08,160 --> 00:32:10,421 But you know what is unique to you? 483 00:32:10,421 --> 00:32:15,812 All of those documents that provided winning outcomes for your clients, right? 484 00:32:15,812 --> 00:32:30,569 All the precedent libraries of model documents that have been battle tested and the way legal works today, it's a very bespoke industry and everybody kind of has their own. 485 00:32:30,705 --> 00:32:39,770 You can ask 10 different lawyers at the same law firm in the same practice for the same document and get different iterations of it. 486 00:32:39,811 --> 00:32:44,453 So, you know, I, it makes me wonder something's got to give there, right? 487 00:32:44,453 --> 00:32:55,391 Like if we're going to tech enable legal service delivery, how, how we can't train models to be reflect one lawyers. 488 00:32:55,897 --> 00:33:00,299 perspective, even though that is what Crosby's doing, by the way, I don't know if you know about Crosby AI. 489 00:33:00,299 --> 00:33:11,494 Um, they, they're, I think it was Sequoia is one of the VCs in their cap table and they, they have a podcast called training data and I listened to that and that's, that is what 490 00:33:11,494 --> 00:33:11,983 they're doing. 491 00:33:11,983 --> 00:33:14,295 They just, but they just have a handful of attorneys. 492 00:33:14,295 --> 00:33:20,737 So they are literally training the attorneys, if I understood it correctly, um, to think like that lawyer. 493 00:33:21,028 --> 00:33:24,611 and that's, that's great and makes a lot of sense as 494 00:33:24,611 --> 00:33:26,332 this transitional phase, right? 495 00:33:26,332 --> 00:33:29,574 Like, I don't think that scales, right? 496 00:33:29,574 --> 00:33:34,256 It's expensive and it's one-off and what happens if they lateral? 497 00:33:34,256 --> 00:33:46,294 they take, you know, it's just, so I don't know, what is your thought on kind of the future and how much law firm data is going to come into play and how firms differentiate 498 00:33:46,294 --> 00:33:47,284 themselves? 499 00:33:47,491 --> 00:33:49,131 I think it's gonna be super interesting. 500 00:33:49,131 --> 00:33:55,411 you know, I think, I went to, once I was, I had a consulting appointment at a law firm. 501 00:33:55,411 --> 00:34:02,871 They wanted to implement something very simple, which is just a rag pipeline for all their pleadings, you know, so that they could generate stuff. 502 00:34:03,031 --> 00:34:05,111 And I went over and I said, okay, where's your data? 503 00:34:05,111 --> 00:34:07,191 And they said, well, some of it's in OneDrive. 504 00:34:07,191 --> 00:34:08,451 And I said, okay. 505 00:34:08,511 --> 00:34:11,771 And some of it's, you know, some of it's in NetDocs. 506 00:34:11,771 --> 00:34:12,771 I said, okay. 507 00:34:12,771 --> 00:34:19,071 And they said, some of it's on this old server that we had, you know, like in a box is that some of it's just paper on a shelf. 508 00:34:19,071 --> 00:34:24,491 But, then when you got to the, got to the planes themselves, all different formats, no good metadata, blah, blah. 509 00:34:24,491 --> 00:34:32,091 And I'm like, it's just going to take a lot to get your, you know, your information into a place where you could actually ingest it into any of these AI systems. 510 00:34:32,171 --> 00:34:35,211 So I think that's obviously step one is just getting 511 00:34:35,211 --> 00:34:43,219 a lot of law firms which have kind of lagged behind just modern data practices, get it all into a data lake or whatever so you can actually make use of this thing is going to be a 512 00:34:43,219 --> 00:34:43,959 big part of it. 513 00:34:43,959 --> 00:34:52,506 But I, you know, I think you're right that there is going to definitely be a period of time where there is going to be a custom tool. 514 00:34:52,506 --> 00:34:56,929 You Joe only do immigration law here is your custom tool that uses your internal things. 515 00:34:56,929 --> 00:35:00,092 and by the way, fine tuning now costs $14 or whatever, you know. 516 00:35:00,092 --> 00:35:09,732 So we can fine tune your little tiny model that just sits on prem on your stuff and really understands your workflow and who Ted is in the context of your firm and all of those 517 00:35:09,732 --> 00:35:10,392 things. 518 00:35:10,392 --> 00:35:12,072 I think there will be a while for that. 519 00:35:12,072 --> 00:35:21,232 think at some point it's probably gonna get eaten up by the big guys where they're gonna, know, VLex is already launching the ability to create your own workflows for your firm. 520 00:35:21,232 --> 00:35:24,892 Harvey has some sort of version of that from what I understand. 521 00:35:24,892 --> 00:35:29,692 So I think they're gonna roll out tools that are probably good for 70 % of the market. 522 00:35:30,054 --> 00:35:40,079 But there may be this place on the margins where you can outperform them substantially by using your own data and leveraging it within an internal ecosystem that is good for AI. 523 00:35:40,079 --> 00:35:43,503 So we were going to talk about a couple of things here. 524 00:35:43,503 --> 00:35:50,960 I want to skip down to legal education because I had a conversation this morning with Ed Walters from Felix. 525 00:35:51,602 --> 00:35:53,143 Yeah, he's a great guy. 526 00:35:53,143 --> 00:35:55,646 He's actually going to be on our 100th episode. 527 00:35:55,646 --> 00:35:57,103 um 528 00:35:57,103 --> 00:35:58,084 He's always a good get. 529 00:35:58,084 --> 00:35:59,785 Always has something interesting to say. 530 00:35:59,805 --> 00:36:01,107 And he's really nice. 531 00:36:01,107 --> 00:36:02,489 So I see him at conferences. 532 00:36:02,489 --> 00:36:05,331 He's just a really, really nice guy too. 533 00:36:06,553 --> 00:36:07,101 Yeah. 534 00:36:07,101 --> 00:36:12,405 we were talking a little bit about lawyer training and you guys are at the forefront of that. 535 00:36:12,566 --> 00:36:27,159 So I'm curious, Haley, like how do you think about how in an academic setting, lawyer training is going to need to evolve from where it is today to where we may be heading 536 00:36:27,159 --> 00:36:29,587 three, four years ago when the 537 00:36:29,587 --> 00:36:37,629 blocking and tackling work is really automated and document generation and we need to elevate lawyers to be more consultants. 538 00:36:37,629 --> 00:36:42,185 Like, how are you guys thinking about it at the law school or are you? 539 00:36:42,185 --> 00:36:45,089 Is it kind of a wait and see situation? 540 00:36:45,089 --> 00:36:45,877 What's that look like? 541 00:36:45,877 --> 00:36:47,618 think it depends on who you talk to. 542 00:36:47,618 --> 00:36:49,820 There's definitely resistance. 543 00:36:49,820 --> 00:36:57,926 I'm a relatively new professor and obviously I like AI and new things and so I think I'm a little bit more forward thinking about all of these things. 544 00:36:57,926 --> 00:37:10,600 But there is a very serious discussion that I think everyone is like engaging in about our students and their future because what first, second, third year attorneys usually do, 545 00:37:10,600 --> 00:37:15,505 Doc review, things like that are all the things that AI is going to be able to do in two seconds, right? 546 00:37:15,505 --> 00:37:22,882 And so what I think is really important for us as educators is to train students to know how to use these tools. 547 00:37:22,882 --> 00:37:27,996 They can come in as the people that can train all the other people in the law firms, right? 548 00:37:27,996 --> 00:37:30,180 So they can bring value that way. 549 00:37:30,180 --> 00:37:36,737 So like in my classes, I don't mind if students use AI to help them with practice problems, generate potential. 550 00:37:36,737 --> 00:37:40,449 multiple choice questions, essay answers, tell them what they did right or wrong. 551 00:37:40,449 --> 00:37:42,101 I encourage them to do that. 552 00:37:42,101 --> 00:37:51,466 And I think, you know, it's pretty similar, you know, even if AI is helping you generate your response to something, like that's what's essentially going on in law firms right now 553 00:37:51,466 --> 00:37:52,487 anyway, right? 554 00:37:52,487 --> 00:37:55,218 There's shell pleadings, motions. 555 00:37:55,218 --> 00:38:04,143 No one ever starts from scratch if you're in a law firm like that, although their data might be in boxes or on old servers, em as Sean recently found out, right? 556 00:38:04,201 --> 00:38:09,193 But I think it's coming no matter what we want to do about it or how we think about it. 557 00:38:09,193 --> 00:38:18,988 And so my general approach is just to encourage students to learn as much as they can, go to the digital initiatives with Sean that he puts on with Kitten all the time, which is 558 00:38:18,988 --> 00:38:20,559 really great for students. 559 00:38:20,559 --> 00:38:27,153 And I'm trying to learn as well so that I can be a resource for them to put them in the best position possible to succeed. 560 00:38:27,153 --> 00:38:29,266 know that Deans are thinking about it for sure. 561 00:38:29,266 --> 00:38:34,671 Because a huge portion of their your US World News ranking is based on your employment data of your students. 562 00:38:34,671 --> 00:38:43,129 And so we're finally, you know, people have been predicting these like catastrophic job layoffs since Chad GVT came around, we're finally now getting studies that are showing 563 00:38:43,129 --> 00:38:44,110 that this is really happening. 564 00:38:44,110 --> 00:38:54,499 I think Accenture just laid off 11,000 people there was that big study that came out of I think it was Stanford that had 62 million back end data points from the payroll system. 565 00:38:54,499 --> 00:38:59,899 that showed a 13.2 % decline in hiring among younger people. 566 00:39:00,159 --> 00:39:10,319 I, know, with the kind of traditional law school or law firm pyramid where this bottom layer of people, this document review discovery, simple, you know, kind of simple drafting 567 00:39:10,319 --> 00:39:14,639 research, like AI does all of those things very well and collapses it down. 568 00:39:14,639 --> 00:39:19,331 you know, I think when we think of law firms going forward, it's either going to be kind of a cylinder. 569 00:39:19,331 --> 00:39:25,931 or even a diamond shape is what other people have predicted, which I think is pretty interesting, but it raises a lot of problems for law schools. 570 00:39:26,191 --> 00:39:35,351 the way that I've thought about filling in what are those people gonna go do if they're not going to work at Kirkland, Junior's, Jackson Walker, McAfee Taft, one of the big firms 571 00:39:35,351 --> 00:39:37,731 down here, go and don't leave me where your mom works. 572 00:39:38,031 --> 00:39:47,171 I think non-traditional careers, you I kind of think we're headed for probably a 2008, 2009 type scenario where they're just not hiring first year associates as much. 573 00:39:47,202 --> 00:39:54,178 And so my impulse is to say to my students, hey, especially the ones my AI in the practice of law class, you know, it'd be really cool career legal ops. 574 00:39:54,619 --> 00:40:02,766 And that's not something that law schools traditionally use legal leverage their career services and send people to clock and talk to people you don't like you who are really 575 00:40:02,766 --> 00:40:03,537 into these areas. 576 00:40:03,537 --> 00:40:09,313 And I'm like, that's a great lifestyle, you know, you don't have to go work at a firm, you can still make great money. 577 00:40:09,313 --> 00:40:12,260 And if you're a techie person, like you know me or 578 00:40:12,260 --> 00:40:15,374 you know, like when I was graduating from law school, wouldn't want to go do document review at big firm. 579 00:40:15,374 --> 00:40:20,931 I would have loved to have done something like that instead, but I didn't know that was an option and it wasn't the traditional path for law students. 580 00:40:20,931 --> 00:40:28,592 So I think we need to be a little more creative and hopefully we're setting up the students that come through here so that they could go right into those careers and be 581 00:40:28,592 --> 00:40:29,713 really effective. 582 00:40:29,779 --> 00:40:30,119 Yeah. 583 00:40:30,119 --> 00:40:37,133 So, you I've been an entrepreneur for 32 years and during that time I have consumed a lot of legal services. 584 00:40:37,133 --> 00:40:47,450 And one of the big challenges I've talked about it once or twice on the show before is Lawyers are really good at identifying risk. 585 00:40:47,450 --> 00:40:50,613 as a business person, that's only half the equation. 586 00:40:50,613 --> 00:40:56,322 is a, almost every decision is a risk reward or a cost benefit. 587 00:40:56,322 --> 00:40:57,733 decision, right? 588 00:40:57,733 --> 00:40:59,114 You're balancing those scales. 589 00:40:59,114 --> 00:41:08,250 And lawyers historically have not taken the time to get to know my business because I'd have to pay them to do that, right? 590 00:41:08,250 --> 00:41:09,701 And that would be really expensive. 591 00:41:09,701 --> 00:41:19,128 I understand why they haven't, but you know, like that's always been a challenge from where I sit because I've had lawyers, you know, basically advise me, don't sign this 592 00:41:19,128 --> 00:41:19,888 agreement. 593 00:41:19,888 --> 00:41:21,959 And I'm like, you can't say that. 594 00:41:22,016 --> 00:41:25,406 What you can tell me is you can inform me on the risk side of the equation. 595 00:41:25,406 --> 00:41:31,008 I, as the business person, need to balance the business opportunity, i.e. 596 00:41:31,008 --> 00:41:33,709 the reward side of the equation. 597 00:41:33,709 --> 00:41:35,769 What you're bringing to me is very important. 598 00:41:35,769 --> 00:41:37,770 It helps me balance the scales. 599 00:41:37,810 --> 00:41:46,905 So what I would love to see as a consumer of legal services is lawyers get more in an advisory capacity and bring the human element. 600 00:41:46,905 --> 00:41:48,247 I'll give you an example. 601 00:41:48,247 --> 00:41:59,690 So we hired someone away from a, they're not really a competitor, but a company who had a non-compete and we had to craft his job description in a way where we wouldn't run a foul 602 00:41:59,690 --> 00:42:00,931 of that non-compete. 603 00:42:00,931 --> 00:42:06,056 And we hired a L &E attorney and he gave great advice and it wasn't really legal. 604 00:42:06,056 --> 00:42:09,662 He's like, look, you know, the reality is, you know, for the first, 605 00:42:09,662 --> 00:42:13,462 X number of months, there's probably going to be a lot of scrutiny around this. 606 00:42:13,462 --> 00:42:22,162 And then over time that will probably, you know, there, there are a bill, a judge wouldn't look kindly if six months down the road, they, they threw a flag, right? 607 00:42:22,162 --> 00:42:23,842 So he was giving me human advice. 608 00:42:23,842 --> 00:42:27,002 Like, you know, this is how the company is going to look at this. 609 00:42:27,002 --> 00:42:31,882 This is how the judge would look at an action that came six months later. 610 00:42:31,882 --> 00:42:37,082 Like that sort of stuff is going to be really hard to get out of chat GPT. 611 00:42:37,082 --> 00:42:38,382 It requires 612 00:42:38,458 --> 00:42:39,730 Experience is it possible? 613 00:42:39,730 --> 00:42:42,724 I guess I right 614 00:42:42,836 --> 00:42:48,489 you have to fill up the current 1 million token context window in order to have all that context if then, you know, even. 615 00:42:48,489 --> 00:43:01,557 Yeah, and even then, this guy was a seasoned vet and had spent 30 years on plaintiff side cases. 616 00:43:01,557 --> 00:43:10,102 And again, how you load that into an AI model and get solid judgment back, I think, presents challenges. 617 00:43:10,102 --> 00:43:11,825 uh 618 00:43:11,825 --> 00:43:16,207 when they survey, they survey, you know, our customers from law firms, they all hate us. 619 00:43:16,207 --> 00:43:17,807 They hate to see us. 620 00:43:17,807 --> 00:43:18,888 They hate to talk to us. 621 00:43:18,888 --> 00:43:26,162 They hate to pick up the phone in advance, which is, know, in many instances would remediate a lot of risk if they could just, hey, Sean, what about this? 622 00:43:26,162 --> 00:43:27,332 They hate doing all that stuff. 623 00:43:27,332 --> 00:43:35,445 So, I mean, that's why when you hear it, like a lot of the access to justice, people talk about this technology, the more optimistic ones, they go look, 78 % of the market doesn't 624 00:43:35,445 --> 00:43:38,956 have access to a civil attorney, an attorney in a civil context. 625 00:43:38,956 --> 00:43:40,633 If this drops the cost. 626 00:43:40,633 --> 00:43:50,178 cost and it allows a whole nother layer of people to have better legal outcomes or God forbid uh makes us have just better interactions with our clients so that they want to 627 00:43:50,178 --> 00:43:50,969 come talk to us. 628 00:43:50,969 --> 00:43:55,507 They enjoy picking up the phone because they feel more reassured about their business or whatever. 629 00:43:55,507 --> 00:43:57,192 You know, there is a lot of promise with this stuff. 630 00:43:57,192 --> 00:44:02,525 I try not to be completely starry eyed and rose colored glasses about it, but I'm just an optimistic person. 631 00:44:02,525 --> 00:44:08,749 I think there are a lot of wins here or just our interactions with our clients and better legal outcomes for regular people. 632 00:44:08,749 --> 00:44:09,749 think too. 633 00:44:10,078 --> 00:44:10,258 Yeah. 634 00:44:10,258 --> 00:44:15,169 And you have to ask yourself, why did, why do they hate answering the phone or calling the lawyers? 635 00:44:15,169 --> 00:44:17,631 You know, and, I actually don't hate it. 636 00:44:17,631 --> 00:44:19,552 Uh, what I do hate. 637 00:44:19,552 --> 00:44:27,475 Yeah, no, I mean, you know, we've got, we've got, you know, probably the number one startup law firm in the country. 638 00:44:27,475 --> 00:44:29,216 You can figure out who that is. 639 00:44:29,216 --> 00:44:35,179 And, it is, it is an education process every time I get to, talk to my counterpart over there. 640 00:44:35,179 --> 00:44:36,683 And I learned so much. 641 00:44:36,683 --> 00:44:37,393 And he's great. 642 00:44:37,393 --> 00:44:40,966 And he does think about things in risk reward. 643 00:44:40,966 --> 00:44:43,947 And that's why he's my attorney. 644 00:44:43,947 --> 00:44:46,209 And I enjoy picking up the phone. 645 00:44:46,209 --> 00:44:48,229 I don't enjoy getting the bills. 646 00:44:49,370 --> 00:44:54,174 That part's not fun, but I do feel like I get a lot of value from it. 647 00:44:54,174 --> 00:44:56,415 I've not always felt that way. 648 00:44:56,436 --> 00:45:00,378 And I don't know if this is a function, cause I've been in a lot of different businesses. 649 00:45:00,378 --> 00:45:02,779 My wife and I own five gyms here in St. 650 00:45:02,779 --> 00:45:03,528 Louis. 651 00:45:03,528 --> 00:45:06,519 fair amount of legal work that went into building that those out. 652 00:45:06,519 --> 00:45:07,859 I owned a collection agency. 653 00:45:07,859 --> 00:45:09,898 used to sue people all the time. 654 00:45:09,898 --> 00:45:11,461 A lot of legal work there. 655 00:45:11,461 --> 00:45:18,624 I've not always enjoyed it and I'm not sure if it's because these guys are super niche and they focus on venture backed startups like we are. 656 00:45:18,624 --> 00:45:25,206 But they really know our business and I learned something every time I talked to them and I enjoy it. 657 00:45:25,692 --> 00:45:28,436 Yeah, and there's a whole, you know, so I think there's two parts to that. 658 00:45:28,436 --> 00:45:36,043 Number one, some lawyers when you talk to them are sort of like when I talk to my IT department, excuse me, and I go, hey, I need to download this really interesting open 659 00:45:36,043 --> 00:45:43,788 source, completely unvetted software, because I need to keep my skills, you know, hot and, you know, up to the bleeding edge. 660 00:45:43,788 --> 00:45:45,229 And I want to use a cool new thing. 661 00:45:45,229 --> 00:45:46,460 And they just go, absolutely not. 662 00:45:46,460 --> 00:45:48,641 have no risk, no risk tolerance whatsoever. 663 00:45:48,641 --> 00:45:53,366 Because what's the incentive to them if I can use a new AI thing? 664 00:45:53,366 --> 00:45:54,546 Zero. 665 00:45:54,746 --> 00:45:58,988 What's the risk if I huge, you know, and so they just dial up risk. 666 00:45:58,988 --> 00:46:07,350 And I'm like, well, I have other interests, like remaining relevant or, you know, with library resources that happens very frequently with a thing we will not put behind 10 667 00:46:07,350 --> 00:46:09,431 different walls before the user can get to it. 668 00:46:09,431 --> 00:46:13,732 And a librarian goes, nobody's going to use it if it's a hassle to get in and use the thing. 669 00:46:13,732 --> 00:46:19,623 So it's just like, like you said, I think just the incentives are just pointed in the wrong direction for some lawyers and law firms. 670 00:46:19,623 --> 00:46:20,907 Yeah. 671 00:46:20,907 --> 00:46:23,667 Do you have anything to add to AI stuff I was talking about? 672 00:46:23,667 --> 00:46:24,987 Sorry, I got a pun intended. 673 00:46:25,107 --> 00:46:33,007 it just kind of reminded me when you're talking about the experience that the labor and employment attorney had that you found really valuable. 674 00:46:33,007 --> 00:46:43,227 I was a law clerk for a few judges and tons of firms and clients found it very valuable because I worked in judges chambers for years at a time. 675 00:46:43,227 --> 00:46:49,379 I understood how the judges I worked for thought, how their chambers operated, all this information that 676 00:46:49,385 --> 00:46:50,936 AI can't get its hands on, right? 677 00:46:50,936 --> 00:47:01,722 And so that was value I could bring to clients that is aside from document review and, you know, it's advice that can actually be beneficial that you're not going to get from AI or 678 00:47:01,722 --> 00:47:02,933 even every attorney. 679 00:47:02,933 --> 00:47:08,887 And so there are ways I think that attorneys just in general can utilize things that AI just can't do. 680 00:47:08,887 --> 00:47:11,148 Now, to me, though, it brings up this issue. 681 00:47:11,148 --> 00:47:13,799 You know, you're talking about the 30 years of experience. 682 00:47:13,799 --> 00:47:16,681 Well, if attorneys can't even get in the door. 683 00:47:16,681 --> 00:47:25,926 you know, after law school, because AI has taken over all of their tasks, we're going to end up with a huge shortage of attorneys at some point that have have any experience and 684 00:47:25,926 --> 00:47:35,261 then that you know, value might be lost if there's not a way to kind of have attorneys recalibrate like what they bring to the table and then get that experience as years go on. 685 00:47:35,261 --> 00:47:42,585 Yeah, I some people have, you know, when I go to other conferences, like Dan Lena was from Northwestern was talking about this where he was saying, you know, essentially, what we're 686 00:47:42,585 --> 00:47:45,256 going to see is just attorneys just highly specialized. 687 00:47:45,256 --> 00:47:53,221 So I'm an expert in e-discovery, in transactional, in California, for these real estate, you know, whatever, just really, really narrowly specialized. 688 00:47:53,221 --> 00:47:55,912 And so maybe that would be a situation like yours. 689 00:47:55,912 --> 00:47:59,514 You now got this attorney who he has dialed into your exact sector. 690 00:47:59,514 --> 00:48:03,617 You know that he's competent, you know, he has the right level of risk tolerance for you. 691 00:48:03,617 --> 00:48:07,279 And he gives you this white glove service that's very customized to Ted. 692 00:48:07,279 --> 00:48:13,054 And so maybe it'll just be that kind of times everything in law as what he, you know, they've hypothesized. 693 00:48:13,054 --> 00:48:14,950 They say the riches are in the niches. 694 00:48:14,950 --> 00:48:16,273 So, um, 695 00:48:16,273 --> 00:48:16,864 what I my students. 696 00:48:16,864 --> 00:48:18,296 I said, you know, this is a great time. 697 00:48:18,296 --> 00:48:24,403 Like you want lost services Chipotle burrito, you know, you want extra guac and so Fritas instead of carnitas. 698 00:48:24,403 --> 00:48:26,115 Let me get you exactly what you want, sir. 699 00:48:26,115 --> 00:48:28,068 That expensive burrito. 700 00:48:28,068 --> 00:48:29,930 Deliver it to your table for you. 701 00:48:29,930 --> 00:48:31,571 And here's the drink that you enjoy. 702 00:48:31,571 --> 00:48:33,665 Well, this has been a great conversation. 703 00:48:33,665 --> 00:48:38,031 really appreciate you guys coming and spending a little bit of time with me on the show. 704 00:48:38,031 --> 00:48:39,147 um 705 00:48:39,147 --> 00:48:42,850 got to kick the tires on our new podcast studio that Jim put together for us. 706 00:48:42,850 --> 00:48:44,211 Yeah, it's been great. 707 00:48:44,211 --> 00:48:44,611 Thank you. 708 00:48:44,611 --> 00:48:46,323 Awesome. 709 00:48:46,323 --> 00:48:47,183 I'm really impressed with it. 710 00:48:47,183 --> 00:48:49,543 I'm gonna do all my big presentations in here from now on. 711 00:48:49,543 --> 00:48:50,796 Yeah, you should. 712 00:48:50,796 --> 00:48:57,030 How do folks find out more about the work that you're doing there at the law school and you individually? 713 00:48:57,030 --> 00:48:58,863 What's the best way for them to do that? 714 00:48:59,111 --> 00:49:03,126 Probably for me, LinkedIn is kind of where I consolidate all my professional stuff. 715 00:49:03,126 --> 00:49:06,059 And it's, I'll have links to like my SSRN articles. 716 00:49:06,059 --> 00:49:09,102 And when I presented a thing, if I'm going to Ulta and presenting or whatever. 717 00:49:09,102 --> 00:49:09,793 Yeah. 718 00:49:09,793 --> 00:49:12,716 And then we're both, we both have pages on OU Laws website. 719 00:49:12,716 --> 00:49:17,470 So you can just search for our name, all of our papers and work will kind of be linked there as well. 720 00:49:17,470 --> 00:49:18,230 Okay. 721 00:49:18,230 --> 00:49:18,820 Well, awesome. 722 00:49:18,820 --> 00:49:22,332 We'll include those links in the show notes so folks can access them. 723 00:49:22,332 --> 00:49:28,893 And thank you so much for being on the show and hopefully we get to meet in person at a future conference. 724 00:49:30,974 --> 00:49:32,085 You as well. 00:00:02,083 Sean Haley, how are you today? 2 00:00:02,083 --> 00:00:03,453 Good to see you. 3 00:00:03,453 --> 00:00:05,655 I really appreciate you guys joining me, man. 4 00:00:05,655 --> 00:00:12,260 I enjoyed your Michael Scott paper quite a bit and we're going to talk about it today. 5 00:00:12,260 --> 00:00:14,602 But before we do, let's get you guys introduced. 6 00:00:14,602 --> 00:00:20,985 So you're both academics at University of, is it Oklahoma? 7 00:00:21,766 --> 00:00:22,786 University of Oklahoma. 8 00:00:22,786 --> 00:00:24,027 In my notes, have Arkansas. 9 00:00:24,027 --> 00:00:25,988 That's wrong. 10 00:00:26,068 --> 00:00:29,450 Why don't, Hailey, we start with you, introduce yourself and then we'll... 11 00:00:29,450 --> 00:00:30,811 Hand it over to Sean. 12 00:00:31,017 --> 00:00:31,387 Sure. 13 00:00:31,387 --> 00:00:33,128 So I'm Haley Stilwell. 14 00:00:33,128 --> 00:00:37,950 I am an associate professor of law at the University of Oklahoma College of Law. 15 00:00:37,950 --> 00:00:40,211 I teach evidence and criminal law. 16 00:00:40,211 --> 00:00:45,813 I write in those fields as well, but I also have a particular interest in exploring AI in my scholarship. 17 00:00:45,813 --> 00:00:49,015 And that's kind of where Sean and I's relationship began. 18 00:00:49,015 --> 00:00:49,255 Yeah. 19 00:00:49,255 --> 00:00:50,025 And I'm Sean Harrington. 20 00:00:50,025 --> 00:00:57,178 I'm the director of technology innovation here at OU Law, which is kind of a nebulous title, you know, at different institutions that means different things. 21 00:00:57,178 --> 00:00:59,711 Here what it means is I teach AI in the practice of law. 22 00:00:59,711 --> 00:01:06,231 And I also run what's called their digital initiative, which gets tech training to all the students, essentially. 23 00:01:06,326 --> 00:01:09,869 And I met someone from your law school. 24 00:01:09,869 --> 00:01:10,440 Yeah. 25 00:01:10,440 --> 00:01:11,930 Was it Kenton? 26 00:01:12,792 --> 00:01:13,572 Yeah. 27 00:01:13,572 --> 00:01:19,107 Um, we were, he happened to be attending a session that I was helping, helping facilitate and sat at my table. 28 00:01:19,107 --> 00:01:23,601 So it was pretty cool when I saw your paper and was like, yeah. 29 00:01:23,601 --> 00:01:26,404 And connected the dots and, here we are today. 30 00:01:27,626 --> 00:01:28,459 So 31 00:01:28,459 --> 00:01:29,190 to your podcast. 32 00:01:29,190 --> 00:01:31,432 And so he was like, my gosh, you got to go on that. 33 00:01:31,432 --> 00:01:33,373 I was like, Okay, yeah, I'm in. 34 00:01:33,434 --> 00:01:34,014 Good. 35 00:01:34,014 --> 00:01:34,374 Good. 36 00:01:34,374 --> 00:01:34,594 Yeah. 37 00:01:34,594 --> 00:01:44,574 We started off, you know, we started off primarily in the knowledge management and innovation realm at large law firms as our audience. 38 00:01:44,574 --> 00:01:46,174 And it's really evolved from that. 39 00:01:46,174 --> 00:01:49,954 I mean, we have, I've got people from the marketing function. 40 00:01:49,954 --> 00:01:52,985 I've got attorneys, I've got managing partners, I have academics. 41 00:01:52,985 --> 00:01:59,721 So, you know, the innovation topic is very timely in the legal world. 42 00:01:59,721 --> 00:02:00,477 right. 43 00:02:00,477 --> 00:02:03,718 Like the timing was, the timing was very fortunate for us. 44 00:02:03,718 --> 00:02:07,940 So I started right around the time chat GPT was released. 45 00:02:07,940 --> 00:02:08,740 Yeah. 46 00:02:08,740 --> 00:02:10,260 So, I mean, just pure luck. 47 00:02:10,260 --> 00:02:13,181 Like I didn't see that and go, we need a podcast. 48 00:02:13,181 --> 00:02:18,754 was just, we had been selling into not the world of knowledge management for many, many years. 49 00:02:18,754 --> 00:02:26,164 And then we saw this interesting dynamic where all saw these innovation roles start to pop up and 50 00:02:26,164 --> 00:02:26,958 Right. 51 00:02:27,244 --> 00:02:31,688 started having conversations with, what's the difference between knowledge management and innovation? 52 00:02:31,688 --> 00:02:35,612 And those conversations were very enlightening. 53 00:02:35,612 --> 00:02:37,372 And I thought, you know what? 54 00:02:37,473 --> 00:02:40,115 I should record these and make a podcast. 55 00:02:40,115 --> 00:02:54,757 So I recorded three episodes, didn't tell anyone about it, posted them, you know, just on the usual platforms, Spotify, Apple podcasts, like three weeks before, Ilta Ilta con where 56 00:02:54,757 --> 00:02:55,338 we met. 57 00:02:55,338 --> 00:03:00,578 Kenton and I had like 10 people come up to me in the booth and go, Hey, I your podcast. 58 00:03:00,578 --> 00:03:01,018 It's great. 59 00:03:01,018 --> 00:03:03,398 I was like, how the hell did you find it? 60 00:03:03,398 --> 00:03:12,958 And I guess people just, I don't know if they keep Google alerts on certain keywords, but, um, that was almost a hundred episodes. 61 00:03:12,958 --> 00:03:15,458 We're going to have our hundredth episode here in a few weeks. 62 00:03:15,458 --> 00:03:18,115 And, uh, yeah, it's been, it's been a lot of fun. 63 00:03:18,115 --> 00:03:18,495 awesome. 64 00:03:18,495 --> 00:03:27,835 mean, the thing is you go to something like Ilta and you just have, you know, 100 of those types of conversations and you run into Damien real or whatever, you know, and you have a 65 00:03:27,835 --> 00:03:32,075 conversation like, man, I wish I had somebody to save this, you know, other people could repurpose this. 66 00:03:32,075 --> 00:03:37,175 I want to have so I have this little, uh, have you seen the limitless AI pendant? 67 00:03:37,334 --> 00:03:42,692 No, but I've used, I've actually used the, think it's called plod, which I think it's similar. 68 00:03:43,437 --> 00:03:43,849 Yeah. 69 00:03:43,849 --> 00:03:46,802 And so he just sticks right here and then you have that knowledge base for forever. 70 00:03:46,802 --> 00:03:50,816 And I'm like, man, anytime I want to go to one of those conferences, I'm to be wearing that thing. 71 00:03:50,816 --> 00:03:59,666 Yeah, you know, the whole privacy thing around that, like, do you have to tell everybody like, hey, I'm recording that, you know, I was, yeah. 72 00:03:59,666 --> 00:04:05,546 of don't have like the social etiquette for it because obviously, you know, people knew that you were recording them, they would speak to you very differently. 73 00:04:05,546 --> 00:04:13,146 But you want the other conversation because the utility of retaining that information and having it in a data set that you could manipulate is like so high. 74 00:04:13,146 --> 00:04:20,886 It's just like, we're gonna have to figure out something like a little green, you know, light on my lapel that says Sean's always recording or something. 75 00:04:20,886 --> 00:04:22,406 think I turn it off. 76 00:04:22,406 --> 00:04:25,146 Nobody's want Nobody's gonna want to talk to me anyways. 77 00:04:25,146 --> 00:04:26,582 That's only gonna make it worse. 78 00:04:26,582 --> 00:04:29,211 in the legal crowd, right? 79 00:04:29,211 --> 00:04:31,918 I mean, they're very privacy sensitive. 80 00:04:31,918 --> 00:04:37,265 I had a faculty member here that I was just explaining what the thing did. 81 00:04:37,265 --> 00:04:38,096 It was in its dock. 82 00:04:38,096 --> 00:04:39,149 I was like, it's not recording. 83 00:04:39,149 --> 00:04:40,551 I was explaining what it was. 84 00:04:40,551 --> 00:04:44,115 And she made up an excuse to get up and leave my office almost immediately. 85 00:04:44,383 --> 00:04:45,003 That's funny. 86 00:04:45,003 --> 00:04:45,423 Yeah. 87 00:04:45,423 --> 00:04:54,763 I mean, we, um, we use a platform called fireflies for our meetings and 100 % of our, of our client base is law, our law firms, large law firms. 88 00:04:55,183 --> 00:04:59,983 And, um, surprisingly we don't get a lot of pushback on that. 89 00:05:00,063 --> 00:05:09,803 And, you know, but I do think that people manage their candor when they see and, and rightly so I get it. 90 00:05:09,803 --> 00:05:11,319 Um, but 91 00:05:11,355 --> 00:05:14,071 Anyway, let's talk about your paper. 92 00:05:14,071 --> 00:05:15,706 So it really caught my eye. 93 00:05:15,706 --> 00:05:19,112 It was, and the reason is because I'm a big office fan. 94 00:05:19,484 --> 00:05:20,464 Love it. 95 00:05:20,464 --> 00:05:22,324 So the title was all Haley. 96 00:05:22,324 --> 00:05:24,264 She came up with that. 97 00:05:25,924 --> 00:05:27,264 Oh, yeah. 98 00:05:27,264 --> 00:05:27,535 Yeah. 99 00:05:27,535 --> 00:05:29,077 they're great. 100 00:05:29,077 --> 00:05:35,343 So if I remember correctly, the title was Michael Scott is not a juror. 101 00:05:36,244 --> 00:05:36,766 Okay. 102 00:05:36,766 --> 00:05:40,654 And, um, give us a little bit of background on the paper. 103 00:05:40,654 --> 00:05:45,203 Uh, I don't know whoever wants to start and then we can jump into that a little bit. 104 00:05:45,321 --> 00:05:46,491 Yeah, you started, it your idea. 105 00:05:46,491 --> 00:05:51,744 Yeah, so the origin story kind of comes from my just academic research in general. 106 00:05:51,744 --> 00:06:01,828 There's this standard that's often utilized em in courts called the reasonable juror standard where courts are basically trying to determine what a perfectly behaving 107 00:06:01,828 --> 00:06:07,210 impartial juror that follows all the court's directions, how they would think and understand evidence. 108 00:06:07,210 --> 00:06:14,387 And it comes into play a lot when em courts are assessing whether the admission or exclusion of evidence was 109 00:06:14,387 --> 00:06:16,939 in violation of a rule or the constitution. 110 00:06:16,939 --> 00:06:26,874 And so I was bothered by some of the court's reasonable jury determinations because it just didn't track at all with how I thought real people would think and understand about 111 00:06:26,874 --> 00:06:27,985 certain evidence. 112 00:06:27,985 --> 00:06:38,390 So that prompted me to want to create a mock jury study to compare what the court has determined reasonable jurors would think and compare it to what real people would think 113 00:06:38,390 --> 00:06:43,301 and kind of highlight where the court has got it wrong and in turn results in 114 00:06:43,301 --> 00:06:45,922 probable constitutional violations. 115 00:06:45,922 --> 00:06:57,508 And so with that project, what I wanted to then do is find a way to empirically ground the reasonable juror where possible to bring the reasonable juror and real jurors closer 116 00:06:57,508 --> 00:07:00,389 together when courts were making those determinations. 117 00:07:00,389 --> 00:07:06,682 And the easiest way to do that is with mock jury data like we did in this study, but it takes way too much time. 118 00:07:06,682 --> 00:07:08,867 It's way too expensive to do. 119 00:07:08,867 --> 00:07:20,567 And so when uh Sean and I were kind of talking about it, we got the idea, what if we could create a mock jury bot that could output mock jury information instantaneously to help 120 00:07:20,567 --> 00:07:27,852 inform courts when they're making these determinations to protect criminal defendants and all people that are appealing their cases. 121 00:07:27,852 --> 00:07:38,704 And so we thought it would be pretty easy, just use existing platforms to feed them, you know, what this, the demographic profile of a juror and spit out information and 122 00:07:38,704 --> 00:07:41,156 Hopefully it would track what real jurors would do. 123 00:07:41,156 --> 00:07:43,587 And that is just not even close to what happened. 124 00:07:43,587 --> 00:07:44,958 And that's where this paper came. 125 00:07:44,958 --> 00:07:45,308 Yeah. 126 00:07:45,308 --> 00:07:50,642 And like the other thing is, you know, we looked up a bunch of studies that were trying to do this and it's like sample size, 112 people. 127 00:07:50,642 --> 00:07:56,566 So there just wasn't a lot of research and uh actual surveys of humans done on this stuff. 128 00:07:56,566 --> 00:08:00,669 So, you know, you quickly realize that it's really expensive to survey people. 129 00:08:00,669 --> 00:08:05,773 And so there just wasn't like a good academic research in this area. 130 00:08:05,773 --> 00:08:06,649 So we're like, well, 131 00:08:06,649 --> 00:08:10,635 That's even another reason then to integrate AI into this thing. 132 00:08:10,801 --> 00:08:11,111 Yeah. 133 00:08:11,111 --> 00:08:20,087 And you know, I, we talked to, we get a lot of attorneys on the show, some practicing, some not the ones who practice. 134 00:08:20,087 --> 00:08:28,072 I always like to ask, you know, what are the valuable use cases with AI trial prep is near the top of the list. 135 00:08:28,072 --> 00:08:31,034 And the data doesn't show this. 136 00:08:31,034 --> 00:08:33,396 really it's kind of drafting due diligence. 137 00:08:33,396 --> 00:08:36,923 you see as kind of the most common, document review. 138 00:08:36,923 --> 00:08:38,559 Yeah, document review would seem like fun. 139 00:08:38,559 --> 00:08:41,859 Yeah, it seemed to be the most common use cases cited. 140 00:08:41,879 --> 00:08:49,650 But when I hear about in terms of value, we had a guy on the podcast, name's Stephen Embry. 141 00:08:49,650 --> 00:08:50,750 You familiar with? 142 00:08:50,750 --> 00:08:51,650 Yeah. 143 00:08:51,810 --> 00:08:58,650 So Steve is a former, he was a 30 year attorney at I believe Finnegan, which is a, they're a client of ours. 144 00:08:58,650 --> 00:09:00,930 They're an intellectual property firm. 145 00:09:01,270 --> 00:09:04,914 And he was telling me how 146 00:09:05,111 --> 00:09:19,716 either him or his colleagues would use it for trial preparation in basically having AI challenge the evidence in some way, like ask all of the questions that opposing counsel 147 00:09:19,716 --> 00:09:22,977 might present in a courtroom. 148 00:09:22,977 --> 00:09:30,249 he said, you know, he talked about how, how useful that was, which I found very intriguing and makes perfect sense. 149 00:09:30,249 --> 00:09:32,760 Like I'm a daily user of AI. 150 00:09:32,860 --> 00:09:33,877 super familiar with. 151 00:09:33,877 --> 00:09:34,647 drive here for you. 152 00:09:34,647 --> 00:09:36,118 said, pretend. 153 00:09:36,118 --> 00:09:37,298 I said, here's our paper. 154 00:09:37,298 --> 00:09:40,469 Pretend that you're Ted and ask me some stuff. 155 00:09:40,469 --> 00:09:45,541 I mean, it really lowers the temperature coming into the environment because you're like, okay, I'm feeling pretty confident. 156 00:09:45,541 --> 00:09:49,172 You know, I've kind of been challenged on things that where I wasn't super confident. 157 00:09:49,172 --> 00:09:50,132 I looked a few things up. 158 00:09:50,132 --> 00:09:51,983 So it's like, you feel ready to roll. 159 00:09:51,983 --> 00:09:55,230 So I could definitely see that having a lot of value for attorneys. 160 00:09:55,230 --> 00:09:58,441 Yeah, I mean, it's almost like a practice run that you get. 161 00:09:58,441 --> 00:10:03,943 So when I heard the study, the name was catchy, so that drew me in initially. 162 00:10:03,943 --> 00:10:11,946 But then when I read what you guys were actually doing, I was like, wow, this seems like a perfect use case for AI. 163 00:10:11,946 --> 00:10:18,468 And it turns out it wasn't that perfect using the frontier models. 164 00:10:18,468 --> 00:10:21,699 Tell us about what your findings were initially. 165 00:10:22,083 --> 00:10:28,305 Well, the first thing we tried to do was just let the frontier models try to create a jury. 166 00:10:28,305 --> 00:10:34,016 So we said, create for us a jury pool that is similar to what a federal jury pool would be. 167 00:10:34,016 --> 00:10:36,438 And that's where Michael Scott emerged. 168 00:10:36,438 --> 00:10:39,100 It was really hilarious. 169 00:10:39,100 --> 00:10:41,411 They would output the demographics of the jurors. 170 00:10:41,411 --> 00:10:48,433 So it was a white man in his mid-40s who is the manager of a mid-sized paper firm in Scranton, Pennsylvania. 171 00:10:48,433 --> 00:10:52,145 which you and I would obviously know is Michael Scott. 172 00:10:52,145 --> 00:10:57,867 Michael Scott is not a real person, let alone a real juror in the federal jury pool, right? 173 00:10:57,867 --> 00:11:06,069 We also had a lot of other interesting combinations of, there was a 90 year old woman who was a part-time botanist, part-time DJ. 174 00:11:06,069 --> 00:11:07,437 I love that one. 175 00:11:07,437 --> 00:11:10,311 We had an abolitionist podcaster. 176 00:11:10,311 --> 00:11:15,824 So it seemed like when these platforms were left to their own devices, they were generating 177 00:11:15,824 --> 00:11:25,106 jurors that were more for show kind of eye catching types of backgrounds that really didn't reflect what we needed for our purposes. 178 00:11:25,106 --> 00:11:29,717 What real people on a jury would actually look like demographically. 179 00:11:29,717 --> 00:11:30,478 Yeah. 180 00:11:30,478 --> 00:11:39,560 And you can tell that, know, there a kid in, you know, Washington is using them right now to study who's 12 years old and maybe using it for creative writing. 181 00:11:39,560 --> 00:11:44,079 So, you know, there's a big range of why people are using these tools and they have the dial. 182 00:11:44,079 --> 00:11:51,150 on certain types of representation, which could be very useful, obviously, in a creative writing context, but in ours, that was, you know, catastrophic, because it was wasn't 183 00:11:51,150 --> 00:11:52,451 representing reality. 184 00:11:52,896 --> 00:11:55,888 Well, and look, I love Michael Scott. 185 00:11:55,888 --> 00:11:59,200 He'd be the last person I'd ever choose to be on a jury. 186 00:11:59,200 --> 00:12:01,301 Not, not known for good judgment. 187 00:12:01,301 --> 00:12:04,543 Um, well, and that's interesting. 188 00:12:04,543 --> 00:12:19,501 So I have had a lot of debate myself and I've seen a lot of debate in cyberspace, LinkedIn and Reddit about the frontier models coming and essentially crushing the legal specific 189 00:12:19,501 --> 00:12:27,750 tools just based on just how quickly the trajectory that they're on in terms of capabilities and 190 00:12:27,750 --> 00:12:38,941 I think your findings will first tell us what you found as potential root cause as to why this experiment was not successful with the Frontier models. 191 00:12:38,941 --> 00:12:39,461 Yeah. 192 00:12:39,461 --> 00:12:45,143 So first of all, you know, they have lengthy system prompts, which people, you know, leak on GitHub all the time. 193 00:12:45,143 --> 00:12:50,608 And you can see kind of why they're training these models to do things like be a good study tool for a student. 194 00:12:50,608 --> 00:12:57,852 And then we also just saw that just rampant bias on certain topics that just was completely at odds with reality. 195 00:12:57,852 --> 00:13:07,117 And so when we were looking, you when we actually finally ran our actual mock juror surveys, what we discovered is that people are like fairly moderate on average. 196 00:13:07,117 --> 00:13:10,320 And these would be kind of extreme versions of people. 197 00:13:10,320 --> 00:13:17,727 And again, you know, I think it's just because they're, you know, kind of meant to do everything instead of replicate juror juror scenarios. 198 00:13:17,727 --> 00:13:18,787 What else? 199 00:13:19,108 --> 00:13:24,132 Well, I think to covering kind of the second part of our experiment is important to explain that as well. 200 00:13:24,132 --> 00:13:33,380 After we left the platforms to their own devices, that's where the mock jury data came in to where, OK, we were going to tell the platforms now. 201 00:13:33,518 --> 00:13:35,409 here are the demographics of the jurors. 202 00:13:35,409 --> 00:13:37,099 You're not gonna pick them anymore. 203 00:13:37,099 --> 00:13:38,349 No more Michael Scott's. 204 00:13:38,349 --> 00:13:38,699 So. 205 00:13:38,699 --> 00:13:44,451 part coded in all of the demographics with a lengthy system prompt ah using Lang chain if anyone cares. 206 00:13:44,451 --> 00:13:54,354 Yeah, so we use basically the demographics from the real humans and told the platforms generate those exact humans and predict how incriminating this piece of evidence was. 207 00:13:54,354 --> 00:13:58,395 So we had an apples to apples comparison, how accurate were these platforms? 208 00:13:58,395 --> 00:13:59,956 And they weren't very accurate. 209 00:13:59,956 --> 00:14:01,368 Not only were they not accurate, 210 00:14:01,368 --> 00:14:03,340 but they were biased in different ways. 211 00:14:03,340 --> 00:14:16,733 ChatGPT was actually not very biased, then Jim and I thought the evidence was a little more incriminating on average, and then Claude was a little underrating in terms of 212 00:14:16,733 --> 00:14:17,364 incrimination. 213 00:14:17,364 --> 00:14:23,951 That was consistent basically across every type of demographic, so we saw those consistent skews in every single one. 214 00:14:23,951 --> 00:14:26,012 you know, X type of person, was deterministic. 215 00:14:26,012 --> 00:14:32,196 So if you're X type of person, then you necessarily would believe why you would, you know, if you're a certain type person, maybe you don't like the police. 216 00:14:32,196 --> 00:14:37,419 And so you would always find them, you know, guilty, you know, those types of things just happened over and over and over again. 217 00:14:37,419 --> 00:14:41,162 And we knew from our real world human data that that's not how real people behave at all. 218 00:14:41,162 --> 00:14:43,964 You could be a type of person and think something completely different. 219 00:14:43,964 --> 00:14:48,727 And so that, you know, just the replication of stereotypes was rampant through them. 220 00:14:48,727 --> 00:14:51,515 And this was something that as we were doing it, 221 00:14:51,515 --> 00:14:56,108 as somebody who sits in these things all day effectively, I was like, yeah, they're definitely gonna do that. 222 00:14:56,169 --> 00:14:59,801 But that was the most interesting thing to Haley, understandably, I think. 223 00:14:59,801 --> 00:15:04,505 And then when I presented this topic at a couple of different conferences, that was the thing that everybody latched onto. 224 00:15:04,505 --> 00:15:09,979 Like really, these are just wildly over-representing stereotypes and replicating stereotypes. 225 00:15:09,979 --> 00:15:12,360 That's catastrophic for the law in many ways, obviously. 226 00:15:12,360 --> 00:15:14,973 So that was something where I was like, people are interested in this. 227 00:15:14,973 --> 00:15:15,677 And so we... 228 00:15:15,677 --> 00:15:22,277 Spent a long time in the paper talking about that because it was something that maybe doesn't occur to a lot of people like the system prompts. 229 00:15:22,337 --> 00:15:25,597 If you're not, maybe Ted, you know what a system prompt is. 230 00:15:25,597 --> 00:15:34,377 But I think the average chat GPT user doesn't realize before you enter anything into the system, there's a five and 10 and 20 page potentially system prompt that has all of these 231 00:15:34,377 --> 00:15:37,377 instructions, do this, don't do that type stuff. 232 00:15:37,517 --> 00:15:41,737 so those types of things, I think maybe don't occur to the average lawyer. 233 00:15:41,737 --> 00:15:45,417 And they're definitely happening on the legal tools as well. 234 00:15:45,841 --> 00:15:47,739 You know, Westlaw has a lengthy system prompt. 235 00:15:47,739 --> 00:15:51,655 Lexus says they all have lengthy system prompts that tell it what it can and cannot do. 236 00:15:51,655 --> 00:15:52,195 Interesting. 237 00:15:52,195 --> 00:16:03,144 So is this kind of the Swiss army knife scenario where, know, they've got a lot of tools, including a little, little scissors there, but if you ever had to cut a thick piece of 238 00:16:03,144 --> 00:16:05,325 canvas with it, it's just not a fit. 239 00:16:05,325 --> 00:16:07,997 So it's good at doing a lot of things. 240 00:16:07,997 --> 00:16:08,588 Okay. 241 00:16:08,588 --> 00:16:11,410 But not really anything specialized. 242 00:16:11,410 --> 00:16:13,671 Is that really the dynamic at play? 243 00:16:13,870 --> 00:16:17,612 Yeah, and that's how tell the students, know, lawyers are very worried about hallucinations. 244 00:16:17,612 --> 00:16:18,972 In large language models. 245 00:16:18,972 --> 00:16:25,715 I saw students hallucinations at this point in time are basically solved if you're using Vlex's Vincent AI for using Lexus AI fusing Westlaw. 246 00:16:25,715 --> 00:16:27,476 It has a hyperlink to the case. 247 00:16:27,476 --> 00:16:30,477 It's your duty as an attorney to click that and go read the case. 248 00:16:30,477 --> 00:16:31,698 That's why they pay you the big bucks. 249 00:16:31,698 --> 00:16:33,899 But if you're using chat GBT, that's not the case. 250 00:16:33,899 --> 00:16:38,059 And so I think, you know, it is, think the Swiss Army Knife is a really good call. 251 00:16:38,059 --> 00:16:41,099 You're using a Swiss Army Knife when you should be using a skillsaw or something. 252 00:16:41,099 --> 00:16:46,159 And that skillsaw is Lexus, know, Vlex, whatever it is, which is a law specific tool. 253 00:16:46,159 --> 00:16:54,859 And here, what our results really showed is that these platforms were not good at replicating the noise in the real human responses. 254 00:16:54,859 --> 00:17:00,419 You know, it really seemed like demographics ended up being deterministic of what 255 00:17:00,455 --> 00:17:08,084 how incriminating they thought each evidence was, but we saw in our actual real human data that there was a lot more variation among demographics. 256 00:17:08,084 --> 00:17:18,587 so really at a high level, these platforms are just not equipped to replicate human judgment the way it actually comes out with real humans and their varying backgrounds. 257 00:17:18,587 --> 00:17:18,867 Yeah. 258 00:17:18,867 --> 00:17:22,250 And you know, I mean, I've actually noticed some interesting dynamics. 259 00:17:22,250 --> 00:17:26,773 The safety and alignment protocols are implemented very differently. 260 00:17:26,773 --> 00:17:35,568 Um, you know, there's four models that I use primarily, which are in chat, GBT, Claude, Gemini and Grok. 261 00:17:35,568 --> 00:17:40,912 And really I use them in that order in terms of frequency. 262 00:17:40,912 --> 00:17:45,685 but I ran into a really interesting little exercise where 263 00:17:45,823 --> 00:17:50,395 I saw there was a graphic on LinkedIn and it was a caricature of two people that I know. 264 00:17:50,395 --> 00:17:52,436 Um, in fact, they're both podcasters. 265 00:17:52,436 --> 00:17:55,217 was Richard Trowman's and sac of Bramowitz. 266 00:17:55,217 --> 00:18:02,380 And they had this picture and I was like, the guy who's supposed to be Richard Trowman's looks just like this actor, but I can't think of who it is. 267 00:18:02,380 --> 00:18:12,164 So I took a quick screenshot and put it into, I think I did Claude first and was just like, what famous actor does this per does the person given the thumbs up look like? 268 00:18:12,204 --> 00:18:13,458 And it said, 269 00:18:13,458 --> 00:18:14,228 I'm sorry. 270 00:18:14,228 --> 00:18:17,593 I actually have the, let's see if I can find it real quick. 271 00:18:17,593 --> 00:18:19,854 what it said, cause it was really interesting. 272 00:18:19,854 --> 00:18:28,569 it basically refused to do it and it did so on the basis of it was trying to be, here we go. 273 00:18:28,569 --> 00:18:29,189 Let's see. 274 00:18:29,189 --> 00:18:35,262 It says, I see the image shows two people in what appears to be a promotional blah, blah. 275 00:18:35,262 --> 00:18:35,965 However, 276 00:18:35,965 --> 00:18:48,165 I can't identify who specific people in images look like or compare them to famous individuals as this could involve making assumptions about people's identity based on 277 00:18:48,165 --> 00:18:49,385 their appearance. 278 00:18:49,405 --> 00:18:52,545 And I was like, all right, that's interesting. 279 00:18:52,545 --> 00:18:54,125 Let me go to ChatGPT. 280 00:18:54,125 --> 00:18:56,525 It's a little goofy, but okay, whatever. 281 00:18:56,525 --> 00:19:03,085 They're probably trying to solve some sort of a bias. 282 00:19:03,085 --> 00:19:05,361 or privacy thing or like you said, bias. 283 00:19:05,361 --> 00:19:08,939 It doesn't want to say that I look like somebody stereotypically a thing, whatever. 284 00:19:08,939 --> 00:19:09,510 Yeah. 285 00:19:09,510 --> 00:19:10,852 So I went to ChatGBT. 286 00:19:10,852 --> 00:19:12,905 ChatGBT just flat out refused. 287 00:19:12,905 --> 00:19:15,188 It just said, I cannot help you with this request. 288 00:19:15,188 --> 00:19:16,891 Gemini did it. 289 00:19:16,891 --> 00:19:20,207 I still don't know who the actor was, um unfortunately. 290 00:19:20,207 --> 00:19:22,027 So that's surprising when we were running. 291 00:19:22,027 --> 00:19:23,529 we're talking about refusal rates. 292 00:19:23,529 --> 00:19:26,822 When we were running ours, Jim and I had the highest refusal rate, right? 293 00:19:26,822 --> 00:19:27,703 Jim and I had it. 294 00:19:27,703 --> 00:19:32,717 And even for the nerds in the audience, even through the API, it would say, sorry, I can't do that. 295 00:19:32,717 --> 00:19:34,479 I can't, I can't engage in legal advice. 296 00:19:34,479 --> 00:19:40,724 Now subsequently, you know, and this is all obviously moving, moving terrain under our feet at all times. 297 00:19:40,724 --> 00:19:42,766 So you can do it one day and it'll refuse to do it the next day. 298 00:19:42,766 --> 00:19:45,869 We'll do it because they've dialed something back or down. 299 00:19:45,869 --> 00:19:47,864 But that was um for us, it was 300 00:19:47,864 --> 00:19:50,426 Gemini and Claude were the two highest refusal rates. 301 00:19:50,426 --> 00:19:52,097 ChatGPT was a little bit better. 302 00:19:52,097 --> 00:19:57,561 And even through the API, I just like, I'm not going to engage in, you know, essentially legal advice. 303 00:19:57,561 --> 00:20:00,273 And you know, what you realize is there was a time there. 304 00:20:00,273 --> 00:20:09,449 And I think that that time has probably passed where, know, if the state of California can sue legal zoom for forms and that's considered unauthorized practice of law, these are 305 00:20:09,449 --> 00:20:12,802 pretty uncomfortable with, you know, anything kind of approximating legal advice. 306 00:20:12,802 --> 00:20:14,783 Now, now they give legal advice, no problem. 307 00:20:14,783 --> 00:20:16,290 They'll issue a bunch of caveats. 308 00:20:16,290 --> 00:20:18,775 but they will give you lots of legal advice. 309 00:20:18,775 --> 00:20:24,327 I don't know what movement behind the scenes has happened, but they all kind of collectively said, we can give some legal advice. 310 00:20:24,327 --> 00:20:25,100 That's what it seems like. 311 00:20:25,100 --> 00:20:26,220 You should try your picture out again. 312 00:20:26,220 --> 00:20:26,821 Yeah. 313 00:20:26,821 --> 00:20:27,633 now it'll do it. 314 00:20:27,633 --> 00:20:28,079 Yeah. 315 00:20:28,079 --> 00:20:28,990 that's a good point. 316 00:20:28,990 --> 00:20:32,063 This was, this was probably a couple of months ago. 317 00:20:32,063 --> 00:20:36,308 So, yeah, there's not a date on this, but Grok did it, of course, which you would expect. 318 00:20:36,308 --> 00:20:39,282 Grok, not a lot of guard rails with Grok. 319 00:20:39,282 --> 00:20:45,067 And, you know, did, did you guys try Grok at all in this experiment? 320 00:20:45,192 --> 00:20:47,442 No, so we just stuck with those three. 321 00:20:47,442 --> 00:20:52,057 I just for a dumb reason, I just didn't have a GROCK subscription at that point so I can get to the API. 322 00:20:52,057 --> 00:20:53,188 So I didn't, I do now. 323 00:20:53,188 --> 00:20:55,370 And there's some stuff where GROCK is really useful. 324 00:20:55,370 --> 00:20:56,640 know, first of all, it's very fast. 325 00:20:56,640 --> 00:20:57,831 just in the chat interface. 326 00:20:57,831 --> 00:21:00,394 And then number two, it performs really, really well. 327 00:21:00,394 --> 00:21:09,761 think, you know, Elon Musk has kind of been in the political news so much that they don't report on his model the same way, but that it's a really interesting and powerful model. 328 00:21:09,761 --> 00:21:11,822 And the other thing is like, if you need 329 00:21:11,860 --> 00:21:19,204 You know, if you're on your phone and you need kind of a survey of kind of what is going on in social media, it's really useful because as Twitter data. 330 00:21:19,947 --> 00:21:22,087 I've done stock stuff for my wife with it. 331 00:21:22,087 --> 00:21:27,113 yeah, I was curious if there were any different results with Grok. 332 00:21:27,113 --> 00:21:32,297 Well, what about, um okay, so you tried the Frontier models, weren't successful. 333 00:21:32,297 --> 00:21:34,789 What was the next step? 334 00:21:34,789 --> 00:21:37,561 It sounds like you might've tried some open source models. 335 00:21:37,703 --> 00:21:39,445 Yeah, so I went on and eat. 336 00:21:39,445 --> 00:21:47,115 So I went on the leaderboard and kind of tried to find something that we could run locally so we could save token costs, of course, and was trying to look at, you know, less 337 00:21:47,115 --> 00:21:49,097 censored models was the idea. 338 00:21:49,097 --> 00:21:59,036 And actually, a handful of the models, even one of the Mistral models that was a French company, their Frontier Lab, and even one of those wouldn't do kind of law, advice, 339 00:21:59,036 --> 00:22:02,495 practice of law, replicate human experimentation type things. 340 00:22:02,495 --> 00:22:08,587 I was able to find a less censored version of Mistral that would do kind of whatever I told it to do, which was very useful. 341 00:22:08,587 --> 00:22:10,228 It was very useful for what we're doing. 342 00:22:10,228 --> 00:22:21,501 And so then we decided to try to take all of our real human responses, turn that into a JSON training set, and then fine tune that less censored model as just kind of like, will 343 00:22:21,501 --> 00:22:22,881 this do anything? 344 00:22:22,881 --> 00:22:26,332 And it ended up being much better than any of the other models that we did. 345 00:22:26,332 --> 00:22:28,621 So that was pretty interesting. 346 00:22:28,621 --> 00:22:29,981 And it was cheap too. 347 00:22:29,981 --> 00:22:34,661 So I remember I used to fine tune models kind of early in the chat, GPC revolution. 348 00:22:34,661 --> 00:22:36,001 And it was expensive. 349 00:22:36,001 --> 00:22:41,261 I mean, it would be too even for a small model, like a 24 billion parameter model would be hundreds of dollars. 350 00:22:41,261 --> 00:22:46,121 When I did this one, it was, I think it was $14 or something. 351 00:22:47,941 --> 00:22:49,321 I use a service. 352 00:22:49,321 --> 00:22:50,741 Well, I just use tokens for mine. 353 00:22:50,741 --> 00:22:52,121 So I did it with two different packages. 354 00:22:52,121 --> 00:22:54,061 There's an open source package called oxalotl. 355 00:22:54,061 --> 00:22:58,436 And then there's a service called open pipe, which is no, I guess no code. 356 00:22:58,452 --> 00:23:04,338 As long as you have the JSON training set, you can just throw it in there and you know, do whatever model and it'll also host it for inference, which is pretty cool. 357 00:23:04,338 --> 00:23:08,421 But initially I just did it at home on my home computer with axolotl. 358 00:23:08,883 --> 00:23:10,964 Open source, fine tuning package. 359 00:23:11,387 --> 00:23:12,969 did you try llama? 360 00:23:13,102 --> 00:23:14,822 No, we so we didn't do the llama. 361 00:23:14,822 --> 00:23:18,602 Oh, so we tested the llama model, but it ended up being even worse. 362 00:23:18,602 --> 00:23:24,382 And so I couldn't even I couldn't even fine tune it enough to get it to be anywhere near human level. 363 00:23:24,382 --> 00:23:29,353 And so I was like, all right, we have to find something less censored than llama, all the meta models. 364 00:23:29,353 --> 00:23:37,413 And then after the fine tuning, were, what, was the, how was the testing after, after you fine tuned? 365 00:23:37,838 --> 00:23:42,518 So we essentially just did the same thing that we did to the real humans and scaled it. 366 00:23:42,518 --> 00:23:45,358 I think we did it up to, was it 5,000? 367 00:23:45,358 --> 00:23:45,678 Yeah. 368 00:23:45,678 --> 00:23:46,478 It was 5,000. 369 00:23:46,478 --> 00:23:50,018 So instead of, we only had 1200 actual juror responses. 370 00:23:50,018 --> 00:23:59,698 So we scaled it up to 5,000 real human responses where we would say, you know, go read this, first take on this, you know, this demographic or take on a random demographic from 371 00:23:59,698 --> 00:24:07,158 the range present in the real jurors, then read the thing, then respond to the thing as if you are from, you know, said demographic. 372 00:24:07,247 --> 00:24:08,747 And then we just scaled it. 373 00:24:08,747 --> 00:24:14,655 it's just like, just ran my computer for like an evening, basically, just let it grind through 5,000 iterations. 374 00:24:14,655 --> 00:24:18,178 And then we took that and measured it against the real human responses. 375 00:24:18,395 --> 00:24:20,192 and how did they compare? 376 00:24:20,192 --> 00:24:27,512 Much better, much better, which it was one of those things where I wasn't sure if just fine tuning was going to be enough. 377 00:24:27,512 --> 00:24:29,672 It felt like something that was more. 378 00:24:29,672 --> 00:24:36,372 You might have to have like some kind of a genetic workflow and you know, frankly, we can't afford to pre train a model of our own. 379 00:24:36,372 --> 00:24:39,732 So I, I wasn't sure if just fine tuning would be enough. 380 00:24:39,732 --> 00:24:48,959 And then the other problem is when you're fine tuning a model, even a small model, like 24 billion parameters, we had 1200 Jason. 381 00:24:48,959 --> 00:24:52,639 data points, know, JSON rows or data points to use. 382 00:24:52,719 --> 00:24:58,099 Most people do like 50,000 and we'll see benefits up to 100,000 fine tuning. 383 00:24:58,099 --> 00:25:04,259 So we just thought it was possibly just way too small of a training set to have any, you know, measurable impact on it. 384 00:25:04,259 --> 00:25:10,999 And the fact that at 1200 it did was like, Whoa, Whoa, you know, this is something that again, it was fairly cheap. 385 00:25:10,999 --> 00:25:11,899 It could do it for free. 386 00:25:11,899 --> 00:25:15,959 If you had the right hardware, like the Mac studio that we're running all this stuff on, we can do it on that. 387 00:25:16,093 --> 00:25:17,925 So it was really interesting to go. 388 00:25:17,925 --> 00:25:22,947 was like one of those breaks, know, one of the million aha moments I seem to have every month with this technology. 389 00:25:22,947 --> 00:25:24,993 It was like, whoa, we can actually do this. 390 00:25:24,993 --> 00:25:26,646 You know, it's like what it felt like. 391 00:25:26,646 --> 00:25:31,206 And when you say much better, like give me a sense of magnitude. 392 00:25:31,206 --> 00:25:32,926 Like how, how did it do? 393 00:25:32,926 --> 00:25:46,466 I don't know if this was a percentage that you use to measure alignment, but like give me a sense of, it doesn't have to be the exact number, but was it two, two X three X better 394 00:25:46,466 --> 00:25:48,206 50 % better. 395 00:25:48,655 --> 00:25:55,106 It was, geez, I don't want to state it, because I had it right here, but my computer locked. 396 00:25:55,106 --> 00:25:59,408 It was, I wrote it down just so I could specifically say this. 397 00:25:59,408 --> 00:26:02,394 I remember reading it was significant. 398 00:26:02,394 --> 00:26:12,013 95 % accuracy and geez, like essentially the margin of error was half, half of what it was with the model, the frontier models. 399 00:26:12,013 --> 00:26:17,839 So it was, you know, twice as good effectively as chat GPT trying to do this on its own. 400 00:26:17,839 --> 00:26:24,385 And so, and again, like, you know, there was a lot of technical things that I didn't do that I would do differently. 401 00:26:24,385 --> 00:26:27,188 Now I would use a different style of fine tuning if I was going to do it. 402 00:26:27,188 --> 00:26:28,330 We would have, you know, 403 00:26:28,330 --> 00:26:29,950 even scale with synthetic data. 404 00:26:29,950 --> 00:26:36,870 So that was something that I thought about, but she didn't want to, uh, cause a lot of people had just have a reflexive bad reaction to synthetic data. 405 00:26:36,870 --> 00:26:44,830 But had we taken that and then scaled it up to, know, 50,000, you know, take 12, take 1200 and then use that to scale up to 50,000. 406 00:26:44,830 --> 00:26:46,530 And then they use that to train a model. 407 00:26:46,530 --> 00:26:49,210 It probably would have been substantially better even then. 408 00:26:49,210 --> 00:26:56,710 So, uh, you know, just for a small investment and a small amount of training data, we got pretty big wins. 409 00:26:56,710 --> 00:26:58,077 And that was like the real like, 410 00:26:58,077 --> 00:26:59,601 breakthrough, I guess, of the paper. 411 00:26:59,601 --> 00:27:01,435 Especially as compared to the platform. 412 00:27:01,435 --> 00:27:07,962 Yeah, and definitely substantially better if you just use like the API to Gemini or Claude or Google or OpenAI. 413 00:27:07,962 --> 00:27:08,313 Yeah. 414 00:27:08,313 --> 00:27:10,835 And what conclusions can we draw from this? 415 00:27:10,835 --> 00:27:24,621 So back to my earlier comment about there's a, it is a constant point of discussion in the legal tech community about are these legal specific solutions going to get steamrolled by 416 00:27:24,621 --> 00:27:26,322 the frontier models? 417 00:27:26,362 --> 00:27:33,916 And your study makes me think maybe, maybe not, because if the frontier models are 418 00:27:33,916 --> 00:27:41,270 have controls, we'll call them, in place that make them a Swiss army knife where they're good for a lot of things. 419 00:27:41,270 --> 00:27:46,953 So they're a mile wide and an inch deep, as opposed to being an inch wide and a mile deep. 420 00:27:47,134 --> 00:27:57,720 It seems like, I don't know if it's a fair conclusion or not, but your study for niche legal use cases, the frontier models might not be the best solution. 421 00:27:57,720 --> 00:28:00,341 I don't know, Haley, what's your thought on that? 422 00:28:00,562 --> 00:28:03,143 think that's exactly right, at least as of now. 423 00:28:03,143 --> 00:28:14,755 I think that our project really proves the need to have these models specifically trained on human data, human judgment in the realm of evaluating evidence in these legal 424 00:28:14,755 --> 00:28:15,415 scenarios. 425 00:28:15,415 --> 00:28:23,829 I think it could be huge in terms of, as you mentioned earlier, trial prep, things that lots of trial boutiques would really latch onto and utilize. 426 00:28:23,829 --> 00:28:25,699 But right now they're just not there. 427 00:28:25,699 --> 00:28:29,310 And I think it is because of that Swiss army knife nature. 428 00:28:29,478 --> 00:28:33,250 But I do think our fine tuning model shows that it is possible. 429 00:28:33,250 --> 00:28:44,038 So with lots more data and lots more training, there could be an essentially mock jury bot at some point in the future that could be reliable for trial prep and strategy and a lot 430 00:28:44,038 --> 00:28:44,789 of other things. 431 00:28:44,789 --> 00:28:48,792 Yeah, I know, right now the big the frontier models obviously have the data problem. 432 00:28:48,792 --> 00:28:55,457 There's only four good places you can get, you know, US legal data, Vlex, now Clio, guess, Vlex still, Westlaw, Lexus, Bloomberg. 433 00:28:55,457 --> 00:28:57,096 Now those are the only data sets. 434 00:28:57,096 --> 00:29:04,192 and they haven't purchased one of those yet, which probably indicates, which, you when I hear people talk about it, it's that they just don't think it's a big enough market right 435 00:29:04,192 --> 00:29:04,401 now. 436 00:29:04,401 --> 00:29:06,254 They're going after literally everyone. 437 00:29:06,254 --> 00:29:14,400 But if at some point they do purchase one of those data sets and have that on the backend, then that's really gonna blow up the business model of like a Lexus or Westlaw or Vlex, 438 00:29:14,400 --> 00:29:22,352 unless they're offering something else in addition to it, some sort of agentic workflow, some sort of tool that plugs into your firm, which I think is. 439 00:29:22,352 --> 00:29:27,216 why probably the Clio Vlex acquisition is really interesting to me right now. 440 00:29:27,216 --> 00:29:31,170 But right now, the frontier models have a data problem and they're just not focused on legal. 441 00:29:31,170 --> 00:29:38,045 They're too big for legal, which is crazy to think about with the 1.5 billion or whatever that's going into legal investment this year. 442 00:29:38,045 --> 00:29:41,659 It's like, no, we're still considered small dollars to them, I think. 443 00:29:41,659 --> 00:29:46,170 Yeah, well, and honestly, and the industry is extremely fragmented. 444 00:29:46,170 --> 00:29:51,452 And if you add up the entire Amlaw 100 revenue, it's $140 billion. 445 00:29:51,452 --> 00:29:55,694 Like that's, that would be like a fortune 100 company. 446 00:29:55,694 --> 00:29:58,689 So that's the entire, that's the top 100. 447 00:29:58,835 --> 00:30:05,598 So it is, it is a relatively small industry, when compared to others, but you know, 448 00:30:05,778 --> 00:30:09,040 There was an AI company that did try to buy one of the information providers. 449 00:30:09,040 --> 00:30:12,450 Harvey made a play for Vlex, I think as part of their series. 450 00:30:12,450 --> 00:30:17,865 They were trying to raise, yeah, five, 600 million to buy Vlex and that didn't happen. 451 00:30:17,865 --> 00:30:18,590 And then... 452 00:30:18,590 --> 00:30:26,445 I know some people who work at like, you know, the open legal data sets that are out there, some of the more free ones, and they've been approached by the likes of open AI for 453 00:30:26,445 --> 00:30:26,995 purchasing. 454 00:30:26,995 --> 00:30:31,006 So they don't have a complete set, but they have a really good set that could be really useful. 455 00:30:31,006 --> 00:30:38,058 You know, the benefit of having the Lexus and Westlaw is you've got all the secondary materials, which are beautiful for training, uh even on, you know, very niche kind of 456 00:30:38,058 --> 00:30:39,378 specific areas of the law. 457 00:30:39,378 --> 00:30:44,140 You've got this really good rich data that's curated by attorneys that have worked in the field for a billion years and whatever. 458 00:30:44,140 --> 00:30:44,831 So. 459 00:30:44,831 --> 00:30:45,942 That is the benefit of those ones. 460 00:30:45,942 --> 00:30:50,213 You Felix went the really interesting opposite route, which is we don't have the secondary materials. 461 00:30:50,213 --> 00:30:53,889 We're just going to make it all a Gentic and we're going to craft you an on-demand treatise. 462 00:30:53,889 --> 00:30:59,763 So I think, you know, unless they can get ahold of Westlar or Lexis, I can't imagine Westlar wants to sell ever. 463 00:30:59,763 --> 00:31:01,624 You know, their data is there. 464 00:31:01,624 --> 00:31:05,387 That is their gold mine that they're sitting on, you know, Thompson writers. 465 00:31:05,387 --> 00:31:08,509 But if they can get ahold of Lexis, that would be some pretty interesting content. 466 00:31:08,509 --> 00:31:11,961 And, know, Harvey now has the API to Lexis. 467 00:31:11,961 --> 00:31:13,252 So I don't know. 468 00:31:13,252 --> 00:31:13,582 see. 469 00:31:13,582 --> 00:31:14,305 We'll see. 470 00:31:14,305 --> 00:31:18,674 Yeah, I've heard that integration is very surface level. 471 00:31:18,674 --> 00:31:19,961 um 472 00:31:19,961 --> 00:31:20,932 struck me as strange. 473 00:31:20,932 --> 00:31:26,681 think a lot of techie people, I think it struck us as strange because it's like, know, what you want is good structured data. 474 00:31:26,681 --> 00:31:33,520 You know, it's like saying, here's this big gulp and here's one those little coffee straws to drink out of it, you know, with the API. 475 00:31:33,520 --> 00:31:37,154 So it's like, ah maybe they'll fix that in the future. 476 00:31:37,154 --> 00:31:37,667 I don't know. 477 00:31:37,667 --> 00:31:42,469 Yeah, it's and you know, this brings up another interesting question and something I talk about a lot. 478 00:31:42,469 --> 00:31:57,285 So I'm of the opinion that in the future, let's say five years from now, once we have a enter the tech enabled legal service delivery era, um, I'm a big believer that law firms 479 00:31:57,285 --> 00:32:02,138 are going to have to leverage their data and in order to differentiate, right? 480 00:32:02,138 --> 00:32:04,329 Buying an off the shelf tool is not going to 481 00:32:04,329 --> 00:32:08,160 to differentiate you because your competitor can go buy the same tool. 482 00:32:08,160 --> 00:32:10,421 But you know what is unique to you? 483 00:32:10,421 --> 00:32:15,812 All of those documents that provided winning outcomes for your clients, right? 484 00:32:15,812 --> 00:32:30,569 All the precedent libraries of model documents that have been battle tested and the way legal works today, it's a very bespoke industry and everybody kind of has their own. 485 00:32:30,705 --> 00:32:39,770 You can ask 10 different lawyers at the same law firm in the same practice for the same document and get different iterations of it. 486 00:32:39,811 --> 00:32:44,453 So, you know, I, it makes me wonder something's got to give there, right? 487 00:32:44,453 --> 00:32:55,391 Like if we're going to tech enable legal service delivery, how, how we can't train models to be reflect one lawyers. 488 00:32:55,897 --> 00:33:00,299 perspective, even though that is what Crosby's doing, by the way, I don't know if you know about Crosby AI. 489 00:33:00,299 --> 00:33:11,494 Um, they, they're, I think it was Sequoia is one of the VCs in their cap table and they, they have a podcast called training data and I listened to that and that's, that is what 490 00:33:11,494 --> 00:33:11,983 they're doing. 491 00:33:11,983 --> 00:33:14,295 They just, but they just have a handful of attorneys. 492 00:33:14,295 --> 00:33:20,737 So they are literally training the attorneys, if I understood it correctly, um, to think like that lawyer. 493 00:33:21,028 --> 00:33:24,611 and that's, that's great and makes a lot of sense as 494 00:33:24,611 --> 00:33:26,332 this transitional phase, right? 495 00:33:26,332 --> 00:33:29,574 Like, I don't think that scales, right? 496 00:33:29,574 --> 00:33:34,256 It's expensive and it's one-off and what happens if they lateral? 497 00:33:34,256 --> 00:33:46,294 they take, you know, it's just, so I don't know, what is your thought on kind of the future and how much law firm data is going to come into play and how firms differentiate 498 00:33:46,294 --> 00:33:47,284 themselves? 499 00:33:47,491 --> 00:33:49,131 I think it's gonna be super interesting. 500 00:33:49,131 --> 00:33:55,411 you know, I think, I went to, once I was, I had a consulting appointment at a law firm. 501 00:33:55,411 --> 00:34:02,871 They wanted to implement something very simple, which is just a rag pipeline for all their pleadings, you know, so that they could generate stuff. 502 00:34:03,031 --> 00:34:05,111 And I went over and I said, okay, where's your data? 503 00:34:05,111 --> 00:34:07,191 And they said, well, some of it's in OneDrive. 504 00:34:07,191 --> 00:34:08,451 And I said, okay. 505 00:34:08,511 --> 00:34:11,771 And some of it's, you know, some of it's in NetDocs. 506 00:34:11,771 --> 00:34:12,771 I said, okay. 507 00:34:12,771 --> 00:34:19,071 And they said, some of it's on this old server that we had, you know, like in a box is that some of it's just paper on a shelf. 508 00:34:19,071 --> 00:34:24,491 But, then when you got to the, got to the planes themselves, all different formats, no good metadata, blah, blah. 509 00:34:24,491 --> 00:34:32,091 And I'm like, it's just going to take a lot to get your, you know, your information into a place where you could actually ingest it into any of these AI systems. 510 00:34:32,171 --> 00:34:35,211 So I think that's obviously step one is just getting 511 00:34:35,211 --> 00:34:43,219 a lot of law firms which have kind of lagged behind just modern data practices, get it all into a data lake or whatever so you can actually make use of this thing is going to be a 512 00:34:43,219 --> 00:34:43,959 big part of it. 513 00:34:43,959 --> 00:34:52,506 But I, you know, I think you're right that there is going to definitely be a period of time where there is going to be a custom tool. 514 00:34:52,506 --> 00:34:56,929 You Joe only do immigration law here is your custom tool that uses your internal things. 515 00:34:56,929 --> 00:35:00,092 and by the way, fine tuning now costs $14 or whatever, you know. 516 00:35:00,092 --> 00:35:09,732 So we can fine tune your little tiny model that just sits on prem on your stuff and really understands your workflow and who Ted is in the context of your firm and all of those 517 00:35:09,732 --> 00:35:10,392 things. 518 00:35:10,392 --> 00:35:12,072 I think there will be a while for that. 519 00:35:12,072 --> 00:35:21,232 think at some point it's probably gonna get eaten up by the big guys where they're gonna, know, VLex is already launching the ability to create your own workflows for your firm. 520 00:35:21,232 --> 00:35:24,892 Harvey has some sort of version of that from what I understand. 521 00:35:24,892 --> 00:35:29,692 So I think they're gonna roll out tools that are probably good for 70 % of the market. 522 00:35:30,054 --> 00:35:40,079 But there may be this place on the margins where you can outperform them substantially by using your own data and leveraging it within an internal ecosystem that is good for AI. 523 00:35:40,079 --> 00:35:43,503 So we were going to talk about a couple of things here. 524 00:35:43,503 --> 00:35:50,960 I want to skip down to legal education because I had a conversation this morning with Ed Walters from Felix. 525 00:35:51,602 --> 00:35:53,143 Yeah, he's a great guy. 526 00:35:53,143 --> 00:35:55,646 He's actually going to be on our 100th episode. 527 00:35:55,646 --> 00:35:57,103 um 528 00:35:57,103 --> 00:35:58,084 He's always a good get. 529 00:35:58,084 --> 00:35:59,785 Always has something interesting to say. 530 00:35:59,805 --> 00:36:01,107 And he's really nice. 531 00:36:01,107 --> 00:36:02,489 So I see him at conferences. 532 00:36:02,489 --> 00:36:05,331 He's just a really, really nice guy too. 533 00:36:06,553 --> 00:36:07,101 Yeah. 534 00:36:07,101 --> 00:36:12,405 we were talking a little bit about lawyer training and you guys are at the forefront of that. 535 00:36:12,566 --> 00:36:27,159 So I'm curious, Haley, like how do you think about how in an academic setting, lawyer training is going to need to evolve from where it is today to where we may be heading 536 00:36:27,159 --> 00:36:29,587 three, four years ago when the 537 00:36:29,587 --> 00:36:37,629 blocking and tackling work is really automated and document generation and we need to elevate lawyers to be more consultants. 538 00:36:37,629 --> 00:36:42,185 Like, how are you guys thinking about it at the law school or are you? 539 00:36:42,185 --> 00:36:45,089 Is it kind of a wait and see situation? 540 00:36:45,089 --> 00:36:45,877 What's that look like? 541 00:36:45,877 --> 00:36:47,618 think it depends on who you talk to. 542 00:36:47,618 --> 00:36:49,820 There's definitely resistance. 543 00:36:49,820 --> 00:36:57,926 I'm a relatively new professor and obviously I like AI and new things and so I think I'm a little bit more forward thinking about all of these things. 544 00:36:57,926 --> 00:37:10,600 But there is a very serious discussion that I think everyone is like engaging in about our students and their future because what first, second, third year attorneys usually do, 545 00:37:10,600 --> 00:37:15,505 Doc review, things like that are all the things that AI is going to be able to do in two seconds, right? 546 00:37:15,505 --> 00:37:22,882 And so what I think is really important for us as educators is to train students to know how to use these tools. 547 00:37:22,882 --> 00:37:27,996 They can come in as the people that can train all the other people in the law firms, right? 548 00:37:27,996 --> 00:37:30,180 So they can bring value that way. 549 00:37:30,180 --> 00:37:36,737 So like in my classes, I don't mind if students use AI to help them with practice problems, generate potential. 550 00:37:36,737 --> 00:37:40,449 multiple choice questions, essay answers, tell them what they did right or wrong. 551 00:37:40,449 --> 00:37:42,101 I encourage them to do that. 552 00:37:42,101 --> 00:37:51,466 And I think, you know, it's pretty similar, you know, even if AI is helping you generate your response to something, like that's what's essentially going on in law firms right now 553 00:37:51,466 --> 00:37:52,487 anyway, right? 554 00:37:52,487 --> 00:37:55,218 There's shell pleadings, motions. 555 00:37:55,218 --> 00:38:04,143 No one ever starts from scratch if you're in a law firm like that, although their data might be in boxes or on old servers, em as Sean recently found out, right? 556 00:38:04,201 --> 00:38:09,193 But I think it's coming no matter what we want to do about it or how we think about it. 557 00:38:09,193 --> 00:38:18,988 And so my general approach is just to encourage students to learn as much as they can, go to the digital initiatives with Sean that he puts on with Kitten all the time, which is 558 00:38:18,988 --> 00:38:20,559 really great for students. 559 00:38:20,559 --> 00:38:27,153 And I'm trying to learn as well so that I can be a resource for them to put them in the best position possible to succeed. 560 00:38:27,153 --> 00:38:29,266 know that Deans are thinking about it for sure. 561 00:38:29,266 --> 00:38:34,671 Because a huge portion of their your US World News ranking is based on your employment data of your students. 562 00:38:34,671 --> 00:38:43,129 And so we're finally, you know, people have been predicting these like catastrophic job layoffs since Chad GVT came around, we're finally now getting studies that are showing 563 00:38:43,129 --> 00:38:44,110 that this is really happening. 564 00:38:44,110 --> 00:38:54,499 I think Accenture just laid off 11,000 people there was that big study that came out of I think it was Stanford that had 62 million back end data points from the payroll system. 565 00:38:54,499 --> 00:38:59,899 that showed a 13.2 % decline in hiring among younger people. 566 00:39:00,159 --> 00:39:10,319 I, know, with the kind of traditional law school or law firm pyramid where this bottom layer of people, this document review discovery, simple, you know, kind of simple drafting 567 00:39:10,319 --> 00:39:14,639 research, like AI does all of those things very well and collapses it down. 568 00:39:14,639 --> 00:39:19,331 you know, I think when we think of law firms going forward, it's either going to be kind of a cylinder. 569 00:39:19,331 --> 00:39:25,931 or even a diamond shape is what other people have predicted, which I think is pretty interesting, but it raises a lot of problems for law schools. 570 00:39:26,191 --> 00:39:35,351 the way that I've thought about filling in what are those people gonna go do if they're not going to work at Kirkland, Junior's, Jackson Walker, McAfee Taft, one of the big firms 571 00:39:35,351 --> 00:39:37,731 down here, go and don't leave me where your mom works. 572 00:39:38,031 --> 00:39:47,171 I think non-traditional careers, you I kind of think we're headed for probably a 2008, 2009 type scenario where they're just not hiring first year associates as much. 573 00:39:47,202 --> 00:39:54,178 And so my impulse is to say to my students, hey, especially the ones my AI in the practice of law class, you know, it'd be really cool career legal ops. 574 00:39:54,619 --> 00:40:02,766 And that's not something that law schools traditionally use legal leverage their career services and send people to clock and talk to people you don't like you who are really 575 00:40:02,766 --> 00:40:03,537 into these areas. 576 00:40:03,537 --> 00:40:09,313 And I'm like, that's a great lifestyle, you know, you don't have to go work at a firm, you can still make great money. 577 00:40:09,313 --> 00:40:12,260 And if you're a techie person, like you know me or 578 00:40:12,260 --> 00:40:15,374 you know, like when I was graduating from law school, wouldn't want to go do document review at big firm. 579 00:40:15,374 --> 00:40:20,931 I would have loved to have done something like that instead, but I didn't know that was an option and it wasn't the traditional path for law students. 580 00:40:20,931 --> 00:40:28,592 So I think we need to be a little more creative and hopefully we're setting up the students that come through here so that they could go right into those careers and be 581 00:40:28,592 --> 00:40:29,713 really effective. 582 00:40:29,779 --> 00:40:30,119 Yeah. 583 00:40:30,119 --> 00:40:37,133 So, you I've been an entrepreneur for 32 years and during that time I have consumed a lot of legal services. 584 00:40:37,133 --> 00:40:47,450 And one of the big challenges I've talked about it once or twice on the show before is Lawyers are really good at identifying risk. 585 00:40:47,450 --> 00:40:50,613 as a business person, that's only half the equation. 586 00:40:50,613 --> 00:40:56,322 is a, almost every decision is a risk reward or a cost benefit. 587 00:40:56,322 --> 00:40:57,733 decision, right? 588 00:40:57,733 --> 00:40:59,114 You're balancing those scales. 589 00:40:59,114 --> 00:41:08,250 And lawyers historically have not taken the time to get to know my business because I'd have to pay them to do that, right? 590 00:41:08,250 --> 00:41:09,701 And that would be really expensive. 591 00:41:09,701 --> 00:41:19,128 I understand why they haven't, but you know, like that's always been a challenge from where I sit because I've had lawyers, you know, basically advise me, don't sign this 592 00:41:19,128 --> 00:41:19,888 agreement. 593 00:41:19,888 --> 00:41:21,959 And I'm like, you can't say that. 594 00:41:22,016 --> 00:41:25,406 What you can tell me is you can inform me on the risk side of the equation. 595 00:41:25,406 --> 00:41:31,008 I, as the business person, need to balance the business opportunity, i.e. 596 00:41:31,008 --> 00:41:33,709 the reward side of the equation. 597 00:41:33,709 --> 00:41:35,769 What you're bringing to me is very important. 598 00:41:35,769 --> 00:41:37,770 It helps me balance the scales. 599 00:41:37,810 --> 00:41:46,905 So what I would love to see as a consumer of legal services is lawyers get more in an advisory capacity and bring the human element. 600 00:41:46,905 --> 00:41:48,247 I'll give you an example. 601 00:41:48,247 --> 00:41:59,690 So we hired someone away from a, they're not really a competitor, but a company who had a non-compete and we had to craft his job description in a way where we wouldn't run a foul 602 00:41:59,690 --> 00:42:00,931 of that non-compete. 603 00:42:00,931 --> 00:42:06,056 And we hired a L &E attorney and he gave great advice and it wasn't really legal. 604 00:42:06,056 --> 00:42:09,662 He's like, look, you know, the reality is, you know, for the first, 605 00:42:09,662 --> 00:42:13,462 X number of months, there's probably going to be a lot of scrutiny around this. 606 00:42:13,462 --> 00:42:22,162 And then over time that will probably, you know, there, there are a bill, a judge wouldn't look kindly if six months down the road, they, they threw a flag, right? 607 00:42:22,162 --> 00:42:23,842 So he was giving me human advice. 608 00:42:23,842 --> 00:42:27,002 Like, you know, this is how the company is going to look at this. 609 00:42:27,002 --> 00:42:31,882 This is how the judge would look at an action that came six months later. 610 00:42:31,882 --> 00:42:37,082 Like that sort of stuff is going to be really hard to get out of chat GPT. 611 00:42:37,082 --> 00:42:38,382 It requires 612 00:42:38,458 --> 00:42:39,730 Experience is it possible? 613 00:42:39,730 --> 00:42:42,724 I guess I right 614 00:42:42,836 --> 00:42:48,489 you have to fill up the current 1 million token context window in order to have all that context if then, you know, even. 615 00:42:48,489 --> 00:43:01,557 Yeah, and even then, this guy was a seasoned vet and had spent 30 years on plaintiff side cases. 616 00:43:01,557 --> 00:43:10,102 And again, how you load that into an AI model and get solid judgment back, I think, presents challenges. 617 00:43:10,102 --> 00:43:11,825 uh 618 00:43:11,825 --> 00:43:16,207 when they survey, they survey, you know, our customers from law firms, they all hate us. 619 00:43:16,207 --> 00:43:17,807 They hate to see us. 620 00:43:17,807 --> 00:43:18,888 They hate to talk to us. 621 00:43:18,888 --> 00:43:26,162 They hate to pick up the phone in advance, which is, know, in many instances would remediate a lot of risk if they could just, hey, Sean, what about this? 622 00:43:26,162 --> 00:43:27,332 They hate doing all that stuff. 623 00:43:27,332 --> 00:43:35,445 So, I mean, that's why when you hear it, like a lot of the access to justice, people talk about this technology, the more optimistic ones, they go look, 78 % of the market doesn't 624 00:43:35,445 --> 00:43:38,956 have access to a civil attorney, an attorney in a civil context. 625 00:43:38,956 --> 00:43:40,633 If this drops the cost. 626 00:43:40,633 --> 00:43:50,178 cost and it allows a whole nother layer of people to have better legal outcomes or God forbid uh makes us have just better interactions with our clients so that they want to 627 00:43:50,178 --> 00:43:50,969 come talk to us. 628 00:43:50,969 --> 00:43:55,507 They enjoy picking up the phone because they feel more reassured about their business or whatever. 629 00:43:55,507 --> 00:43:57,192 You know, there is a lot of promise with this stuff. 630 00:43:57,192 --> 00:44:02,525 I try not to be completely starry eyed and rose colored glasses about it, but I'm just an optimistic person. 631 00:44:02,525 --> 00:44:08,749 I think there are a lot of wins here or just our interactions with our clients and better legal outcomes for regular people. 632 00:44:08,749 --> 00:44:09,749 think too. 633 00:44:10,078 --> 00:44:10,258 Yeah. 634 00:44:10,258 --> 00:44:15,169 And you have to ask yourself, why did, why do they hate answering the phone or calling the lawyers? 635 00:44:15,169 --> 00:44:17,631 You know, and, I actually don't hate it. 636 00:44:17,631 --> 00:44:19,552 Uh, what I do hate. 637 00:44:19,552 --> 00:44:27,475 Yeah, no, I mean, you know, we've got, we've got, you know, probably the number one startup law firm in the country. 638 00:44:27,475 --> 00:44:29,216 You can figure out who that is. 639 00:44:29,216 --> 00:44:35,179 And, it is, it is an education process every time I get to, talk to my counterpart over there. 640 00:44:35,179 --> 00:44:36,683 And I learned so much. 641 00:44:36,683 --> 00:44:37,393 And he's great. 642 00:44:37,393 --> 00:44:40,966 And he does think about things in risk reward. 643 00:44:40,966 --> 00:44:43,947 And that's why he's my attorney. 644 00:44:43,947 --> 00:44:46,209 And I enjoy picking up the phone. 645 00:44:46,209 --> 00:44:48,229 I don't enjoy getting the bills. 646 00:44:49,370 --> 00:44:54,174 That part's not fun, but I do feel like I get a lot of value from it. 647 00:44:54,174 --> 00:44:56,415 I've not always felt that way. 648 00:44:56,436 --> 00:45:00,378 And I don't know if this is a function, cause I've been in a lot of different businesses. 649 00:45:00,378 --> 00:45:02,779 My wife and I own five gyms here in St. 650 00:45:02,779 --> 00:45:03,528 Louis. 651 00:45:03,528 --> 00:45:06,519 fair amount of legal work that went into building that those out. 652 00:45:06,519 --> 00:45:07,859 I owned a collection agency. 653 00:45:07,859 --> 00:45:09,898 used to sue people all the time. 654 00:45:09,898 --> 00:45:11,461 A lot of legal work there. 655 00:45:11,461 --> 00:45:18,624 I've not always enjoyed it and I'm not sure if it's because these guys are super niche and they focus on venture backed startups like we are. 656 00:45:18,624 --> 00:45:25,206 But they really know our business and I learned something every time I talked to them and I enjoy it. 657 00:45:25,692 --> 00:45:28,436 Yeah, and there's a whole, you know, so I think there's two parts to that. 658 00:45:28,436 --> 00:45:36,043 Number one, some lawyers when you talk to them are sort of like when I talk to my IT department, excuse me, and I go, hey, I need to download this really interesting open 659 00:45:36,043 --> 00:45:43,788 source, completely unvetted software, because I need to keep my skills, you know, hot and, you know, up to the bleeding edge. 660 00:45:43,788 --> 00:45:45,229 And I want to use a cool new thing. 661 00:45:45,229 --> 00:45:46,460 And they just go, absolutely not. 662 00:45:46,460 --> 00:45:48,641 have no risk, no risk tolerance whatsoever. 663 00:45:48,641 --> 00:45:53,366 Because what's the incentive to them if I can use a new AI thing? 664 00:45:53,366 --> 00:45:54,546 Zero. 665 00:45:54,746 --> 00:45:58,988 What's the risk if I huge, you know, and so they just dial up risk. 666 00:45:58,988 --> 00:46:07,350 And I'm like, well, I have other interests, like remaining relevant or, you know, with library resources that happens very frequently with a thing we will not put behind 10 667 00:46:07,350 --> 00:46:09,431 different walls before the user can get to it. 668 00:46:09,431 --> 00:46:13,732 And a librarian goes, nobody's going to use it if it's a hassle to get in and use the thing. 669 00:46:13,732 --> 00:46:19,623 So it's just like, like you said, I think just the incentives are just pointed in the wrong direction for some lawyers and law firms. 670 00:46:19,623 --> 00:46:20,907 Yeah. 671 00:46:20,907 --> 00:46:23,667 Do you have anything to add to AI stuff I was talking about? 672 00:46:23,667 --> 00:46:24,987 Sorry, I got a pun intended. 673 00:46:25,107 --> 00:46:33,007 it just kind of reminded me when you're talking about the experience that the labor and employment attorney had that you found really valuable. 674 00:46:33,007 --> 00:46:43,227 I was a law clerk for a few judges and tons of firms and clients found it very valuable because I worked in judges chambers for years at a time. 675 00:46:43,227 --> 00:46:49,379 I understood how the judges I worked for thought, how their chambers operated, all this information that 676 00:46:49,385 --> 00:46:50,936 AI can't get its hands on, right? 677 00:46:50,936 --> 00:47:01,722 And so that was value I could bring to clients that is aside from document review and, you know, it's advice that can actually be beneficial that you're not going to get from AI or 678 00:47:01,722 --> 00:47:02,933 even every attorney. 679 00:47:02,933 --> 00:47:08,887 And so there are ways I think that attorneys just in general can utilize things that AI just can't do. 680 00:47:08,887 --> 00:47:11,148 Now, to me, though, it brings up this issue. 681 00:47:11,148 --> 00:47:13,799 You know, you're talking about the 30 years of experience. 682 00:47:13,799 --> 00:47:16,681 Well, if attorneys can't even get in the door. 683 00:47:16,681 --> 00:47:25,926 you know, after law school, because AI has taken over all of their tasks, we're going to end up with a huge shortage of attorneys at some point that have have any experience and 684 00:47:25,926 --> 00:47:35,261 then that you know, value might be lost if there's not a way to kind of have attorneys recalibrate like what they bring to the table and then get that experience as years go on. 685 00:47:35,261 --> 00:47:42,585 Yeah, I some people have, you know, when I go to other conferences, like Dan Lena was from Northwestern was talking about this where he was saying, you know, essentially, what we're 686 00:47:42,585 --> 00:47:45,256 going to see is just attorneys just highly specialized. 687 00:47:45,256 --> 00:47:53,221 So I'm an expert in e-discovery, in transactional, in California, for these real estate, you know, whatever, just really, really narrowly specialized. 688 00:47:53,221 --> 00:47:55,912 And so maybe that would be a situation like yours. 689 00:47:55,912 --> 00:47:59,514 You now got this attorney who he has dialed into your exact sector. 690 00:47:59,514 --> 00:48:03,617 You know that he's competent, you know, he has the right level of risk tolerance for you. 691 00:48:03,617 --> 00:48:07,279 And he gives you this white glove service that's very customized to Ted. 692 00:48:07,279 --> 00:48:13,054 And so maybe it'll just be that kind of times everything in law as what he, you know, they've hypothesized. 693 00:48:13,054 --> 00:48:14,950 They say the riches are in the niches. 694 00:48:14,950 --> 00:48:16,273 So, um, 695 00:48:16,273 --> 00:48:16,864 what I my students. 696 00:48:16,864 --> 00:48:18,296 I said, you know, this is a great time. 697 00:48:18,296 --> 00:48:24,403 Like you want lost services Chipotle burrito, you know, you want extra guac and so Fritas instead of carnitas. 698 00:48:24,403 --> 00:48:26,115 Let me get you exactly what you want, sir. 699 00:48:26,115 --> 00:48:28,068 That expensive burrito. 700 00:48:28,068 --> 00:48:29,930 Deliver it to your table for you. 701 00:48:29,930 --> 00:48:31,571 And here's the drink that you enjoy. 702 00:48:31,571 --> 00:48:33,665 Well, this has been a great conversation. 703 00:48:33,665 --> 00:48:38,031 really appreciate you guys coming and spending a little bit of time with me on the show. 704 00:48:38,031 --> 00:48:39,147 um 705 00:48:39,147 --> 00:48:42,850 got to kick the tires on our new podcast studio that Jim put together for us. 706 00:48:42,850 --> 00:48:44,211 Yeah, it's been great. 707 00:48:44,211 --> 00:48:44,611 Thank you. 708 00:48:44,611 --> 00:48:46,323 Awesome. 709 00:48:46,323 --> 00:48:47,183 I'm really impressed with it. 710 00:48:47,183 --> 00:48:49,543 I'm gonna do all my big presentations in here from now on. 711 00:48:49,543 --> 00:48:50,796 Yeah, you should. 712 00:48:50,796 --> 00:48:57,030 How do folks find out more about the work that you're doing there at the law school and you individually? 713 00:48:57,030 --> 00:48:58,863 What's the best way for them to do that? 714 00:48:59,111 --> 00:49:03,126 Probably for me, LinkedIn is kind of where I consolidate all my professional stuff. 715 00:49:03,126 --> 00:49:06,059 And it's, I'll have links to like my SSRN articles. 716 00:49:06,059 --> 00:49:09,102 And when I presented a thing, if I'm going to Ulta and presenting or whatever. 717 00:49:09,102 --> 00:49:09,793 Yeah. 718 00:49:09,793 --> 00:49:12,716 And then we're both, we both have pages on OU Laws website. 719 00:49:12,716 --> 00:49:17,470 So you can just search for our name, all of our papers and work will kind of be linked there as well. 720 00:49:17,470 --> 00:49:18,230 Okay. 721 00:49:18,230 --> 00:49:18,820 Well, awesome. 722 00:49:18,820 --> 00:49:22,332 We'll include those links in the show notes so folks can access them. 723 00:49:22,332 --> 00:49:28,893 And thank you so much for being on the show and hopefully we get to meet in person at a future conference. 724 00:49:30,974 --> 00:49:32,085 You as well. -->