1 00:00:02,654 --> 00:00:08,027 Jack, how are you this morning or I guess afternoon where you are? 2 00:00:08,075 --> 00:00:08,626 Yeah, I'm good. 3 00:00:08,626 --> 00:00:09,136 Thanks, Ted. 4 00:00:09,136 --> 00:00:13,760 Yeah, afternoon, but it's certainly getting quite dark, so it feels like the evening now. 5 00:00:13,854 --> 00:00:15,075 Gotcha. 6 00:00:15,235 --> 00:00:21,141 So, well, first of all, I really appreciate you taking the time to have this chat. 7 00:00:21,221 --> 00:00:31,631 I really enjoy your posts on LinkedIn and feel that you like to push a little bit and challenge and that's, those make for the best conversations. 8 00:00:31,631 --> 00:00:34,734 So I'm looking forward to our talk here this morning. 9 00:00:34,818 --> 00:00:37,846 Yeah, thank you and likewise and thanks for inviting me. 10 00:00:38,022 --> 00:00:39,104 Absolutely. 11 00:00:39,104 --> 00:00:44,313 So your background, you were an attorney at Freshfields. 12 00:00:44,313 --> 00:00:48,340 You're now at I-Manage in a knowledge consulting role. 13 00:00:48,340 --> 00:00:52,206 Tell us a little bit about who you are, what you do and where you do it. 14 00:00:53,366 --> 00:01:04,864 Yeah, so it's pretty hard to describe what I do, to be honest, because ever since I left private practice as a lawyer, I used to be a bankruptcy lawyer, did that for a few years. 15 00:01:04,864 --> 00:01:08,737 But ever since I've left that, it's been kind of hard to describe what I actually do. 16 00:01:08,737 --> 00:01:12,238 And the reality is that I have my fingers in all sorts of pies. 17 00:01:12,499 --> 00:01:16,452 My official role that I manage is to, as you say, help with knowledge consulting. 18 00:01:16,452 --> 00:01:23,316 So that's all things around cleaning up data within firms so that people are 19 00:01:23,514 --> 00:01:28,735 of finding good examples of things rather than bad drafts of things. 20 00:01:29,056 --> 00:01:41,369 Anything from that to taxonomies, metadata schemas, all the way through to the cultural aspects of knowledge management in law firms and how you can incentivize people to partake 21 00:01:41,369 --> 00:01:43,319 in knowledge sharing. 22 00:01:43,340 --> 00:01:51,842 So I do a lot of that stuff, but also like to think I've influenced a little bit the strategy of the company and the product side of things as well. 23 00:01:52,120 --> 00:01:53,432 There's something I'm interested in. 24 00:01:53,432 --> 00:01:56,737 I'll have a go at it and people can tell me to go away. 25 00:01:57,540 --> 00:02:00,034 I like to get involved with a lot of things. 26 00:02:00,034 --> 00:02:03,860 My general mission being just to improve how lawyers work really. 27 00:02:04,372 --> 00:02:05,243 Good stuff. 28 00:02:05,243 --> 00:02:09,746 And that's very relevant to the agenda that we carved out for today. 29 00:02:09,746 --> 00:02:23,908 But before we get into that, you and I were having a conversation earlier and we're talking a little bit about the episode with Damien Real and the concept of reasoning and 30 00:02:23,908 --> 00:02:27,841 which is a very hot topic and we're seeing more. 31 00:02:27,841 --> 00:02:30,263 We're seeing the word more and more. 32 00:02:30,484 --> 00:02:33,428 I just read Gemini 2's announcement. 33 00:02:33,428 --> 00:02:39,480 today and reasoning, the word reason and reasoning showed up multiple times. 34 00:02:39,480 --> 00:02:43,161 We hear about O-one and reasoning and chain of thought. 35 00:02:43,441 --> 00:02:48,903 And it's a really interesting topic to me because I think it matters. 36 00:02:48,903 --> 00:02:53,004 And I know a lot of people don't, whether it's truly reasoning or not. 37 00:02:53,004 --> 00:02:56,085 And what is your position on the topic? 38 00:02:56,085 --> 00:02:58,566 Do you believe that LLMs can reason? 39 00:03:00,035 --> 00:03:00,824 Wow. 40 00:03:02,774 --> 00:03:06,776 I don't think that is the most important question. 41 00:03:06,996 --> 00:03:15,259 I think that when we're looking at any technology, the question is always, what value can it add? 42 00:03:15,320 --> 00:03:25,084 And in the case of AI, it might be that AI can do some great things and achieve some great outcomes. 43 00:03:25,404 --> 00:03:30,106 And when we're looking at those things, we've got to always compare it with 44 00:03:30,636 --> 00:03:33,729 what the process looks like when humans are doing that same thing. 45 00:03:33,729 --> 00:03:39,934 You always got to benchmark it, both in terms of like efficiency, speed, but also quality. 46 00:03:39,934 --> 00:03:43,997 And one thing that I think people often miss is what you lose in the process. 47 00:03:44,077 --> 00:03:55,026 The example I always cite here is, let's do a quick thought experiment and let's say LLMs can produce 100 % accurate legal research memos with no hallucinations. 48 00:03:55,026 --> 00:03:56,907 Every citation is correct. 49 00:03:57,609 --> 00:03:59,790 Should we, does that mean we should use? 50 00:03:59,822 --> 00:04:03,022 AI to produce all our legal research memos? 51 00:04:03,022 --> 00:04:09,262 Some people might say yes and think I'm an idiot for even raising that question, but actually my answer is not yes. 52 00:04:09,262 --> 00:04:14,962 Because for me, the real question is, okay, what's the value of a legal research memo? 53 00:04:14,962 --> 00:04:18,182 Is it the words on the page of the research memo? 54 00:04:18,202 --> 00:04:22,342 If it is, then yeah, go for the AI that's 100 % accurate because it can do it quicker. 55 00:04:22,342 --> 00:04:26,818 But in my experience as a lawyer, clients don't really want a legal research memo. 56 00:04:26,818 --> 00:04:30,601 whenever I sent one of those out to clients, never really read it. 57 00:04:30,601 --> 00:04:40,568 The purpose of the legal research memo was to make sure that I had understood the case law and all the legislation so that we could then have a discussion about it so that the 58 00:04:40,568 --> 00:04:42,670 client could properly kick their tires. 59 00:04:42,670 --> 00:04:51,135 And fundamentally, the legal research memo really acted as evidence that we knew our stuff and that we are advising them on the best possible route. 60 00:04:51,476 --> 00:04:54,798 So when it comes to reasoning, 61 00:04:54,926 --> 00:05:04,811 I think my main answer to that question is I don't really care because for me the process is less important than kind of appraising all the pros and cons. 62 00:05:04,811 --> 00:05:09,406 But I know that you have a slightly different view, which I think I may even agree with. 63 00:05:09,406 --> 00:05:10,046 Yeah. 64 00:05:10,046 --> 00:05:22,093 Well, the reason I think it matters is because ultimately it's less about the LLM's ability to reason and more about their ability to comprehend, which is a first step in 65 00:05:22,093 --> 00:05:23,173 reasoning. 66 00:05:23,194 --> 00:05:38,236 So, you know, leveraging first principles, which is breaking down a problem to its most basic components and reasoning your way to a solution requires comprehension and 67 00:05:38,236 --> 00:05:44,030 at least in the traditional, in the human metaphor, right? 68 00:05:44,070 --> 00:05:50,674 Someone can't reason their way to a solution of a problem without understanding the problem, unless it's just pure luck. 69 00:05:51,035 --> 00:06:02,122 So the reason I think it's really important in terms of comprehension in a legal context is it really dictates what use cases become viable and not. 70 00:06:02,823 --> 00:06:07,422 I think it's a good guide for us to figure out 71 00:06:07,422 --> 00:06:11,207 where to apply the technology and where to stay away from. 72 00:06:11,416 --> 00:06:12,266 Yeah. 73 00:06:13,087 --> 00:06:18,430 Well, I think that I do actually agree with all of that. 74 00:06:19,591 --> 00:06:24,585 when ChatGPT came out, what was it, three years ago? 75 00:06:24,585 --> 00:06:26,225 Is it three years or two years? 76 00:06:26,225 --> 00:06:26,565 Right. 77 00:06:26,565 --> 00:06:27,655 Feels like it's been here forever. 78 00:06:27,655 --> 00:06:38,162 Anyway, I remember all the discussion was about what are the use cases and loads of people jumped to things like the obvious ones like drafting contracts and 79 00:06:38,658 --> 00:06:48,002 drafting research memos, a lot of people said things like blue booking, which I'll admit is not something that's familiar to me because I'm not a US attorney, but all these sort 80 00:06:48,002 --> 00:06:50,963 of weird and wonderful use cases came out of the woodwork. 81 00:06:50,963 --> 00:07:00,667 And for me, I think when you're appraising the ability of any technology to meet a given use case, having an understanding of how the technology works is pretty important. 82 00:07:00,927 --> 00:07:08,044 And I read a really good article year and a half ago by Stephen Wolfram, which is kind of very 83 00:07:08,044 --> 00:07:12,338 well-known article about how chat GPT works and what it's doing. 84 00:07:12,338 --> 00:07:19,725 And what I took from that 18 months ago was it's a statistical next word predictor. 85 00:07:19,825 --> 00:07:23,488 And I don't mean that to say, it's just predicting the next word. 86 00:07:23,489 --> 00:07:29,234 The thing that's magic about this is the fact that a next word predictor can produce such amazing outputs. 87 00:07:29,435 --> 00:07:33,282 And I don't think humans are next word predictors. 88 00:07:33,282 --> 00:07:41,194 I'm not saying that because to diminish the value of next word predictors, there are some situations where that would be a really, really useful thing to have, like spotting 89 00:07:41,194 --> 00:07:43,545 patterns over large amounts of information. 90 00:07:43,545 --> 00:07:50,167 But I exercises that require emotion, experience, principles, I think is less good at that. 91 00:07:50,167 --> 00:07:56,358 So I think to the extent you're appraising use cases, I think it's important to understand how these things work. 92 00:07:56,358 --> 00:07:59,829 And if you're going to push me to say, they reason or do they not reason? 93 00:08:00,229 --> 00:08:02,978 I'm pretty clearly in the camp of they don't reason, but 94 00:08:02,978 --> 00:08:08,859 The reason I say I don't care is because people get bogged down a little bit in, what does reasoning even mean? 95 00:08:08,999 --> 00:08:20,902 The thing that I always guard against about on this, I will say this is a bias, is that I am naturally very skeptical of anybody who tries to hype up any technology as being game 96 00:08:20,902 --> 00:08:21,302 changing. 97 00:08:21,302 --> 00:08:23,543 I always challenge people on that. 98 00:08:23,543 --> 00:08:32,286 And for me, the reason I don't like this whole reasoning thing is it seems to me people use that term to try and anthropomorphize the technology and to 99 00:08:32,286 --> 00:08:36,108 make everyone think of the future and robots and humans being out of jobs. 100 00:08:36,108 --> 00:08:38,329 And I think that is complete nonsense. 101 00:08:38,329 --> 00:08:47,624 So any language that gives that potentially could give a misleading capability, misleading impression of a technology capability, I generally try and avoid. 102 00:08:47,624 --> 00:08:56,058 that is a that's a bias I've had just through being in a position of buying technology that was promised to be amazing and turned out not to be quite so amazing. 103 00:08:56,830 --> 00:08:59,551 Yeah, there's a lot of that in the marketing. 104 00:08:59,551 --> 00:09:08,730 I think today the gap between the marketing message and the true capabilities in this space is as get as wise as it's ever been. 105 00:09:08,730 --> 00:09:20,060 Um, I thought that the, know, there's been a lot of controversy around the Stanford paper and I, I read the study multiple times and I actually got a lot of value from it. 106 00:09:20,060 --> 00:09:25,552 I do think they miscategorized things as hallucinations that were, were not. 107 00:09:25,684 --> 00:09:38,303 But some of the conclusions that they drew, I think are valuable in understanding the comprehensive or comprehension capabilities of LLMs. 108 00:09:38,423 --> 00:09:53,514 So they pointed out that LLMs or these legal research tools specifically that were geared towards legal, they don't understand holdings. 109 00:09:53,544 --> 00:09:56,156 They don't respect the order of authority. 110 00:09:56,156 --> 00:10:02,920 have difficulty distinguishing between legal actors, which all point to a lack of comprehension. 111 00:10:03,941 --> 00:10:15,480 So I think now there was a study, I just saw that it literally just came out this week, like Monday or Tuesday from Metta, Metta and University of California, San Diego. 112 00:10:15,480 --> 00:10:16,590 That's really interesting. 113 00:10:16,590 --> 00:10:21,554 I'm still skimming it, but they are proposing almost like a, 114 00:10:22,634 --> 00:10:26,494 a alternative to chain of thought. 115 00:10:26,553 --> 00:10:35,334 And the gaps that they identified are with, with LLMs, they allocate equal compute to every token. 116 00:10:35,334 --> 00:10:49,598 And the reality is that in a sentence or a prompt, are token words or tokens that require much more emphasis and resource cognitive resources. 117 00:10:49,598 --> 00:11:00,035 than others and they proposed a way to better allocate compute resources to the important areas of a question or prompt. 118 00:11:00,035 --> 00:11:01,706 It's really fascinating. 119 00:11:02,318 --> 00:11:18,458 I posted a link to it in that clip with Damien and I, but I think that's really where the rubber hits the road is, how does the lack of comprehension today dictate where we can use 120 00:11:18,458 --> 00:11:19,306 this in legal? 121 00:11:19,306 --> 00:11:20,845 I think that's what's important. 122 00:11:20,982 --> 00:11:22,143 Yeah, I agree. 123 00:11:22,143 --> 00:11:32,428 And that sounds interesting what you're talking about, though I've not seen that study, but potentially, presumably that could mean that for legal specific applications, you 124 00:11:32,428 --> 00:11:39,632 could choose to have compute power on particular terms that could do those things like respect the order of authority. 125 00:11:40,653 --> 00:11:47,176 But all this stuff, to be honest, it does make me wonder why we're going for these really difficult use cases to start with. 126 00:11:47,176 --> 00:11:49,197 This technology is still quite new. 127 00:11:50,530 --> 00:12:03,195 The issue I have with this stuff is there's a bunch of things that are grinding to a halt in law firms and in-house legal teams that are not really these kinds of problems. 128 00:12:03,996 --> 00:12:13,559 Like, example, people saving things in the wrong place or misleading comments in documents or bad version control. 129 00:12:13,600 --> 00:12:19,970 Why aren't we trying to look at ways of leveraging LLMs to try and improve those quite basic things? 130 00:12:19,970 --> 00:12:27,865 That seems to me to be quite a good idea because in doing that, we might learn from those things, learn from those applications, which by the way, are real tangible problems that 131 00:12:27,865 --> 00:12:29,816 people care about solving now. 132 00:12:29,856 --> 00:12:36,680 I don't know why we're not doing that rather than going for these really impressive sounding, but really complicated and difficult use cases. 133 00:12:36,680 --> 00:12:46,496 Like it seems, it seems like there's a bit of a scenario emerging where sometimes I feel like people developing these really interesting sounding capabilities are. 134 00:12:46,496 --> 00:12:52,831 more interested in the intellectual challenge of developing them than they are actually solving the problems people care about. 135 00:12:52,831 --> 00:13:01,338 But again, that's my bias coming in where I'm seeing lots of talk about capabilities maybe being overstretched a little bit in the market. 136 00:13:01,338 --> 00:13:06,472 And I feel like we could all help ourselves a little bit just by stepping back and simplifying things a little bit. 137 00:13:07,102 --> 00:13:11,073 Yeah, I have been beating that drum, a similar drum. 138 00:13:11,073 --> 00:13:24,067 I've been advocating for firms to really think about business of law use cases first, simply because it's, when you look at this, there's a risk reward equation that has to 139 00:13:24,067 --> 00:13:30,629 strike a balance whenever you're looking at deploying a technology or a process improvement initiative. 140 00:13:30,629 --> 00:13:35,850 And the risk reward on the business of law side, I think, 141 00:13:36,358 --> 00:13:38,959 evens out more, much more favorably. 142 00:13:38,959 --> 00:13:44,140 And it's an incremental step towards a broader AI strategy. 143 00:13:44,520 --> 00:13:45,541 It's lower risk. 144 00:13:45,541 --> 00:13:49,582 don't, mean, you've got HR, finance, marketing, KM. 145 00:13:49,582 --> 00:13:56,504 There's, there's so many places where you could deploy this technology and get familiar and then move into those. 146 00:13:56,504 --> 00:14:04,366 Cause the high ROI, everybody, you know, goes for the sexy, you know, use cases because 147 00:14:04,986 --> 00:14:11,090 the work is high level and the ROI is higher to solve those problems. 148 00:14:11,090 --> 00:14:25,680 But it also comes with much more risk, especially given not just privacy, ethics, capabilities, the opportunities for errors to cause real problems. 149 00:14:25,940 --> 00:14:34,776 when you look at it, people are selling this, including founders, under the premise of disrupting the practice of law. 150 00:14:34,776 --> 00:14:35,602 Hmm. 151 00:14:35,602 --> 00:14:40,244 purposes of creating savings. 152 00:14:40,244 --> 00:14:52,648 And it's not as sexy and interesting for a hundred dollar an hour business of law resource as it is a $1,500 a law, $1,500 an hour practice of law resource. 153 00:14:52,648 --> 00:14:54,828 So yeah, I agree with you, man. 154 00:14:54,828 --> 00:14:56,149 I start with the basics. 155 00:14:56,149 --> 00:15:00,700 And when is that ever a bad idea in technology, starting with the basics? 156 00:15:00,716 --> 00:15:03,058 like at the very least have a portfolio of it. 157 00:15:03,058 --> 00:15:05,740 So focus on all of the above. 158 00:15:05,740 --> 00:15:09,163 Have an R &D team that's doing the really interesting core stuff. 159 00:15:09,163 --> 00:15:18,601 But if we don't do that, we can end up in a world where a lawyer and a law firm is still manually typing out their time narratives every single day by looking back through their 160 00:15:18,601 --> 00:15:20,823 sent items in their inboxes. 161 00:15:20,823 --> 00:15:24,557 But they can have this magic legal research memo drafting. 162 00:15:24,557 --> 00:15:26,456 That just doesn't seem right to me. 163 00:15:26,456 --> 00:15:35,061 Like we've got a, my kind of impression is that a lot of people are saying they've got a portfolio of these things and they're doing some basic applications, but also the really 164 00:15:35,061 --> 00:15:36,322 interesting R and D stuff. 165 00:15:36,322 --> 00:15:45,487 But if you actually scratch beneath the surface, it's mostly the R and D stuff to be honest, because unsurprisingly, that's what all the data scientists want to be working on. 166 00:15:45,487 --> 00:15:46,458 get it. 167 00:15:46,778 --> 00:15:53,292 It's just a drum that I keep banging because I know a lot of lawyers was one myself and 168 00:15:54,134 --> 00:16:02,803 I just think you lose trust a little bit with these guys when you're lauding these amazing capabilities and they're still using a computer that takes 10 minutes to start up in the 169 00:16:02,803 --> 00:16:03,614 morning. 170 00:16:03,774 --> 00:16:04,685 thousand percent. 171 00:16:04,685 --> 00:16:08,347 That's where we get into the risk reward balance. 172 00:16:08,347 --> 00:16:13,661 And attorneys have a very low tolerance for missteps. 173 00:16:13,661 --> 00:16:25,038 before you put a technology in front of them, especially with lofty claims and ambitions and aspirations, like it's got to be right, or you're going to have trouble getting them 174 00:16:25,038 --> 00:16:28,700 back to the table the next time when it's actually ready. 175 00:16:28,825 --> 00:16:37,096 Well, I think that's why this accuracy thing is such a big deal around LLMs because a lawyer using a tool, they'll ask it a question that I can guarantee you the first question 176 00:16:37,096 --> 00:16:38,859 I'll ask it would be to test it. 177 00:16:38,859 --> 00:16:41,893 And then if it gets it even slightly wrong, they'll be like, no, I don't trust this. 178 00:16:41,893 --> 00:16:42,794 Never using it again. 179 00:16:42,794 --> 00:16:44,616 You've to be really careful about that. 180 00:16:45,050 --> 00:16:46,031 100%. 181 00:16:46,031 --> 00:16:47,091 Well, that's good stuff. 182 00:16:47,091 --> 00:16:52,612 You and I talked about search the last time we spoke, which is an interesting topic for us. 183 00:16:52,612 --> 00:16:55,143 So Infodash is an intranet extranet platform. 184 00:16:55,143 --> 00:17:02,925 We're not a search company, but we are the launch, the entry point for search. 185 00:17:02,925 --> 00:17:11,297 So most law firms enterprise search process initiates through their internet. 186 00:17:11,297 --> 00:17:14,128 That's not a hundred percent of the time, but it's common. 187 00:17:14,281 --> 00:17:17,552 So we're interested in the topic. 188 00:17:17,552 --> 00:17:27,234 And if I'm being honest, I don't know that I've ever seen a enterprise search project with a positive ROI in legal. 189 00:17:27,234 --> 00:17:29,275 And that's a big statement, right? 190 00:17:29,275 --> 00:17:33,856 It's not that there hasn't been good outcomes, because I have seen good outcomes. 191 00:17:33,856 --> 00:17:42,158 But when I look at how much was invested and then I look at what the adoption looks like long term, 192 00:17:43,210 --> 00:17:45,034 I have lot of questions about ROI. 193 00:17:45,034 --> 00:17:51,366 So do you think, you know, gen AI is going to change that equation at all? 194 00:17:52,471 --> 00:17:56,514 yeah, I really don't think so. 195 00:17:56,514 --> 00:18:03,959 think that Enterprise Search, I agree, is a very, very difficult project to pull off. 196 00:18:03,959 --> 00:18:13,166 I think there probably have been a few firms that have done it successfully, but not without investing a huge amount of money and in expertise of people who really know what 197 00:18:13,166 --> 00:18:14,266 they're doing. 198 00:18:14,267 --> 00:18:18,690 I think any kind of systems integration issue 199 00:18:19,762 --> 00:18:20,492 System is great. 200 00:18:20,492 --> 00:18:22,353 Your product is always going to be very, very difficult. 201 00:18:22,353 --> 00:18:23,504 You've got to match up. 202 00:18:23,504 --> 00:18:26,135 You've got to have data warehouse strategies in place. 203 00:18:26,135 --> 00:18:29,467 You've got to monitor the ingestion of content you're dealing with. 204 00:18:29,467 --> 00:18:34,639 If you're doing things like trying to index your entire time recording system, it has all your narratives in you. 205 00:18:34,639 --> 00:18:38,291 You're dealing with millions, if not billions of rows of data. 206 00:18:38,291 --> 00:18:42,893 It's very, very, very difficult to get things right. 207 00:18:42,893 --> 00:18:45,374 And then if you get things wrong, 208 00:18:45,378 --> 00:18:49,180 That also means it's quite difficult to re-engineer them because you've got so much data you're dealing with. 209 00:18:49,180 --> 00:18:55,003 So all that is to say that the ROI on enterprise search has to be extremely compelling. 210 00:18:55,224 --> 00:19:07,818 And what I spend quite a lot of time doing with firms and legal teams is exploring whether having more of a curated knowledge strategy in addition to that might be helpful because 211 00:19:07,818 --> 00:19:09,632 you're dealing with small amounts of content. 212 00:19:09,632 --> 00:19:13,536 Of course, the difficulty there is that there's a cultural barrier. 213 00:19:13,536 --> 00:19:15,076 there's an incentive blocker, et cetera. 214 00:19:15,076 --> 00:19:16,197 We can talk about that later. 215 00:19:16,197 --> 00:19:19,908 But do I think that gen AI is going to solve any of this? 216 00:19:20,488 --> 00:19:22,408 I don't see how it can really. 217 00:19:22,408 --> 00:19:30,571 think there's probably, there's probably, there's a couple of ways we can, we can try and answer this question. 218 00:19:30,831 --> 00:19:42,934 There are a lot of people out there using generative AI to, to deliver this kind of natural cert, natural language search or vector search capability. 219 00:19:43,082 --> 00:19:52,790 And what that basically does is it enables you to discover documents that conceptually match your term, even if the keyword you typed in isn't actually in the document. 220 00:19:52,811 --> 00:19:56,214 And that can certainly be really, really helpful. 221 00:19:56,214 --> 00:20:02,629 There's obviously cost implications in doing that because you'd have to do what's called vectorize your underlying document. 222 00:20:02,629 --> 00:20:07,623 I have to have a vector index to make that work, which is not the cheapest thing in the world to spin up. 223 00:20:07,924 --> 00:20:10,890 And it can help you discover things without 224 00:20:10,890 --> 00:20:16,353 knowing what the right keyword is to type in, but also it can generate a lot of noise in the search results. 225 00:20:16,353 --> 00:20:24,477 So sometimes you do want to search for a very specific keyword and you don't, you actively don't want other things to come up that don't match that keyword. 226 00:20:24,477 --> 00:20:27,519 So sometimes useful, sometimes it's not. 227 00:20:27,519 --> 00:20:38,535 I don't think it's going to solve any of the issues around enterprise search because that is a systems integration project, large amounts of data, GenAI, GenAI, GenAI may actually 228 00:20:38,535 --> 00:20:40,246 make that harder to be honest. 229 00:20:40,916 --> 00:20:48,128 Um, the other angle we can look at for gen AI is around nomination or labeling of content. 230 00:20:48,128 --> 00:20:58,251 So this might be more relevant for firms pursuing a knowledge strategy that isn't just a search over all of our documents, but more kind of can we build a knowledge base and can 231 00:20:58,251 --> 00:21:00,611 we categorize the documents in that knowledge base? 232 00:21:00,611 --> 00:21:05,033 Can we proactively suggest documents that could become a part of that knowledge base? 233 00:21:05,033 --> 00:21:06,753 There are some potentials there. 234 00:21:06,753 --> 00:21:10,694 I don't think that AI is in a place where it can 235 00:21:10,702 --> 00:21:15,062 nominate a document as being a good quality example for reuse in the future. 236 00:21:15,062 --> 00:21:17,342 I think that still requires a human trigger. 237 00:21:17,342 --> 00:21:21,542 But can it categorize documents effectively in accordance with some concepts? 238 00:21:21,542 --> 00:21:22,702 Absolutely. 239 00:21:22,702 --> 00:21:32,142 At iManage, we actually use traditional machine learning algorithms, if you remember those, on things like contract type, things like that, which is a lot quicker than GenAI. 240 00:21:32,142 --> 00:21:37,760 But we're also playing around with using GenAI to generate summaries of documents that might 241 00:21:37,760 --> 00:21:42,656 influence search and might provide more context to people when they're looking through search results. 242 00:21:42,897 --> 00:21:45,540 to answer your question, think enterprise search is really, really tough. 243 00:21:45,540 --> 00:21:47,232 I think a few people have already got it right. 244 00:21:47,232 --> 00:21:48,234 It's expensive. 245 00:21:48,234 --> 00:21:53,600 I don't think GenAI is going to help people massively on it, if I'm completely honest. 246 00:21:53,704 --> 00:21:57,941 So no perplexity for the enterprise anytime soon. 247 00:21:58,528 --> 00:22:02,229 Well, mean, maybe it depends. 248 00:22:02,229 --> 00:22:04,480 Again, it goes back to what your use case is. 249 00:22:04,480 --> 00:22:16,153 Like I think that sometimes with enterprise search, where I've seen those projects fail, it's usually because the project is conceived without any real idea of what the ROI is on 250 00:22:16,153 --> 00:22:16,583 it. 251 00:22:16,583 --> 00:22:19,244 It's kind of something that feels like the right thing to be doing. 252 00:22:19,244 --> 00:22:22,765 And maybe it's like an intellectually challenging project to do. 253 00:22:22,765 --> 00:22:28,236 And where those things, the motivation, you tend to have a really, really complex project. 254 00:22:28,398 --> 00:22:36,958 Whereas when it's focused around a particular use case, which might be, for example, let's say the use cases that a business development team wants to find pitches to help them win 255 00:22:36,958 --> 00:22:43,558 more work or spot connections between what different people are doing to help cement client relationships. 256 00:22:43,558 --> 00:22:52,038 If you're going to focus these things around those kinds of use cases rather than a broad bucket of search overall of our content, I think you might get somewhere, but it's all 257 00:22:52,038 --> 00:22:57,062 about trying to reduce that overall corpus of content and not just... 258 00:22:57,260 --> 00:23:00,553 trying to solve your data hygiene issues by throwing a search over it. 259 00:23:00,553 --> 00:23:03,624 I personally haven't seen too many of those projects work. 260 00:23:03,976 --> 00:23:10,204 Yeah, what is, I manage his insight plus and what is the strategy there? 261 00:23:10,978 --> 00:23:15,560 So Insight Plus is what we refer to as a knowledge search product. 262 00:23:15,640 --> 00:23:19,161 And it's a relatively new product. 263 00:23:19,161 --> 00:23:23,943 And we've got customers using Insight Plus over their knowledge bases. 264 00:23:23,943 --> 00:23:29,065 The way the product is kind of architected is around different search experiences. 265 00:23:29,065 --> 00:23:33,167 So it looks a little bit like an enterprise search when you go into it. 266 00:23:33,247 --> 00:23:37,378 But for reasons we may or may not go into, we don't call it enterprise search. 267 00:23:37,378 --> 00:23:40,418 But you have like a tab for knowledge, for example. 268 00:23:40,418 --> 00:23:45,678 And the whole search experience is geared around that particular use case of finding your firm's best practice. 269 00:23:45,678 --> 00:23:51,668 We've got workflows around helping people submit, share, approve, maintain content to go in there. 270 00:23:51,668 --> 00:24:01,697 And importantly, it's a knowledge search product, but what we found from our product research is that actually a lot of lawyers value browse a lot more over search. 271 00:24:01,697 --> 00:24:08,502 And we try to use ways to use metadata to power browsable experiences as well as search experiences. 272 00:24:08,814 --> 00:24:15,099 But there's also a new suite of search experiences coming out next year, which is more similar to the enterprise search stuff. 273 00:24:15,099 --> 00:24:22,486 And this is all about enriching workspace information, the document management system with third party information. 274 00:24:22,486 --> 00:24:26,930 So things like your your CRM or your billing information, that kind of thing. 275 00:24:26,930 --> 00:24:36,504 And those are really built around firms that maybe have a strategy of for various reasons, they don't want to pursue a curated knowledge strategy. 276 00:24:36,504 --> 00:24:40,555 but they just want more levers to pull when people are searching for content. 277 00:24:40,696 --> 00:24:47,529 But it can also deliver business development use cases like people wanting to search for matters that are particularly profitable. 278 00:24:47,529 --> 00:24:56,352 So Insight Plus is going to expand into things like closing books or deal bibles, as we call them in Europe as well over the years. 279 00:24:56,352 --> 00:25:00,984 And we'll also be looking at ways you can look at people results and things like that. 280 00:25:00,984 --> 00:25:06,026 But it's not enterprise search because we're not just throwing loads and loads of content at it. 281 00:25:06,198 --> 00:25:14,388 All of these things are designed around particular use cases, which helps us manage the process and make these things scalable in the future. 282 00:25:15,006 --> 00:25:25,790 Yeah, know, one of the, we were partners with an enterprise search company a years ago and we had a front row seat to a lot of implementations. 283 00:25:25,790 --> 00:25:41,277 In fact, we helped with some of the implementations and one of the challenges that we saw, their strategy was to recrawl and index the entire DM corpus, which yeah, which is, so we 284 00:25:41,277 --> 00:25:44,338 have one, we had one firm, this is just last year. 285 00:25:44,522 --> 00:25:49,442 Um, maybe Amla a hundred, maybe, maybe Amla 80. 286 00:25:49,442 --> 00:25:49,742 I don't know. 287 00:25:49,742 --> 00:26:02,742 They were about 800 attorneys ish and they were spending, was it 40, 20,000, $25,000 a month in Azure compute, um, to do this. 288 00:26:02,742 --> 00:26:05,602 And that did not include the licensing of the product itself. 289 00:26:05,602 --> 00:26:09,962 If I had to guess, I would say that was probably another, so double it. 290 00:26:09,962 --> 00:26:13,322 So close to a million dollars a year to do this. 291 00:26:13,322 --> 00:26:14,050 And 292 00:26:14,050 --> 00:26:15,421 What are the challenges with that? 293 00:26:15,421 --> 00:26:20,575 Because honestly, from where I said, it seems like killing a fly with a sledgehammer. 294 00:26:21,377 --> 00:26:23,518 What are your thoughts on that approach? 295 00:26:23,756 --> 00:26:30,570 Yeah, I think I've heard of people doing this and it's quite a bold approach. 296 00:26:31,091 --> 00:26:38,796 I'm going to be a bit biased here, obviously, because I work for iManage, but we have all your documents indexed. 297 00:26:39,937 --> 00:26:51,545 And to take a local copy of all of those documents or index them locally, again, potentially you can do it, but it's expensive. 298 00:26:51,785 --> 00:26:53,058 It's going to take time. 299 00:26:53,058 --> 00:26:57,409 There's reconciliation issues between you're maintaining two indexes over the same content. 300 00:26:57,409 --> 00:26:59,860 So how are going to make sure you keep them in sync? 301 00:26:59,860 --> 00:27:02,301 Probably risk issues around it as well. 302 00:27:02,941 --> 00:27:08,353 Might even be user experience issues as well when people come to access content for one system. 303 00:27:08,353 --> 00:27:12,141 I'm not saying it's impossible. 304 00:27:12,141 --> 00:27:13,824 I'm not saying it's necessarily a bad idea. 305 00:27:13,824 --> 00:27:20,096 What I'm saying is that it's quite expensive and I think it introduces a lot of complexity. 306 00:27:20,419 --> 00:27:25,086 So you need some really smart people if you're to get that right and a very, very compelling ROI. 307 00:27:25,662 --> 00:27:29,486 It is a heavy, heavy lift for sure. 308 00:27:29,486 --> 00:27:35,231 And yeah, I it's a, have found, we have taken a different approach. 309 00:27:35,231 --> 00:27:45,770 We, no longer partner in that capacity with them and we provide a federated experience with using Azure AI search. 310 00:27:45,770 --> 00:27:52,906 For example, we also do real time API access into the DMS and 311 00:27:52,942 --> 00:27:53,567 Mm. 312 00:27:53,567 --> 00:28:05,953 if we find that approach is a much more streamlined, less resource intensive and close to, we, we, we get pretty good results doing it. 313 00:28:05,953 --> 00:28:08,656 And a lot of firms are leaning in that direction now. 314 00:28:08,780 --> 00:28:09,650 Yeah. 315 00:28:09,770 --> 00:28:18,413 mean, of the conversations I've had with firms actually, we started talking about this stuff and I probed a little bit on it I was like, why do you want this capability? 316 00:28:18,413 --> 00:28:20,473 What's the demand behind this? 317 00:28:20,473 --> 00:28:25,210 And sometimes it's been things like, well, the lawyers just can't find their documents. 318 00:28:25,210 --> 00:28:26,495 They can't find where it is. 319 00:28:26,495 --> 00:28:29,836 And then we have a little chat about why that is. 320 00:28:29,836 --> 00:28:34,597 And they show me under the hood of how they're using their DMS. 321 00:28:34,657 --> 00:28:38,158 And we've managed to solve the issues by 322 00:28:38,318 --> 00:28:42,338 improving people's working practices in DMS. 323 00:28:42,538 --> 00:28:52,058 Because when you deploy iManage, I mean, it's the same with any DMS, I suppose, but you obviously have certain folders in each workspace or matter that's set up. 324 00:28:52,078 --> 00:28:56,298 And the firms I'm talking about here just have like two folders, documents and emails. 325 00:28:56,298 --> 00:29:00,958 And it means there's no consistency from one workspace or matter to the next. 326 00:29:00,958 --> 00:29:04,338 And people are saving things in a random haphazard function. 327 00:29:04,338 --> 00:29:07,014 On cross-border matters, they're saving them. 328 00:29:07,266 --> 00:29:14,049 one transaction in the New York library and another transaction in the London library, even though it's largely the same transaction with the same client. 329 00:29:14,049 --> 00:29:17,450 So it's kind of no wonder people can't find things. 330 00:29:17,650 --> 00:29:19,311 And there's two ways of solving that. 331 00:29:19,311 --> 00:29:29,095 Either you do the enterprise search route and you chuck everything into a search, or you try and solve the issue at source and you try and put in place some better working 332 00:29:29,095 --> 00:29:36,618 practices to help people categorize things a little bit better in their DMS so that they're not facing the problem in the first place. 333 00:29:36,768 --> 00:29:45,275 It's just interesting to see where there's a lot of stuff I do in my consultancy work where people come in with a solution in mind and we actually end up pivoting to something 334 00:29:45,275 --> 00:29:51,370 completely different that's culturally quite expensive, but financially less expensive. 335 00:29:51,540 --> 00:30:04,197 Yeah, you know, we see as well a lot of there's been so much consolidation, at least here in the U S how big firms have gotten big is largely through acquisition. 336 00:30:04,377 --> 00:30:14,182 And what that has led to is there has always been, I'm speaking in broad brushstrokes here, poor data hygiene in the DMS. 337 00:30:14,302 --> 00:30:16,963 Lawyers don't like to profile documents. 338 00:30:17,110 --> 00:30:21,446 They don't file, you know, follow naming conventions and 339 00:30:22,275 --> 00:30:35,323 so what, what we've seen is firms that have grown by acquisition, have one mess of a DMS and then they have three or four or five or six acquisitions with a diff, a mess of a 340 00:30:35,323 --> 00:30:39,106 different nature that they slam in together. 341 00:30:39,106 --> 00:30:51,146 And what you have is five, six different messes and things become really unwieldy and, um, untidy and finding things. 342 00:30:51,146 --> 00:30:56,566 You know, it's a, you know, when I think about a search index, it's like a house, right? 343 00:30:56,566 --> 00:31:10,226 It's a, you, the building of a house and the raw materials for that house are the inputs, the documents, the emails, and you know, the, the, how the house turns out is based on the 344 00:31:10,226 --> 00:31:12,256 architectural plans, right? 345 00:31:12,256 --> 00:31:19,038 Which is equivalent to the, um, you know, the technology that's providing 346 00:31:19,038 --> 00:31:25,942 these capabilities, it's a function of the input, the quality of the materials that were used to build the house. 347 00:31:25,942 --> 00:31:38,489 And then doing this again is like tearing the house down and rebuilding it somewhere else with proper architecture, which seems very inefficient to me. 348 00:31:38,489 --> 00:31:48,424 it sounds like, are you seeing the same thing in terms of the mess that is sometimes a law firm DMS? 349 00:31:48,526 --> 00:31:58,566 Oh yeah, yeah, I mean, there'll be some, there'll be some mergers and acquisitions that happened decades ago and I'll still see libraries that were named after the one of the 350 00:31:58,566 --> 00:32:00,146 firms that no longer exists. 351 00:32:00,146 --> 00:32:04,126 So it's kind of a problem that people don't really want to deal with. 352 00:32:04,126 --> 00:32:05,886 And I get that, you know, it's expensive. 353 00:32:05,886 --> 00:32:17,166 And I think a theme that comes up with a lot of these kinds of things in tech general is you can pull all this architecture in place, whether or not the attorneys will follow it 354 00:32:17,166 --> 00:32:17,666 or not. 355 00:32:17,666 --> 00:32:19,107 is another question. 356 00:32:20,428 --> 00:32:22,189 So you've got to have that in place. 357 00:32:22,189 --> 00:32:32,247 The one thing I would say is that I have actually noticed a real improvement on this stuff and firms wanting to get their house in order. 358 00:32:32,247 --> 00:32:44,970 And that I think is actually driven by people's belief that if they want to harness generative AI properly, then the data sets over which that stuff is run have to be 359 00:32:44,970 --> 00:32:49,451 or have to be clean, to be well organized and good quality content. 360 00:32:49,831 --> 00:32:59,952 I can tell you from some of the experiments we've done, we've got a feature that will basically do a natural retrieval, augmented generation knowledge search, basically, which 361 00:32:59,952 --> 00:33:01,264 I mean, that's a lot of jargon. 362 00:33:01,264 --> 00:33:03,450 So I better just break that down for people who don't know what that means. 363 00:33:03,450 --> 00:33:12,428 But it's basically where you type a question in, the system will identify documents relevant to the question, and then it will generate an answer to your question based on 364 00:33:12,428 --> 00:33:14,238 the documents identified. 365 00:33:14,380 --> 00:33:24,883 And if you start doing that against an entire DMS made out of tens of millions of documents, it's interesting what documents it brings back, which are things like email. 366 00:33:24,883 --> 00:33:35,026 If you're asking, for example, a question about bribery, it will start looking at things like email footers or offering memoranda with some mention of bribery in there, which 367 00:33:35,026 --> 00:33:42,058 aren't really authoritative sources about the question you ask, but nonetheless seem relevant to the question. 368 00:33:43,444 --> 00:33:54,882 So for me, it's pretty much proven based on the experimentation that we've done that you do need a good level of data governance around your content if you want to leverage some 369 00:33:54,882 --> 00:33:55,882 of this stuff. 370 00:33:56,210 --> 00:33:57,330 Interesting. 371 00:33:57,371 --> 00:33:59,272 So what do you think? 372 00:33:59,272 --> 00:34:11,659 There's been a lot of talk recently about foundational or frontier models for legal purposes versus legal specific models. 373 00:34:11,659 --> 00:34:23,602 And, you know, you and I chatted and I think this is true that it's really kind of hard to tell which is the right path until you build it and test it. 374 00:34:23,602 --> 00:34:24,562 Right. 375 00:34:24,824 --> 00:34:26,255 Where are you landing? 376 00:34:26,255 --> 00:34:30,890 you optimistic or skeptical on the need for legal specific models? 377 00:34:33,389 --> 00:34:35,049 It's a tough one, isn't it? 378 00:34:35,049 --> 00:34:48,649 If you go back to the way these models work, which is statistical next token prediction, you'd expect the embeddings graph or the vectors to kind of be more geared towards the 379 00:34:48,649 --> 00:34:50,769 dataset that they're based on. 380 00:34:50,769 --> 00:34:59,209 So it would predict its statistical next token prediction would be driven by, you know, a series of case law or legislation or whatever. 381 00:34:59,429 --> 00:35:02,304 So you'd expect that the output might be a bit more relevant. 382 00:35:02,304 --> 00:35:04,494 It might be bit more interesting. 383 00:35:04,775 --> 00:35:13,537 the issues we face in the use of these things are often related to the risk of hallucination. 384 00:35:13,697 --> 00:35:26,901 And I am personally unconvinced that changing the data set will really change that because these things are not designed to produce accurate output. 385 00:35:26,901 --> 00:35:28,921 That is not a test criteria. 386 00:35:28,921 --> 00:35:31,422 They're designed to statistically predict 387 00:35:31,422 --> 00:35:32,782 next token. 388 00:35:33,163 --> 00:35:43,265 And I don't understand how changing the underlying data set can solve the problem of hallucinations, which seems to be plaguing everybody. 389 00:35:43,685 --> 00:35:54,468 The other issue I throw in here, which I talk about a little bit in an article I wrote recently, which is that if our hope is that these things can become autonomous legal 390 00:35:54,468 --> 00:36:00,650 advisors, and we're trying to improve accuracy, I've touched on this a little bit already, but 391 00:36:00,726 --> 00:36:04,908 I don't necessarily know whether those are the use cases we should be shooting out right now. 392 00:36:04,908 --> 00:36:06,448 Like first of all, because they're hard. 393 00:36:06,448 --> 00:36:09,850 But secondly, don't know whether that's really what we want. 394 00:36:09,850 --> 00:36:15,652 I don't know if we want people to be reading and engaging with the primary materials less. 395 00:36:15,652 --> 00:36:19,494 don't, people always talk about this blank page problem. 396 00:36:19,494 --> 00:36:24,056 And this is an area a lot of people listening will disagree with me on, but I love the blank page problem. 397 00:36:24,056 --> 00:36:25,297 That's what makes me think. 398 00:36:25,297 --> 00:36:29,248 Like if I have a blank page and have to write something, 399 00:36:29,408 --> 00:36:38,285 I've got to structure my thoughts in my head and then I write them down and that flexes my brain muscles quite significantly and it makes me think about things that I wouldn't have 400 00:36:38,285 --> 00:36:42,227 thought about or wouldn't have realized had I not gone through that problem. 401 00:36:42,308 --> 00:36:52,936 And people can say, well, the LLM will solve that blank page problem for you by producing a first draft that's highly accurate because it's trained on legal specific material. 402 00:36:52,936 --> 00:36:56,298 But I'm thinking, well, great, but that means that 403 00:36:56,298 --> 00:37:03,942 my thought process is then confined to the words that are already written on the page, rather than me wrangle with issues that the LLM didn't mention. 404 00:37:03,942 --> 00:37:06,973 And I do worry about the impact that will have on people. 405 00:37:06,973 --> 00:37:08,744 But that's just my own personal preference. 406 00:37:08,744 --> 00:37:13,806 I'm quite open also to the idea that my mind works differently from a lot of other people's minds. 407 00:37:13,806 --> 00:37:19,118 So yeah, think I'm interested to see what I deliver. 408 00:37:19,118 --> 00:37:22,700 I'm not kind of saying that it's a waste of time at all. 409 00:37:22,700 --> 00:37:26,924 I'm probably more veering towards the skeptical end of the spectrum, but I'm watching it really closely. 410 00:37:26,924 --> 00:37:29,637 I think there might be some interesting stuff that comes out of it. 411 00:37:29,637 --> 00:37:34,932 But as usual, we need to really be thinking about why we're doing these things and what the use cases are. 412 00:37:35,630 --> 00:37:40,246 The white screen of death is the new, um, the new terminology, right? 413 00:37:40,246 --> 00:37:42,099 Instead of the blue screen of death. 414 00:37:42,099 --> 00:37:43,119 Yeah. 415 00:37:43,180 --> 00:37:47,385 Which it's better. 416 00:37:47,385 --> 00:37:49,068 It's better than the blue screen of death. 417 00:37:49,068 --> 00:37:54,354 If you've ever, if you're a windows user and you've seen that it's usually not a good day. 418 00:37:54,464 --> 00:37:55,391 Nope. 419 00:37:56,342 --> 00:38:03,332 Um, so do, do we need to build these models before we can evaluate the need for them? 420 00:38:03,332 --> 00:38:10,982 Or is it possible to test this hypothesis without investing hundreds of millions of dollars and actually building them? 421 00:38:13,243 --> 00:38:14,543 Do you know what? 422 00:38:14,643 --> 00:38:16,902 I don't know the answer to that question. 423 00:38:16,902 --> 00:38:21,725 I feel like I need to be, I feel like I'm at risk of saying, no, it's a waste of time. 424 00:38:21,725 --> 00:38:22,725 Don't do it. 425 00:38:22,725 --> 00:38:29,497 And then someone says, who knows far more about me than this kind of stuff will say, actually, that could be some interesting things. 426 00:38:29,497 --> 00:38:30,888 You're not quite right. 427 00:38:30,888 --> 00:38:35,599 When you talked about hallucinations and next word prediction, I am not a data scientist. 428 00:38:35,599 --> 00:38:38,600 Like I learned all this stuff just by reading on the internet. 429 00:38:38,600 --> 00:38:39,950 So I'm 430 00:38:39,950 --> 00:38:42,620 I think people should continue doing this work. 431 00:38:42,620 --> 00:38:51,830 think it's all these avenues need exploring because if they can pull it off and it can do things that we can't even think about now, I'm all for it. 432 00:38:51,830 --> 00:39:02,430 And by the way, I do think also that these problems we're having around like hallucinations and generally all of the kind of drawbacks of LLMs that we're seeing, it's 433 00:39:02,430 --> 00:39:09,358 highly likely that in the next, take a long-term view on it, 10, 20 years, like my own prediction is that in 434 00:39:09,358 --> 00:39:11,880 15, 20 years, maybe shorter, I don't know. 435 00:39:11,880 --> 00:39:15,953 Someone will come up with something else that is completely different from an LLM. 436 00:39:15,953 --> 00:39:19,004 It's not next token prediction, not a thing. 437 00:39:19,645 --> 00:39:23,208 Maybe it can, I don't know how it works, but maybe it can just produce accurate things. 438 00:39:23,208 --> 00:39:25,029 I've quite a strong belief that is going to happen. 439 00:39:25,029 --> 00:39:34,576 So it seems to me to be highly valuable to start engaging in some of these thought experiments about what happens when we get there so that we're prepared when we get there. 440 00:39:34,576 --> 00:39:38,274 And I think the current technology might start those, but I don't think it's quite. 441 00:39:38,274 --> 00:39:41,845 going to live up to the promise people maybe thought it had a couple of years ago. 442 00:39:42,260 --> 00:39:52,173 Yeah, I completely agree that this is very unlikely the last iteration of architecture for AI. 443 00:39:52,268 --> 00:39:53,779 Yeah, completely. 444 00:39:53,879 --> 00:40:02,847 I think that I do feel like as long as LLMs are still LLMs and they work the same way, I do think we're going to start to hit a ceiling. 445 00:40:02,847 --> 00:40:11,844 And some of these agentic things like feeding the output back into itself and predicting the next step is like an interesting development, but it's not really changing that much. 446 00:40:11,844 --> 00:40:14,706 It's just the same operational methodology. 447 00:40:14,706 --> 00:40:21,822 Until that changes, I suspect we probably won't get to this holy grail of accurate output. 448 00:40:21,982 --> 00:40:34,809 Yeah, there's a lot of talk lately about scaling laws and I heard an interesting podcast this morning at the gym where they were talking about, you know, there's really kind of 449 00:40:34,809 --> 00:40:39,732 three levers to pull for innovation in with LLMs. 450 00:40:39,732 --> 00:40:45,095 It's, you know, the amount of compute, basically the number of Nvidia chips that you can throw at it. 451 00:40:45,095 --> 00:40:51,542 It's the data, which, you know, the training data, which is finite and 452 00:40:51,542 --> 00:40:58,187 Um, so, so we're the number of chips, um, they're now training on synthetic data as well. 453 00:40:58,448 --> 00:41:04,273 And then the last lever is the algorithm itself. 454 00:41:04,273 --> 00:41:16,164 And there, that is not, there isn't a finite, number of ways there's, there's infinite potential paths to improve. 455 00:41:16,164 --> 00:41:17,125 would imagine. 456 00:41:17,125 --> 00:41:18,986 So it feels like. 457 00:41:19,144 --> 00:41:30,069 Yeah, it's probably not going to be throwing more, you know, GPUs or training data at the same architecture that's gonna get us to the next level. 458 00:41:30,069 --> 00:41:37,628 It's, it's probably going to be something in the tech, not in the tech, um, which seems logical. 459 00:41:38,363 --> 00:41:47,771 I like this stuff, like my understanding was that it actually came from translation use cases and someone was kind of tinkering around with it and found that Transformers 460 00:41:47,771 --> 00:41:50,387 actually could have these other potential applications. 461 00:41:50,387 --> 00:41:54,197 So I suspect it would be something like that that we don't even know about right now. 462 00:41:54,197 --> 00:41:55,948 And I don't doubt that it will happen. 463 00:41:55,948 --> 00:42:02,654 And I think people will be a bit more interested when it does, because I think this stuff, the LLMs was such a step forward. 464 00:42:02,922 --> 00:42:08,558 I hope people don't come away from this thinking I'm like some sort of AI skeptic or deny or whatever. 465 00:42:08,558 --> 00:42:09,749 I think the stuff is incredible. 466 00:42:09,749 --> 00:42:10,750 I really do. 467 00:42:10,750 --> 00:42:22,903 It's just that the bias I was talking about earlier means that I'm always slightly trying to dampen hype on things because I don't know if inflating expectations is necessarily the 468 00:42:22,903 --> 00:42:26,636 best thing when they can't deliver on those expectations. 469 00:42:26,846 --> 00:42:28,447 Yeah, no, we share that view. 470 00:42:28,447 --> 00:42:36,970 I use AI probably 10 times a day, very, very frequent for all kinds of stuff. 471 00:42:36,970 --> 00:42:50,494 So I don't, I don't, I'm very bullish on the future of AI, but I'm, I'm, I'm, I'm very bearish on the marketing messaging coming out right now aligning with capabilities. 472 00:42:50,956 --> 00:42:52,847 Yeah, no, agreed, agreed. 473 00:42:52,847 --> 00:43:05,136 And in the moment, it seems to be there's a lot of stuff around the agentic AI as well, like which you could have another conversation and actually the content around the current 474 00:43:05,136 --> 00:43:07,718 use of LLM is all about the quality of the content. 475 00:43:07,718 --> 00:43:17,484 And then with agentic AI, it's probably the quality of the process and the needs to define the process and stop one person doing the same process in a completely different way from 476 00:43:17,484 --> 00:43:18,595 another person. 477 00:43:18,595 --> 00:43:20,350 It all comes back to this kind of 478 00:43:20,350 --> 00:43:25,388 human hygiene element and kind of putting a bit of control around things. 479 00:43:25,610 --> 00:43:38,150 Yeah, I had a Aaron Amadea from relativity on a few episodes back and we talked about agentic AI, super smart dude, PhD in math and I was an undergrad in math and like barely 480 00:43:38,150 --> 00:43:39,230 got through it. 481 00:43:39,230 --> 00:43:54,782 So, uh, he, he, yeah, it is, uh, you know, I was, I'm really good at like, um, I was really good at calculus and, um, differential equations and matrix theory. 482 00:43:54,782 --> 00:44:00,777 where it went off the rails with me was advanced calculus where you have to go back and prove everything in calculus. 483 00:44:00,777 --> 00:44:11,537 So proofs are extremely difficult and really require a lot of abstract thinking and they, they kicked my ass honestly. 484 00:44:11,537 --> 00:44:13,538 So, um, 485 00:44:13,875 --> 00:44:15,516 That is good though. 486 00:44:15,556 --> 00:44:22,339 I really enjoy, like part of the reason, a lot of people say I just, I'm a kind of person that naturally disagrees with things. 487 00:44:22,339 --> 00:44:25,040 And I guess that might be true. 488 00:44:25,040 --> 00:44:28,101 And the kind of the reason I do it is because I want people to tell me I'm wrong. 489 00:44:28,101 --> 00:44:29,202 is I learned. 490 00:44:29,202 --> 00:44:33,684 I think in this age, the one thing I've learned over the last couple of years is don't be precious around how much you know. 491 00:44:33,684 --> 00:44:39,646 And if someone smarter than you tells you that you're not right, go ahead and accept that and learn and move on. 492 00:44:40,170 --> 00:44:41,130 100%. 493 00:44:41,130 --> 00:44:49,850 Yeah, it's understanding what you don't know is as good and valuable as understanding what you do. 494 00:44:49,950 --> 00:44:53,770 All right, last question for you, because I know we're running out of time here. 495 00:44:53,770 --> 00:45:07,810 What do you think about how the capital markets, I mean, we're seeing so much money flow into, if you were to stack rank all the industries where legal has been predicted to be 496 00:45:07,810 --> 00:45:09,150 transformative, 497 00:45:09,974 --> 00:45:17,296 Um, law, you know, legal is at the top of most lists, um, simply because there hasn't been a innovate. 498 00:45:17,296 --> 00:45:23,298 hasn't been a huge innovation around the concept of language like LLMs ever. 499 00:45:23,298 --> 00:45:23,708 Right. 500 00:45:23,708 --> 00:45:30,880 You know, you could argue maybe search, but, um, I think LLMs are just completely transformative with potential. 501 00:45:30,880 --> 00:45:38,248 What do you, do you think there's, uh, you know, is it problematic that we're seeing all this money flow in and how that's impacting 502 00:45:38,248 --> 00:45:42,461 marketing messaging and people's experience with the technology? 503 00:45:42,461 --> 00:45:43,844 What are your thoughts on that? 504 00:45:43,970 --> 00:45:51,933 Well, as somebody that's passionate about driving change in the legal industry, for me, it's always good news that money's flowing into this kind of investment. 505 00:45:51,953 --> 00:45:58,986 I'll never say it's bad news ever for companies trying to improve how lawyers work to have continued investment. 506 00:45:58,986 --> 00:46:02,070 So broadly, think it's a great thing. 507 00:46:02,070 --> 00:46:06,900 I think though, you mentioned that legal's top of these use cases. 508 00:46:06,900 --> 00:46:12,520 And I think that mostly that's because when people think about lawyers, 509 00:46:12,520 --> 00:46:19,816 first thing that tends to come to their head is long documents, long written words, long briefs, that kind of thing. 510 00:46:21,137 --> 00:46:24,340 And they kind of say, well, lawyers write a lot, don't they? 511 00:46:24,340 --> 00:46:27,322 And LLMs also write a lot. 512 00:46:27,322 --> 00:46:31,486 therefore, I'm going to conclude that lawyers are going to get replaced by LLMs. 513 00:46:31,486 --> 00:46:37,020 And therefore, I'm going to invest an awful lot of money in LLMs, because I think it could be disruptive to the legal industry. 514 00:46:37,020 --> 00:46:42,134 And I think it could have some pretty far reaching impacts on the 515 00:46:42,134 --> 00:46:43,474 legal industry. 516 00:46:43,615 --> 00:46:53,020 The one thing I would say though, is that I don't think it's necessarily the right approach to go in with such a superficial understanding of what actually the legal 517 00:46:53,020 --> 00:46:54,461 industry is about. 518 00:46:55,542 --> 00:47:04,127 Again, one of the other things when I speak to you to understand my bias is towards my prior career as a large big law lawyer. 519 00:47:04,127 --> 00:47:09,750 But if I think about the kind of work I did, and my practice was half contentious and half transactional, 520 00:47:09,934 --> 00:47:14,324 much time did I actually spend writing long documents? 521 00:47:14,324 --> 00:47:19,994 How much time did I spend standing up in court, drafting briefs, doing this kind of stuff? 522 00:47:19,994 --> 00:47:23,194 Well, I think back to it, very little really. 523 00:47:23,194 --> 00:47:31,914 Most of it was around managing my inbox, to be honest, trying to project manage, especially when you're junior, a lot of it is project management. 524 00:47:31,914 --> 00:47:37,570 And then you've got the whole ream of menial tasks like producing closing books or 525 00:47:37,570 --> 00:47:39,331 doing these kinds of things. 526 00:47:39,331 --> 00:47:47,773 And I think that it's great that everyone's bought into the idea of disrupting legal industry through generative AI. 527 00:47:47,773 --> 00:47:53,774 But I really wish they wouldn't confine their mission to a method. 528 00:47:53,774 --> 00:47:57,355 By that, mean, don't just look at generative AI. 529 00:47:57,455 --> 00:48:01,196 Try and get bought into the idea of disrupting the legal industry. 530 00:48:01,196 --> 00:48:07,078 But don't just focus on generative AI, because there are so many things lawyers do that are completely backwards. 531 00:48:07,566 --> 00:48:10,888 could and should have been fixed 20 years ago. 532 00:48:11,969 --> 00:48:16,953 So things like that I would kind of encourage people to focus on. 533 00:48:16,953 --> 00:48:22,517 I suspect actually what might happen is that when these investments were made, people had really high hopes. 534 00:48:22,517 --> 00:48:26,639 They thought lawyers were going to get replaced by machines that could draft all their documents for them. 535 00:48:26,639 --> 00:48:31,173 I suspect over time people realize that probably is not actually quite the case. 536 00:48:31,173 --> 00:48:33,785 Like we'll be in some areas, but not across the board. 537 00:48:33,785 --> 00:48:35,926 And I suspect that they probably will. 538 00:48:36,326 --> 00:48:43,114 kind of pivot a little bit towards focusing more on the realities of how people work and intangible use cases. 539 00:48:43,466 --> 00:48:44,471 So yeah, that's my view. 540 00:48:44,471 --> 00:48:46,377 I'm fully in favour of this investment. 541 00:48:46,377 --> 00:48:47,300 I think it's great. 542 00:48:47,300 --> 00:48:50,764 It's getting people excited about technology who weren't excited about it before. 543 00:48:50,764 --> 00:48:53,056 And those are all good things in my view. 544 00:48:53,074 --> 00:48:53,635 Yeah. 545 00:48:53,635 --> 00:49:03,604 I mean, if you look at Harvey, for example, and their cap table, I mean, they've got open AI, they've got Google ventures, they have a 16 Z, which is Andreessen Horowitz. 546 00:49:03,905 --> 00:49:08,418 You've not seen interest at that level in this, in this area ever. 547 00:49:08,418 --> 00:49:11,039 Yeah, I mean, it's great. 548 00:49:11,039 --> 00:49:11,749 It really is great. 549 00:49:11,749 --> 00:49:14,550 And it'll be interesting to see where Harvey goes. 550 00:49:14,550 --> 00:49:18,642 know, I've been really impressed by what I've seen from it. 551 00:49:18,642 --> 00:49:21,323 It's good that they're being a bit more open about what they're building. 552 00:49:21,323 --> 00:49:26,655 And there's lots of other companies in the mix as well that have good kind of pedigree investors as well. 553 00:49:26,655 --> 00:49:29,546 So I think all in all, it's a very, very good thing. 554 00:49:29,566 --> 00:49:36,970 Yeah, I'm actually really happy to see the pivot in Harvey's PR strategy and opening up more. 555 00:49:36,970 --> 00:49:44,484 And I actually met Winston for the first time at TLTF last week and yeah, very sociable guy. 556 00:49:44,484 --> 00:49:51,878 And, um, I was impressed and he needs to get out more so people can engage with him and get to know him. 557 00:49:51,878 --> 00:50:02,266 this is the thing, like, because I think that the problem with that kind of marketing is that I take Figma as an example, right, the UX design program, like people, historically, 558 00:50:02,266 --> 00:50:05,588 people use Sketch or Photoshop, and then Figma came out and it was better. 559 00:50:05,588 --> 00:50:07,297 And so everyone started using Figma. 560 00:50:07,297 --> 00:50:11,612 And I think a lot of these these companies are entering the legal industry afresh. 561 00:50:11,612 --> 00:50:19,800 It's a lesson I learned, actually, which is that just because you are really passionate about technology, that does not mean that all the lawyers that 562 00:50:19,800 --> 00:50:28,734 have billable hour targets and are really, really super busy and don't have time to get lunch, let alone try a new piece of technology, are as excited about it as you are. 563 00:50:28,734 --> 00:50:37,758 So it does not necessarily mean that just because your tech's really good and exciting, just because you can help people do things quicker, that does not necessarily translate 564 00:50:37,818 --> 00:50:40,439 into value and usage of tools. 565 00:50:40,439 --> 00:50:44,781 And that's a lesson I learned really badly as somebody that is passionate about this stuff. 566 00:50:44,781 --> 00:50:46,764 Not everyone shares my passion. 567 00:50:46,764 --> 00:50:52,359 So I had to develop a whole skill set around how I convince people to give up their time to use these things. 568 00:50:52,359 --> 00:50:56,362 And I don't think AI has changed that, if I'm completely honest with you. 569 00:50:56,372 --> 00:50:58,284 mean, I sell intranets and extranets. 570 00:50:58,284 --> 00:50:59,715 You think that excites people? 571 00:50:59,715 --> 00:51:02,146 Yeah. 572 00:51:03,408 --> 00:51:07,286 Well, it doesn't most people, but yeah, we try to get out there. 573 00:51:07,286 --> 00:51:10,443 I did a real quick plug for a previous episode. 574 00:51:10,443 --> 00:51:15,988 It just released what is today, but by the time this episode airs, it'll be several weeks. 575 00:51:15,988 --> 00:51:20,191 It's with Alex Sue, who is the chief revenue officer at Latitude. 576 00:51:20,191 --> 00:51:26,046 And we talked about, he wrote a really good article that inspired the episode about how difficult 577 00:51:26,046 --> 00:51:27,668 legal is to disrupt. 578 00:51:27,668 --> 00:51:33,255 is extremely difficult and history has proven this over and over and over again. 579 00:51:33,255 --> 00:51:41,594 So for these big Silicon Valley investors that think they're just going to write some checks and all the dominoes are going to fall in their direction, they're sadly mistaken. 580 00:51:41,594 --> 00:51:43,190 It's going to be real work. 581 00:51:43,190 --> 00:51:43,990 absolutely. 582 00:51:43,990 --> 00:51:52,870 And I've got friends who've been at firms where they've had these AI tools rolled out, know, I name any names of the tools, but their reaction to me is kind of like, I don't 583 00:51:52,870 --> 00:51:53,790 know what to use this for. 584 00:51:53,790 --> 00:51:58,050 Like this is people tell me it's AI, but I kind of what so what what do do with it? 585 00:51:58,050 --> 00:52:01,370 And that's why that barrier must be bridged. 586 00:52:01,370 --> 00:52:04,470 And lots of people have different views on this whole use case thing. 587 00:52:04,470 --> 00:52:06,562 And lots of people are we shouldn't tell 588 00:52:06,562 --> 00:52:08,753 lawyers about use cases, know, let them experiment. 589 00:52:08,753 --> 00:52:17,827 We don't want to confine them too narrowly, but you do have to do it a little bit because otherwise people who have, you know, only two or three minutes to play around with 590 00:52:17,827 --> 00:52:20,418 something in a day, they're just not going to use it. 591 00:52:20,418 --> 00:52:22,409 So you do have to do a little bit. 592 00:52:23,690 --> 00:52:24,500 Yeah. 593 00:52:24,621 --> 00:52:27,123 Well, this has been a great conversation. 594 00:52:27,123 --> 00:52:34,590 Before we wrap up, how do people find out more about you, iManage, what's the best way to connect? 595 00:52:35,180 --> 00:52:39,634 Well, any legal technology conference, you'll see, I manage there. 596 00:52:39,634 --> 00:52:43,157 If you're lucky or unlucky, depending how you look at it, I might also be there. 597 00:52:43,157 --> 00:52:46,619 So come and speak to me, come and introduce yourself. 598 00:52:46,619 --> 00:52:50,623 Otherwise I'm fairly active on LinkedIn and I post a lot of stuff on there. 599 00:52:50,623 --> 00:52:55,056 And also Blue Sky, which is something that more and more people are coming on now. 600 00:52:55,056 --> 00:52:57,719 So have a look at me on there or reach out to me. 601 00:52:57,719 --> 00:53:02,363 always, my mantra is I never say no to anyone on LinkedIn where they message me. 602 00:53:02,363 --> 00:53:04,584 So yeah, always offer a chat. 603 00:53:04,820 --> 00:53:05,812 Good stuff. 604 00:53:05,812 --> 00:53:10,309 All right, well, I appreciate your time and we went a little bit over, but it was a great conversation. 605 00:53:10,309 --> 00:53:17,370 So, I'll look for you at the next tech conference and hopefully we can connect in person. 606 00:53:17,398 --> 00:53:17,908 Nice one. 607 00:53:17,908 --> 00:53:19,060 Thanks, Ted. 608 00:53:19,060 --> 00:53:24,500 Some really interesting stuff there and look forward to quizzing you on some calculus in the future. 609 00:53:24,500 --> 00:53:24,990 There we go. 610 00:53:24,990 --> 00:53:26,682 I'm a little rusty, but I'll do my best. 611 00:53:26,682 --> 00:53:28,492 All right. 612 00:53:28,492 --> 00:53:29,304 Have a good afternoon. 613 00:53:29,304 --> 00:53:31,075 All right. 614 00:53:31,075 --> 00:53:32,095 Take care. 00:00:08,027 Jack, how are you this morning or I guess afternoon where you are? 2 00:00:08,075 --> 00:00:08,626 Yeah, I'm good. 3 00:00:08,626 --> 00:00:09,136 Thanks, Ted. 4 00:00:09,136 --> 00:00:13,760 Yeah, afternoon, but it's certainly getting quite dark, so it feels like the evening now. 5 00:00:13,854 --> 00:00:15,075 Gotcha. 6 00:00:15,235 --> 00:00:21,141 So, well, first of all, I really appreciate you taking the time to have this chat. 7 00:00:21,221 --> 00:00:31,631 I really enjoy your posts on LinkedIn and feel that you like to push a little bit and challenge and that's, those make for the best conversations. 8 00:00:31,631 --> 00:00:34,734 So I'm looking forward to our talk here this morning. 9 00:00:34,818 --> 00:00:37,846 Yeah, thank you and likewise and thanks for inviting me. 10 00:00:38,022 --> 00:00:39,104 Absolutely. 11 00:00:39,104 --> 00:00:44,313 So your background, you were an attorney at Freshfields. 12 00:00:44,313 --> 00:00:48,340 You're now at I-Manage in a knowledge consulting role. 13 00:00:48,340 --> 00:00:52,206 Tell us a little bit about who you are, what you do and where you do it. 14 00:00:53,366 --> 00:01:04,864 Yeah, so it's pretty hard to describe what I do, to be honest, because ever since I left private practice as a lawyer, I used to be a bankruptcy lawyer, did that for a few years. 15 00:01:04,864 --> 00:01:08,737 But ever since I've left that, it's been kind of hard to describe what I actually do. 16 00:01:08,737 --> 00:01:12,238 And the reality is that I have my fingers in all sorts of pies. 17 00:01:12,499 --> 00:01:16,452 My official role that I manage is to, as you say, help with knowledge consulting. 18 00:01:16,452 --> 00:01:23,316 So that's all things around cleaning up data within firms so that people are 19 00:01:23,514 --> 00:01:28,735 of finding good examples of things rather than bad drafts of things. 20 00:01:29,056 --> 00:01:41,369 Anything from that to taxonomies, metadata schemas, all the way through to the cultural aspects of knowledge management in law firms and how you can incentivize people to partake 21 00:01:41,369 --> 00:01:43,319 in knowledge sharing. 22 00:01:43,340 --> 00:01:51,842 So I do a lot of that stuff, but also like to think I've influenced a little bit the strategy of the company and the product side of things as well. 23 00:01:52,120 --> 00:01:53,432 There's something I'm interested in. 24 00:01:53,432 --> 00:01:56,737 I'll have a go at it and people can tell me to go away. 25 00:01:57,540 --> 00:02:00,034 I like to get involved with a lot of things. 26 00:02:00,034 --> 00:02:03,860 My general mission being just to improve how lawyers work really. 27 00:02:04,372 --> 00:02:05,243 Good stuff. 28 00:02:05,243 --> 00:02:09,746 And that's very relevant to the agenda that we carved out for today. 29 00:02:09,746 --> 00:02:23,908 But before we get into that, you and I were having a conversation earlier and we're talking a little bit about the episode with Damien Real and the concept of reasoning and 30 00:02:23,908 --> 00:02:27,841 which is a very hot topic and we're seeing more. 31 00:02:27,841 --> 00:02:30,263 We're seeing the word more and more. 32 00:02:30,484 --> 00:02:33,428 I just read Gemini 2's announcement. 33 00:02:33,428 --> 00:02:39,480 today and reasoning, the word reason and reasoning showed up multiple times. 34 00:02:39,480 --> 00:02:43,161 We hear about O-one and reasoning and chain of thought. 35 00:02:43,441 --> 00:02:48,903 And it's a really interesting topic to me because I think it matters. 36 00:02:48,903 --> 00:02:53,004 And I know a lot of people don't, whether it's truly reasoning or not. 37 00:02:53,004 --> 00:02:56,085 And what is your position on the topic? 38 00:02:56,085 --> 00:02:58,566 Do you believe that LLMs can reason? 39 00:03:00,035 --> 00:03:00,824 Wow. 40 00:03:02,774 --> 00:03:06,776 I don't think that is the most important question. 41 00:03:06,996 --> 00:03:15,259 I think that when we're looking at any technology, the question is always, what value can it add? 42 00:03:15,320 --> 00:03:25,084 And in the case of AI, it might be that AI can do some great things and achieve some great outcomes. 43 00:03:25,404 --> 00:03:30,106 And when we're looking at those things, we've got to always compare it with 44 00:03:30,636 --> 00:03:33,729 what the process looks like when humans are doing that same thing. 45 00:03:33,729 --> 00:03:39,934 You always got to benchmark it, both in terms of like efficiency, speed, but also quality. 46 00:03:39,934 --> 00:03:43,997 And one thing that I think people often miss is what you lose in the process. 47 00:03:44,077 --> 00:03:55,026 The example I always cite here is, let's do a quick thought experiment and let's say LLMs can produce 100 % accurate legal research memos with no hallucinations. 48 00:03:55,026 --> 00:03:56,907 Every citation is correct. 49 00:03:57,609 --> 00:03:59,790 Should we, does that mean we should use? 50 00:03:59,822 --> 00:04:03,022 AI to produce all our legal research memos? 51 00:04:03,022 --> 00:04:09,262 Some people might say yes and think I'm an idiot for even raising that question, but actually my answer is not yes. 52 00:04:09,262 --> 00:04:14,962 Because for me, the real question is, okay, what's the value of a legal research memo? 53 00:04:14,962 --> 00:04:18,182 Is it the words on the page of the research memo? 54 00:04:18,202 --> 00:04:22,342 If it is, then yeah, go for the AI that's 100 % accurate because it can do it quicker. 55 00:04:22,342 --> 00:04:26,818 But in my experience as a lawyer, clients don't really want a legal research memo. 56 00:04:26,818 --> 00:04:30,601 whenever I sent one of those out to clients, never really read it. 57 00:04:30,601 --> 00:04:40,568 The purpose of the legal research memo was to make sure that I had understood the case law and all the legislation so that we could then have a discussion about it so that the 58 00:04:40,568 --> 00:04:42,670 client could properly kick their tires. 59 00:04:42,670 --> 00:04:51,135 And fundamentally, the legal research memo really acted as evidence that we knew our stuff and that we are advising them on the best possible route. 60 00:04:51,476 --> 00:04:54,798 So when it comes to reasoning, 61 00:04:54,926 --> 00:05:04,811 I think my main answer to that question is I don't really care because for me the process is less important than kind of appraising all the pros and cons. 62 00:05:04,811 --> 00:05:09,406 But I know that you have a slightly different view, which I think I may even agree with. 63 00:05:09,406 --> 00:05:10,046 Yeah. 64 00:05:10,046 --> 00:05:22,093 Well, the reason I think it matters is because ultimately it's less about the LLM's ability to reason and more about their ability to comprehend, which is a first step in 65 00:05:22,093 --> 00:05:23,173 reasoning. 66 00:05:23,194 --> 00:05:38,236 So, you know, leveraging first principles, which is breaking down a problem to its most basic components and reasoning your way to a solution requires comprehension and 67 00:05:38,236 --> 00:05:44,030 at least in the traditional, in the human metaphor, right? 68 00:05:44,070 --> 00:05:50,674 Someone can't reason their way to a solution of a problem without understanding the problem, unless it's just pure luck. 69 00:05:51,035 --> 00:06:02,122 So the reason I think it's really important in terms of comprehension in a legal context is it really dictates what use cases become viable and not. 70 00:06:02,823 --> 00:06:07,422 I think it's a good guide for us to figure out 71 00:06:07,422 --> 00:06:11,207 where to apply the technology and where to stay away from. 72 00:06:11,416 --> 00:06:12,266 Yeah. 73 00:06:13,087 --> 00:06:18,430 Well, I think that I do actually agree with all of that. 74 00:06:19,591 --> 00:06:24,585 when ChatGPT came out, what was it, three years ago? 75 00:06:24,585 --> 00:06:26,225 Is it three years or two years? 76 00:06:26,225 --> 00:06:26,565 Right. 77 00:06:26,565 --> 00:06:27,655 Feels like it's been here forever. 78 00:06:27,655 --> 00:06:38,162 Anyway, I remember all the discussion was about what are the use cases and loads of people jumped to things like the obvious ones like drafting contracts and 79 00:06:38,658 --> 00:06:48,002 drafting research memos, a lot of people said things like blue booking, which I'll admit is not something that's familiar to me because I'm not a US attorney, but all these sort 80 00:06:48,002 --> 00:06:50,963 of weird and wonderful use cases came out of the woodwork. 81 00:06:50,963 --> 00:07:00,667 And for me, I think when you're appraising the ability of any technology to meet a given use case, having an understanding of how the technology works is pretty important. 82 00:07:00,927 --> 00:07:08,044 And I read a really good article year and a half ago by Stephen Wolfram, which is kind of very 83 00:07:08,044 --> 00:07:12,338 well-known article about how chat GPT works and what it's doing. 84 00:07:12,338 --> 00:07:19,725 And what I took from that 18 months ago was it's a statistical next word predictor. 85 00:07:19,825 --> 00:07:23,488 And I don't mean that to say, it's just predicting the next word. 86 00:07:23,489 --> 00:07:29,234 The thing that's magic about this is the fact that a next word predictor can produce such amazing outputs. 87 00:07:29,435 --> 00:07:33,282 And I don't think humans are next word predictors. 88 00:07:33,282 --> 00:07:41,194 I'm not saying that because to diminish the value of next word predictors, there are some situations where that would be a really, really useful thing to have, like spotting 89 00:07:41,194 --> 00:07:43,545 patterns over large amounts of information. 90 00:07:43,545 --> 00:07:50,167 But I exercises that require emotion, experience, principles, I think is less good at that. 91 00:07:50,167 --> 00:07:56,358 So I think to the extent you're appraising use cases, I think it's important to understand how these things work. 92 00:07:56,358 --> 00:07:59,829 And if you're going to push me to say, they reason or do they not reason? 93 00:08:00,229 --> 00:08:02,978 I'm pretty clearly in the camp of they don't reason, but 94 00:08:02,978 --> 00:08:08,859 The reason I say I don't care is because people get bogged down a little bit in, what does reasoning even mean? 95 00:08:08,999 --> 00:08:20,902 The thing that I always guard against about on this, I will say this is a bias, is that I am naturally very skeptical of anybody who tries to hype up any technology as being game 96 00:08:20,902 --> 00:08:21,302 changing. 97 00:08:21,302 --> 00:08:23,543 I always challenge people on that. 98 00:08:23,543 --> 00:08:32,286 And for me, the reason I don't like this whole reasoning thing is it seems to me people use that term to try and anthropomorphize the technology and to 99 00:08:32,286 --> 00:08:36,108 make everyone think of the future and robots and humans being out of jobs. 100 00:08:36,108 --> 00:08:38,329 And I think that is complete nonsense. 101 00:08:38,329 --> 00:08:47,624 So any language that gives that potentially could give a misleading capability, misleading impression of a technology capability, I generally try and avoid. 102 00:08:47,624 --> 00:08:56,058 that is a that's a bias I've had just through being in a position of buying technology that was promised to be amazing and turned out not to be quite so amazing. 103 00:08:56,830 --> 00:08:59,551 Yeah, there's a lot of that in the marketing. 104 00:08:59,551 --> 00:09:08,730 I think today the gap between the marketing message and the true capabilities in this space is as get as wise as it's ever been. 105 00:09:08,730 --> 00:09:20,060 Um, I thought that the, know, there's been a lot of controversy around the Stanford paper and I, I read the study multiple times and I actually got a lot of value from it. 106 00:09:20,060 --> 00:09:25,552 I do think they miscategorized things as hallucinations that were, were not. 107 00:09:25,684 --> 00:09:38,303 But some of the conclusions that they drew, I think are valuable in understanding the comprehensive or comprehension capabilities of LLMs. 108 00:09:38,423 --> 00:09:53,514 So they pointed out that LLMs or these legal research tools specifically that were geared towards legal, they don't understand holdings. 109 00:09:53,544 --> 00:09:56,156 They don't respect the order of authority. 110 00:09:56,156 --> 00:10:02,920 have difficulty distinguishing between legal actors, which all point to a lack of comprehension. 111 00:10:03,941 --> 00:10:15,480 So I think now there was a study, I just saw that it literally just came out this week, like Monday or Tuesday from Metta, Metta and University of California, San Diego. 112 00:10:15,480 --> 00:10:16,590 That's really interesting. 113 00:10:16,590 --> 00:10:21,554 I'm still skimming it, but they are proposing almost like a, 114 00:10:22,634 --> 00:10:26,494 a alternative to chain of thought. 115 00:10:26,553 --> 00:10:35,334 And the gaps that they identified are with, with LLMs, they allocate equal compute to every token. 116 00:10:35,334 --> 00:10:49,598 And the reality is that in a sentence or a prompt, are token words or tokens that require much more emphasis and resource cognitive resources. 117 00:10:49,598 --> 00:11:00,035 than others and they proposed a way to better allocate compute resources to the important areas of a question or prompt. 118 00:11:00,035 --> 00:11:01,706 It's really fascinating. 119 00:11:02,318 --> 00:11:18,458 I posted a link to it in that clip with Damien and I, but I think that's really where the rubber hits the road is, how does the lack of comprehension today dictate where we can use 120 00:11:18,458 --> 00:11:19,306 this in legal? 121 00:11:19,306 --> 00:11:20,845 I think that's what's important. 122 00:11:20,982 --> 00:11:22,143 Yeah, I agree. 123 00:11:22,143 --> 00:11:32,428 And that sounds interesting what you're talking about, though I've not seen that study, but potentially, presumably that could mean that for legal specific applications, you 124 00:11:32,428 --> 00:11:39,632 could choose to have compute power on particular terms that could do those things like respect the order of authority. 125 00:11:40,653 --> 00:11:47,176 But all this stuff, to be honest, it does make me wonder why we're going for these really difficult use cases to start with. 126 00:11:47,176 --> 00:11:49,197 This technology is still quite new. 127 00:11:50,530 --> 00:12:03,195 The issue I have with this stuff is there's a bunch of things that are grinding to a halt in law firms and in-house legal teams that are not really these kinds of problems. 128 00:12:03,996 --> 00:12:13,559 Like, example, people saving things in the wrong place or misleading comments in documents or bad version control. 129 00:12:13,600 --> 00:12:19,970 Why aren't we trying to look at ways of leveraging LLMs to try and improve those quite basic things? 130 00:12:19,970 --> 00:12:27,865 That seems to me to be quite a good idea because in doing that, we might learn from those things, learn from those applications, which by the way, are real tangible problems that 131 00:12:27,865 --> 00:12:29,816 people care about solving now. 132 00:12:29,856 --> 00:12:36,680 I don't know why we're not doing that rather than going for these really impressive sounding, but really complicated and difficult use cases. 133 00:12:36,680 --> 00:12:46,496 Like it seems, it seems like there's a bit of a scenario emerging where sometimes I feel like people developing these really interesting sounding capabilities are. 134 00:12:46,496 --> 00:12:52,831 more interested in the intellectual challenge of developing them than they are actually solving the problems people care about. 135 00:12:52,831 --> 00:13:01,338 But again, that's my bias coming in where I'm seeing lots of talk about capabilities maybe being overstretched a little bit in the market. 136 00:13:01,338 --> 00:13:06,472 And I feel like we could all help ourselves a little bit just by stepping back and simplifying things a little bit. 137 00:13:07,102 --> 00:13:11,073 Yeah, I have been beating that drum, a similar drum. 138 00:13:11,073 --> 00:13:24,067 I've been advocating for firms to really think about business of law use cases first, simply because it's, when you look at this, there's a risk reward equation that has to 139 00:13:24,067 --> 00:13:30,629 strike a balance whenever you're looking at deploying a technology or a process improvement initiative. 140 00:13:30,629 --> 00:13:35,850 And the risk reward on the business of law side, I think, 141 00:13:36,358 --> 00:13:38,959 evens out more, much more favorably. 142 00:13:38,959 --> 00:13:44,140 And it's an incremental step towards a broader AI strategy. 143 00:13:44,520 --> 00:13:45,541 It's lower risk. 144 00:13:45,541 --> 00:13:49,582 don't, mean, you've got HR, finance, marketing, KM. 145 00:13:49,582 --> 00:13:56,504 There's, there's so many places where you could deploy this technology and get familiar and then move into those. 146 00:13:56,504 --> 00:14:04,366 Cause the high ROI, everybody, you know, goes for the sexy, you know, use cases because 147 00:14:04,986 --> 00:14:11,090 the work is high level and the ROI is higher to solve those problems. 148 00:14:11,090 --> 00:14:25,680 But it also comes with much more risk, especially given not just privacy, ethics, capabilities, the opportunities for errors to cause real problems. 149 00:14:25,940 --> 00:14:34,776 when you look at it, people are selling this, including founders, under the premise of disrupting the practice of law. 150 00:14:34,776 --> 00:14:35,602 Hmm. 151 00:14:35,602 --> 00:14:40,244 purposes of creating savings. 152 00:14:40,244 --> 00:14:52,648 And it's not as sexy and interesting for a hundred dollar an hour business of law resource as it is a $1,500 a law, $1,500 an hour practice of law resource. 153 00:14:52,648 --> 00:14:54,828 So yeah, I agree with you, man. 154 00:14:54,828 --> 00:14:56,149 I start with the basics. 155 00:14:56,149 --> 00:15:00,700 And when is that ever a bad idea in technology, starting with the basics? 156 00:15:00,716 --> 00:15:03,058 like at the very least have a portfolio of it. 157 00:15:03,058 --> 00:15:05,740 So focus on all of the above. 158 00:15:05,740 --> 00:15:09,163 Have an R &D team that's doing the really interesting core stuff. 159 00:15:09,163 --> 00:15:18,601 But if we don't do that, we can end up in a world where a lawyer and a law firm is still manually typing out their time narratives every single day by looking back through their 160 00:15:18,601 --> 00:15:20,823 sent items in their inboxes. 161 00:15:20,823 --> 00:15:24,557 But they can have this magic legal research memo drafting. 162 00:15:24,557 --> 00:15:26,456 That just doesn't seem right to me. 163 00:15:26,456 --> 00:15:35,061 Like we've got a, my kind of impression is that a lot of people are saying they've got a portfolio of these things and they're doing some basic applications, but also the really 164 00:15:35,061 --> 00:15:36,322 interesting R and D stuff. 165 00:15:36,322 --> 00:15:45,487 But if you actually scratch beneath the surface, it's mostly the R and D stuff to be honest, because unsurprisingly, that's what all the data scientists want to be working on. 166 00:15:45,487 --> 00:15:46,458 get it. 167 00:15:46,778 --> 00:15:53,292 It's just a drum that I keep banging because I know a lot of lawyers was one myself and 168 00:15:54,134 --> 00:16:02,803 I just think you lose trust a little bit with these guys when you're lauding these amazing capabilities and they're still using a computer that takes 10 minutes to start up in the 169 00:16:02,803 --> 00:16:03,614 morning. 170 00:16:03,774 --> 00:16:04,685 thousand percent. 171 00:16:04,685 --> 00:16:08,347 That's where we get into the risk reward balance. 172 00:16:08,347 --> 00:16:13,661 And attorneys have a very low tolerance for missteps. 173 00:16:13,661 --> 00:16:25,038 before you put a technology in front of them, especially with lofty claims and ambitions and aspirations, like it's got to be right, or you're going to have trouble getting them 174 00:16:25,038 --> 00:16:28,700 back to the table the next time when it's actually ready. 175 00:16:28,825 --> 00:16:37,096 Well, I think that's why this accuracy thing is such a big deal around LLMs because a lawyer using a tool, they'll ask it a question that I can guarantee you the first question 176 00:16:37,096 --> 00:16:38,859 I'll ask it would be to test it. 177 00:16:38,859 --> 00:16:41,893 And then if it gets it even slightly wrong, they'll be like, no, I don't trust this. 178 00:16:41,893 --> 00:16:42,794 Never using it again. 179 00:16:42,794 --> 00:16:44,616 You've to be really careful about that. 180 00:16:45,050 --> 00:16:46,031 100%. 181 00:16:46,031 --> 00:16:47,091 Well, that's good stuff. 182 00:16:47,091 --> 00:16:52,612 You and I talked about search the last time we spoke, which is an interesting topic for us. 183 00:16:52,612 --> 00:16:55,143 So Infodash is an intranet extranet platform. 184 00:16:55,143 --> 00:17:02,925 We're not a search company, but we are the launch, the entry point for search. 185 00:17:02,925 --> 00:17:11,297 So most law firms enterprise search process initiates through their internet. 186 00:17:11,297 --> 00:17:14,128 That's not a hundred percent of the time, but it's common. 187 00:17:14,281 --> 00:17:17,552 So we're interested in the topic. 188 00:17:17,552 --> 00:17:27,234 And if I'm being honest, I don't know that I've ever seen a enterprise search project with a positive ROI in legal. 189 00:17:27,234 --> 00:17:29,275 And that's a big statement, right? 190 00:17:29,275 --> 00:17:33,856 It's not that there hasn't been good outcomes, because I have seen good outcomes. 191 00:17:33,856 --> 00:17:42,158 But when I look at how much was invested and then I look at what the adoption looks like long term, 192 00:17:43,210 --> 00:17:45,034 I have lot of questions about ROI. 193 00:17:45,034 --> 00:17:51,366 So do you think, you know, gen AI is going to change that equation at all? 194 00:17:52,471 --> 00:17:56,514 yeah, I really don't think so. 195 00:17:56,514 --> 00:18:03,959 think that Enterprise Search, I agree, is a very, very difficult project to pull off. 196 00:18:03,959 --> 00:18:13,166 I think there probably have been a few firms that have done it successfully, but not without investing a huge amount of money and in expertise of people who really know what 197 00:18:13,166 --> 00:18:14,266 they're doing. 198 00:18:14,267 --> 00:18:18,690 I think any kind of systems integration issue 199 00:18:19,762 --> 00:18:20,492 System is great. 200 00:18:20,492 --> 00:18:22,353 Your product is always going to be very, very difficult. 201 00:18:22,353 --> 00:18:23,504 You've got to match up. 202 00:18:23,504 --> 00:18:26,135 You've got to have data warehouse strategies in place. 203 00:18:26,135 --> 00:18:29,467 You've got to monitor the ingestion of content you're dealing with. 204 00:18:29,467 --> 00:18:34,639 If you're doing things like trying to index your entire time recording system, it has all your narratives in you. 205 00:18:34,639 --> 00:18:38,291 You're dealing with millions, if not billions of rows of data. 206 00:18:38,291 --> 00:18:42,893 It's very, very, very difficult to get things right. 207 00:18:42,893 --> 00:18:45,374 And then if you get things wrong, 208 00:18:45,378 --> 00:18:49,180 That also means it's quite difficult to re-engineer them because you've got so much data you're dealing with. 209 00:18:49,180 --> 00:18:55,003 So all that is to say that the ROI on enterprise search has to be extremely compelling. 210 00:18:55,224 --> 00:19:07,818 And what I spend quite a lot of time doing with firms and legal teams is exploring whether having more of a curated knowledge strategy in addition to that might be helpful because 211 00:19:07,818 --> 00:19:09,632 you're dealing with small amounts of content. 212 00:19:09,632 --> 00:19:13,536 Of course, the difficulty there is that there's a cultural barrier. 213 00:19:13,536 --> 00:19:15,076 there's an incentive blocker, et cetera. 214 00:19:15,076 --> 00:19:16,197 We can talk about that later. 215 00:19:16,197 --> 00:19:19,908 But do I think that gen AI is going to solve any of this? 216 00:19:20,488 --> 00:19:22,408 I don't see how it can really. 217 00:19:22,408 --> 00:19:30,571 think there's probably, there's probably, there's a couple of ways we can, we can try and answer this question. 218 00:19:30,831 --> 00:19:42,934 There are a lot of people out there using generative AI to, to deliver this kind of natural cert, natural language search or vector search capability. 219 00:19:43,082 --> 00:19:52,790 And what that basically does is it enables you to discover documents that conceptually match your term, even if the keyword you typed in isn't actually in the document. 220 00:19:52,811 --> 00:19:56,214 And that can certainly be really, really helpful. 221 00:19:56,214 --> 00:20:02,629 There's obviously cost implications in doing that because you'd have to do what's called vectorize your underlying document. 222 00:20:02,629 --> 00:20:07,623 I have to have a vector index to make that work, which is not the cheapest thing in the world to spin up. 223 00:20:07,924 --> 00:20:10,890 And it can help you discover things without 224 00:20:10,890 --> 00:20:16,353 knowing what the right keyword is to type in, but also it can generate a lot of noise in the search results. 225 00:20:16,353 --> 00:20:24,477 So sometimes you do want to search for a very specific keyword and you don't, you actively don't want other things to come up that don't match that keyword. 226 00:20:24,477 --> 00:20:27,519 So sometimes useful, sometimes it's not. 227 00:20:27,519 --> 00:20:38,535 I don't think it's going to solve any of the issues around enterprise search because that is a systems integration project, large amounts of data, GenAI, GenAI, GenAI may actually 228 00:20:38,535 --> 00:20:40,246 make that harder to be honest. 229 00:20:40,916 --> 00:20:48,128 Um, the other angle we can look at for gen AI is around nomination or labeling of content. 230 00:20:48,128 --> 00:20:58,251 So this might be more relevant for firms pursuing a knowledge strategy that isn't just a search over all of our documents, but more kind of can we build a knowledge base and can 231 00:20:58,251 --> 00:21:00,611 we categorize the documents in that knowledge base? 232 00:21:00,611 --> 00:21:05,033 Can we proactively suggest documents that could become a part of that knowledge base? 233 00:21:05,033 --> 00:21:06,753 There are some potentials there. 234 00:21:06,753 --> 00:21:10,694 I don't think that AI is in a place where it can 235 00:21:10,702 --> 00:21:15,062 nominate a document as being a good quality example for reuse in the future. 236 00:21:15,062 --> 00:21:17,342 I think that still requires a human trigger. 237 00:21:17,342 --> 00:21:21,542 But can it categorize documents effectively in accordance with some concepts? 238 00:21:21,542 --> 00:21:22,702 Absolutely. 239 00:21:22,702 --> 00:21:32,142 At iManage, we actually use traditional machine learning algorithms, if you remember those, on things like contract type, things like that, which is a lot quicker than GenAI. 240 00:21:32,142 --> 00:21:37,760 But we're also playing around with using GenAI to generate summaries of documents that might 241 00:21:37,760 --> 00:21:42,656 influence search and might provide more context to people when they're looking through search results. 242 00:21:42,897 --> 00:21:45,540 to answer your question, think enterprise search is really, really tough. 243 00:21:45,540 --> 00:21:47,232 I think a few people have already got it right. 244 00:21:47,232 --> 00:21:48,234 It's expensive. 245 00:21:48,234 --> 00:21:53,600 I don't think GenAI is going to help people massively on it, if I'm completely honest. 246 00:21:53,704 --> 00:21:57,941 So no perplexity for the enterprise anytime soon. 247 00:21:58,528 --> 00:22:02,229 Well, mean, maybe it depends. 248 00:22:02,229 --> 00:22:04,480 Again, it goes back to what your use case is. 249 00:22:04,480 --> 00:22:16,153 Like I think that sometimes with enterprise search, where I've seen those projects fail, it's usually because the project is conceived without any real idea of what the ROI is on 250 00:22:16,153 --> 00:22:16,583 it. 251 00:22:16,583 --> 00:22:19,244 It's kind of something that feels like the right thing to be doing. 252 00:22:19,244 --> 00:22:22,765 And maybe it's like an intellectually challenging project to do. 253 00:22:22,765 --> 00:22:28,236 And where those things, the motivation, you tend to have a really, really complex project. 254 00:22:28,398 --> 00:22:36,958 Whereas when it's focused around a particular use case, which might be, for example, let's say the use cases that a business development team wants to find pitches to help them win 255 00:22:36,958 --> 00:22:43,558 more work or spot connections between what different people are doing to help cement client relationships. 256 00:22:43,558 --> 00:22:52,038 If you're going to focus these things around those kinds of use cases rather than a broad bucket of search overall of our content, I think you might get somewhere, but it's all 257 00:22:52,038 --> 00:22:57,062 about trying to reduce that overall corpus of content and not just... 258 00:22:57,260 --> 00:23:00,553 trying to solve your data hygiene issues by throwing a search over it. 259 00:23:00,553 --> 00:23:03,624 I personally haven't seen too many of those projects work. 260 00:23:03,976 --> 00:23:10,204 Yeah, what is, I manage his insight plus and what is the strategy there? 261 00:23:10,978 --> 00:23:15,560 So Insight Plus is what we refer to as a knowledge search product. 262 00:23:15,640 --> 00:23:19,161 And it's a relatively new product. 263 00:23:19,161 --> 00:23:23,943 And we've got customers using Insight Plus over their knowledge bases. 264 00:23:23,943 --> 00:23:29,065 The way the product is kind of architected is around different search experiences. 265 00:23:29,065 --> 00:23:33,167 So it looks a little bit like an enterprise search when you go into it. 266 00:23:33,247 --> 00:23:37,378 But for reasons we may or may not go into, we don't call it enterprise search. 267 00:23:37,378 --> 00:23:40,418 But you have like a tab for knowledge, for example. 268 00:23:40,418 --> 00:23:45,678 And the whole search experience is geared around that particular use case of finding your firm's best practice. 269 00:23:45,678 --> 00:23:51,668 We've got workflows around helping people submit, share, approve, maintain content to go in there. 270 00:23:51,668 --> 00:24:01,697 And importantly, it's a knowledge search product, but what we found from our product research is that actually a lot of lawyers value browse a lot more over search. 271 00:24:01,697 --> 00:24:08,502 And we try to use ways to use metadata to power browsable experiences as well as search experiences. 272 00:24:08,814 --> 00:24:15,099 But there's also a new suite of search experiences coming out next year, which is more similar to the enterprise search stuff. 273 00:24:15,099 --> 00:24:22,486 And this is all about enriching workspace information, the document management system with third party information. 274 00:24:22,486 --> 00:24:26,930 So things like your your CRM or your billing information, that kind of thing. 275 00:24:26,930 --> 00:24:36,504 And those are really built around firms that maybe have a strategy of for various reasons, they don't want to pursue a curated knowledge strategy. 276 00:24:36,504 --> 00:24:40,555 but they just want more levers to pull when people are searching for content. 277 00:24:40,696 --> 00:24:47,529 But it can also deliver business development use cases like people wanting to search for matters that are particularly profitable. 278 00:24:47,529 --> 00:24:56,352 So Insight Plus is going to expand into things like closing books or deal bibles, as we call them in Europe as well over the years. 279 00:24:56,352 --> 00:25:00,984 And we'll also be looking at ways you can look at people results and things like that. 280 00:25:00,984 --> 00:25:06,026 But it's not enterprise search because we're not just throwing loads and loads of content at it. 281 00:25:06,198 --> 00:25:14,388 All of these things are designed around particular use cases, which helps us manage the process and make these things scalable in the future. 282 00:25:15,006 --> 00:25:25,790 Yeah, know, one of the, we were partners with an enterprise search company a years ago and we had a front row seat to a lot of implementations. 283 00:25:25,790 --> 00:25:41,277 In fact, we helped with some of the implementations and one of the challenges that we saw, their strategy was to recrawl and index the entire DM corpus, which yeah, which is, so we 284 00:25:41,277 --> 00:25:44,338 have one, we had one firm, this is just last year. 285 00:25:44,522 --> 00:25:49,442 Um, maybe Amla a hundred, maybe, maybe Amla 80. 286 00:25:49,442 --> 00:25:49,742 I don't know. 287 00:25:49,742 --> 00:26:02,742 They were about 800 attorneys ish and they were spending, was it 40, 20,000, $25,000 a month in Azure compute, um, to do this. 288 00:26:02,742 --> 00:26:05,602 And that did not include the licensing of the product itself. 289 00:26:05,602 --> 00:26:09,962 If I had to guess, I would say that was probably another, so double it. 290 00:26:09,962 --> 00:26:13,322 So close to a million dollars a year to do this. 291 00:26:13,322 --> 00:26:14,050 And 292 00:26:14,050 --> 00:26:15,421 What are the challenges with that? 293 00:26:15,421 --> 00:26:20,575 Because honestly, from where I said, it seems like killing a fly with a sledgehammer. 294 00:26:21,377 --> 00:26:23,518 What are your thoughts on that approach? 295 00:26:23,756 --> 00:26:30,570 Yeah, I think I've heard of people doing this and it's quite a bold approach. 296 00:26:31,091 --> 00:26:38,796 I'm going to be a bit biased here, obviously, because I work for iManage, but we have all your documents indexed. 297 00:26:39,937 --> 00:26:51,545 And to take a local copy of all of those documents or index them locally, again, potentially you can do it, but it's expensive. 298 00:26:51,785 --> 00:26:53,058 It's going to take time. 299 00:26:53,058 --> 00:26:57,409 There's reconciliation issues between you're maintaining two indexes over the same content. 300 00:26:57,409 --> 00:26:59,860 So how are going to make sure you keep them in sync? 301 00:26:59,860 --> 00:27:02,301 Probably risk issues around it as well. 302 00:27:02,941 --> 00:27:08,353 Might even be user experience issues as well when people come to access content for one system. 303 00:27:08,353 --> 00:27:12,141 I'm not saying it's impossible. 304 00:27:12,141 --> 00:27:13,824 I'm not saying it's necessarily a bad idea. 305 00:27:13,824 --> 00:27:20,096 What I'm saying is that it's quite expensive and I think it introduces a lot of complexity. 306 00:27:20,419 --> 00:27:25,086 So you need some really smart people if you're to get that right and a very, very compelling ROI. 307 00:27:25,662 --> 00:27:29,486 It is a heavy, heavy lift for sure. 308 00:27:29,486 --> 00:27:35,231 And yeah, I it's a, have found, we have taken a different approach. 309 00:27:35,231 --> 00:27:45,770 We, no longer partner in that capacity with them and we provide a federated experience with using Azure AI search. 310 00:27:45,770 --> 00:27:52,906 For example, we also do real time API access into the DMS and 311 00:27:52,942 --> 00:27:53,567 Mm. 312 00:27:53,567 --> 00:28:05,953 if we find that approach is a much more streamlined, less resource intensive and close to, we, we, we get pretty good results doing it. 313 00:28:05,953 --> 00:28:08,656 And a lot of firms are leaning in that direction now. 314 00:28:08,780 --> 00:28:09,650 Yeah. 315 00:28:09,770 --> 00:28:18,413 mean, of the conversations I've had with firms actually, we started talking about this stuff and I probed a little bit on it I was like, why do you want this capability? 316 00:28:18,413 --> 00:28:20,473 What's the demand behind this? 317 00:28:20,473 --> 00:28:25,210 And sometimes it's been things like, well, the lawyers just can't find their documents. 318 00:28:25,210 --> 00:28:26,495 They can't find where it is. 319 00:28:26,495 --> 00:28:29,836 And then we have a little chat about why that is. 320 00:28:29,836 --> 00:28:34,597 And they show me under the hood of how they're using their DMS. 321 00:28:34,657 --> 00:28:38,158 And we've managed to solve the issues by 322 00:28:38,318 --> 00:28:42,338 improving people's working practices in DMS. 323 00:28:42,538 --> 00:28:52,058 Because when you deploy iManage, I mean, it's the same with any DMS, I suppose, but you obviously have certain folders in each workspace or matter that's set up. 324 00:28:52,078 --> 00:28:56,298 And the firms I'm talking about here just have like two folders, documents and emails. 325 00:28:56,298 --> 00:29:00,958 And it means there's no consistency from one workspace or matter to the next. 326 00:29:00,958 --> 00:29:04,338 And people are saving things in a random haphazard function. 327 00:29:04,338 --> 00:29:07,014 On cross-border matters, they're saving them. 328 00:29:07,266 --> 00:29:14,049 one transaction in the New York library and another transaction in the London library, even though it's largely the same transaction with the same client. 329 00:29:14,049 --> 00:29:17,450 So it's kind of no wonder people can't find things. 330 00:29:17,650 --> 00:29:19,311 And there's two ways of solving that. 331 00:29:19,311 --> 00:29:29,095 Either you do the enterprise search route and you chuck everything into a search, or you try and solve the issue at source and you try and put in place some better working 332 00:29:29,095 --> 00:29:36,618 practices to help people categorize things a little bit better in their DMS so that they're not facing the problem in the first place. 333 00:29:36,768 --> 00:29:45,275 It's just interesting to see where there's a lot of stuff I do in my consultancy work where people come in with a solution in mind and we actually end up pivoting to something 334 00:29:45,275 --> 00:29:51,370 completely different that's culturally quite expensive, but financially less expensive. 335 00:29:51,540 --> 00:30:04,197 Yeah, you know, we see as well a lot of there's been so much consolidation, at least here in the U S how big firms have gotten big is largely through acquisition. 336 00:30:04,377 --> 00:30:14,182 And what that has led to is there has always been, I'm speaking in broad brushstrokes here, poor data hygiene in the DMS. 337 00:30:14,302 --> 00:30:16,963 Lawyers don't like to profile documents. 338 00:30:17,110 --> 00:30:21,446 They don't file, you know, follow naming conventions and 339 00:30:22,275 --> 00:30:35,323 so what, what we've seen is firms that have grown by acquisition, have one mess of a DMS and then they have three or four or five or six acquisitions with a diff, a mess of a 340 00:30:35,323 --> 00:30:39,106 different nature that they slam in together. 341 00:30:39,106 --> 00:30:51,146 And what you have is five, six different messes and things become really unwieldy and, um, untidy and finding things. 342 00:30:51,146 --> 00:30:56,566 You know, it's a, you know, when I think about a search index, it's like a house, right? 343 00:30:56,566 --> 00:31:10,226 It's a, you, the building of a house and the raw materials for that house are the inputs, the documents, the emails, and you know, the, the, how the house turns out is based on the 344 00:31:10,226 --> 00:31:12,256 architectural plans, right? 345 00:31:12,256 --> 00:31:19,038 Which is equivalent to the, um, you know, the technology that's providing 346 00:31:19,038 --> 00:31:25,942 these capabilities, it's a function of the input, the quality of the materials that were used to build the house. 347 00:31:25,942 --> 00:31:38,489 And then doing this again is like tearing the house down and rebuilding it somewhere else with proper architecture, which seems very inefficient to me. 348 00:31:38,489 --> 00:31:48,424 it sounds like, are you seeing the same thing in terms of the mess that is sometimes a law firm DMS? 349 00:31:48,526 --> 00:31:58,566 Oh yeah, yeah, I mean, there'll be some, there'll be some mergers and acquisitions that happened decades ago and I'll still see libraries that were named after the one of the 350 00:31:58,566 --> 00:32:00,146 firms that no longer exists. 351 00:32:00,146 --> 00:32:04,126 So it's kind of a problem that people don't really want to deal with. 352 00:32:04,126 --> 00:32:05,886 And I get that, you know, it's expensive. 353 00:32:05,886 --> 00:32:17,166 And I think a theme that comes up with a lot of these kinds of things in tech general is you can pull all this architecture in place, whether or not the attorneys will follow it 354 00:32:17,166 --> 00:32:17,666 or not. 355 00:32:17,666 --> 00:32:19,107 is another question. 356 00:32:20,428 --> 00:32:22,189 So you've got to have that in place. 357 00:32:22,189 --> 00:32:32,247 The one thing I would say is that I have actually noticed a real improvement on this stuff and firms wanting to get their house in order. 358 00:32:32,247 --> 00:32:44,970 And that I think is actually driven by people's belief that if they want to harness generative AI properly, then the data sets over which that stuff is run have to be 359 00:32:44,970 --> 00:32:49,451 or have to be clean, to be well organized and good quality content. 360 00:32:49,831 --> 00:32:59,952 I can tell you from some of the experiments we've done, we've got a feature that will basically do a natural retrieval, augmented generation knowledge search, basically, which 361 00:32:59,952 --> 00:33:01,264 I mean, that's a lot of jargon. 362 00:33:01,264 --> 00:33:03,450 So I better just break that down for people who don't know what that means. 363 00:33:03,450 --> 00:33:12,428 But it's basically where you type a question in, the system will identify documents relevant to the question, and then it will generate an answer to your question based on 364 00:33:12,428 --> 00:33:14,238 the documents identified. 365 00:33:14,380 --> 00:33:24,883 And if you start doing that against an entire DMS made out of tens of millions of documents, it's interesting what documents it brings back, which are things like email. 366 00:33:24,883 --> 00:33:35,026 If you're asking, for example, a question about bribery, it will start looking at things like email footers or offering memoranda with some mention of bribery in there, which 367 00:33:35,026 --> 00:33:42,058 aren't really authoritative sources about the question you ask, but nonetheless seem relevant to the question. 368 00:33:43,444 --> 00:33:54,882 So for me, it's pretty much proven based on the experimentation that we've done that you do need a good level of data governance around your content if you want to leverage some 369 00:33:54,882 --> 00:33:55,882 of this stuff. 370 00:33:56,210 --> 00:33:57,330 Interesting. 371 00:33:57,371 --> 00:33:59,272 So what do you think? 372 00:33:59,272 --> 00:34:11,659 There's been a lot of talk recently about foundational or frontier models for legal purposes versus legal specific models. 373 00:34:11,659 --> 00:34:23,602 And, you know, you and I chatted and I think this is true that it's really kind of hard to tell which is the right path until you build it and test it. 374 00:34:23,602 --> 00:34:24,562 Right. 375 00:34:24,824 --> 00:34:26,255 Where are you landing? 376 00:34:26,255 --> 00:34:30,890 you optimistic or skeptical on the need for legal specific models? 377 00:34:33,389 --> 00:34:35,049 It's a tough one, isn't it? 378 00:34:35,049 --> 00:34:48,649 If you go back to the way these models work, which is statistical next token prediction, you'd expect the embeddings graph or the vectors to kind of be more geared towards the 379 00:34:48,649 --> 00:34:50,769 dataset that they're based on. 380 00:34:50,769 --> 00:34:59,209 So it would predict its statistical next token prediction would be driven by, you know, a series of case law or legislation or whatever. 381 00:34:59,429 --> 00:35:02,304 So you'd expect that the output might be a bit more relevant. 382 00:35:02,304 --> 00:35:04,494 It might be bit more interesting. 383 00:35:04,775 --> 00:35:13,537 the issues we face in the use of these things are often related to the risk of hallucination. 384 00:35:13,697 --> 00:35:26,901 And I am personally unconvinced that changing the data set will really change that because these things are not designed to produce accurate output. 385 00:35:26,901 --> 00:35:28,921 That is not a test criteria. 386 00:35:28,921 --> 00:35:31,422 They're designed to statistically predict 387 00:35:31,422 --> 00:35:32,782 next token. 388 00:35:33,163 --> 00:35:43,265 And I don't understand how changing the underlying data set can solve the problem of hallucinations, which seems to be plaguing everybody. 389 00:35:43,685 --> 00:35:54,468 The other issue I throw in here, which I talk about a little bit in an article I wrote recently, which is that if our hope is that these things can become autonomous legal 390 00:35:54,468 --> 00:36:00,650 advisors, and we're trying to improve accuracy, I've touched on this a little bit already, but 391 00:36:00,726 --> 00:36:04,908 I don't necessarily know whether those are the use cases we should be shooting out right now. 392 00:36:04,908 --> 00:36:06,448 Like first of all, because they're hard. 393 00:36:06,448 --> 00:36:09,850 But secondly, don't know whether that's really what we want. 394 00:36:09,850 --> 00:36:15,652 I don't know if we want people to be reading and engaging with the primary materials less. 395 00:36:15,652 --> 00:36:19,494 don't, people always talk about this blank page problem. 396 00:36:19,494 --> 00:36:24,056 And this is an area a lot of people listening will disagree with me on, but I love the blank page problem. 397 00:36:24,056 --> 00:36:25,297 That's what makes me think. 398 00:36:25,297 --> 00:36:29,248 Like if I have a blank page and have to write something, 399 00:36:29,408 --> 00:36:38,285 I've got to structure my thoughts in my head and then I write them down and that flexes my brain muscles quite significantly and it makes me think about things that I wouldn't have 400 00:36:38,285 --> 00:36:42,227 thought about or wouldn't have realized had I not gone through that problem. 401 00:36:42,308 --> 00:36:52,936 And people can say, well, the LLM will solve that blank page problem for you by producing a first draft that's highly accurate because it's trained on legal specific material. 402 00:36:52,936 --> 00:36:56,298 But I'm thinking, well, great, but that means that 403 00:36:56,298 --> 00:37:03,942 my thought process is then confined to the words that are already written on the page, rather than me wrangle with issues that the LLM didn't mention. 404 00:37:03,942 --> 00:37:06,973 And I do worry about the impact that will have on people. 405 00:37:06,973 --> 00:37:08,744 But that's just my own personal preference. 406 00:37:08,744 --> 00:37:13,806 I'm quite open also to the idea that my mind works differently from a lot of other people's minds. 407 00:37:13,806 --> 00:37:19,118 So yeah, think I'm interested to see what I deliver. 408 00:37:19,118 --> 00:37:22,700 I'm not kind of saying that it's a waste of time at all. 409 00:37:22,700 --> 00:37:26,924 I'm probably more veering towards the skeptical end of the spectrum, but I'm watching it really closely. 410 00:37:26,924 --> 00:37:29,637 I think there might be some interesting stuff that comes out of it. 411 00:37:29,637 --> 00:37:34,932 But as usual, we need to really be thinking about why we're doing these things and what the use cases are. 412 00:37:35,630 --> 00:37:40,246 The white screen of death is the new, um, the new terminology, right? 413 00:37:40,246 --> 00:37:42,099 Instead of the blue screen of death. 414 00:37:42,099 --> 00:37:43,119 Yeah. 415 00:37:43,180 --> 00:37:47,385 Which it's better. 416 00:37:47,385 --> 00:37:49,068 It's better than the blue screen of death. 417 00:37:49,068 --> 00:37:54,354 If you've ever, if you're a windows user and you've seen that it's usually not a good day. 418 00:37:54,464 --> 00:37:55,391 Nope. 419 00:37:56,342 --> 00:38:03,332 Um, so do, do we need to build these models before we can evaluate the need for them? 420 00:38:03,332 --> 00:38:10,982 Or is it possible to test this hypothesis without investing hundreds of millions of dollars and actually building them? 421 00:38:13,243 --> 00:38:14,543 Do you know what? 422 00:38:14,643 --> 00:38:16,902 I don't know the answer to that question. 423 00:38:16,902 --> 00:38:21,725 I feel like I need to be, I feel like I'm at risk of saying, no, it's a waste of time. 424 00:38:21,725 --> 00:38:22,725 Don't do it. 425 00:38:22,725 --> 00:38:29,497 And then someone says, who knows far more about me than this kind of stuff will say, actually, that could be some interesting things. 426 00:38:29,497 --> 00:38:30,888 You're not quite right. 427 00:38:30,888 --> 00:38:35,599 When you talked about hallucinations and next word prediction, I am not a data scientist. 428 00:38:35,599 --> 00:38:38,600 Like I learned all this stuff just by reading on the internet. 429 00:38:38,600 --> 00:38:39,950 So I'm 430 00:38:39,950 --> 00:38:42,620 I think people should continue doing this work. 431 00:38:42,620 --> 00:38:51,830 think it's all these avenues need exploring because if they can pull it off and it can do things that we can't even think about now, I'm all for it. 432 00:38:51,830 --> 00:39:02,430 And by the way, I do think also that these problems we're having around like hallucinations and generally all of the kind of drawbacks of LLMs that we're seeing, it's 433 00:39:02,430 --> 00:39:09,358 highly likely that in the next, take a long-term view on it, 10, 20 years, like my own prediction is that in 434 00:39:09,358 --> 00:39:11,880 15, 20 years, maybe shorter, I don't know. 435 00:39:11,880 --> 00:39:15,953 Someone will come up with something else that is completely different from an LLM. 436 00:39:15,953 --> 00:39:19,004 It's not next token prediction, not a thing. 437 00:39:19,645 --> 00:39:23,208 Maybe it can, I don't know how it works, but maybe it can just produce accurate things. 438 00:39:23,208 --> 00:39:25,029 I've quite a strong belief that is going to happen. 439 00:39:25,029 --> 00:39:34,576 So it seems to me to be highly valuable to start engaging in some of these thought experiments about what happens when we get there so that we're prepared when we get there. 440 00:39:34,576 --> 00:39:38,274 And I think the current technology might start those, but I don't think it's quite. 441 00:39:38,274 --> 00:39:41,845 going to live up to the promise people maybe thought it had a couple of years ago. 442 00:39:42,260 --> 00:39:52,173 Yeah, I completely agree that this is very unlikely the last iteration of architecture for AI. 443 00:39:52,268 --> 00:39:53,779 Yeah, completely. 444 00:39:53,879 --> 00:40:02,847 I think that I do feel like as long as LLMs are still LLMs and they work the same way, I do think we're going to start to hit a ceiling. 445 00:40:02,847 --> 00:40:11,844 And some of these agentic things like feeding the output back into itself and predicting the next step is like an interesting development, but it's not really changing that much. 446 00:40:11,844 --> 00:40:14,706 It's just the same operational methodology. 447 00:40:14,706 --> 00:40:21,822 Until that changes, I suspect we probably won't get to this holy grail of accurate output. 448 00:40:21,982 --> 00:40:34,809 Yeah, there's a lot of talk lately about scaling laws and I heard an interesting podcast this morning at the gym where they were talking about, you know, there's really kind of 449 00:40:34,809 --> 00:40:39,732 three levers to pull for innovation in with LLMs. 450 00:40:39,732 --> 00:40:45,095 It's, you know, the amount of compute, basically the number of Nvidia chips that you can throw at it. 451 00:40:45,095 --> 00:40:51,542 It's the data, which, you know, the training data, which is finite and 452 00:40:51,542 --> 00:40:58,187 Um, so, so we're the number of chips, um, they're now training on synthetic data as well. 453 00:40:58,448 --> 00:41:04,273 And then the last lever is the algorithm itself. 454 00:41:04,273 --> 00:41:16,164 And there, that is not, there isn't a finite, number of ways there's, there's infinite potential paths to improve. 455 00:41:16,164 --> 00:41:17,125 would imagine. 456 00:41:17,125 --> 00:41:18,986 So it feels like. 457 00:41:19,144 --> 00:41:30,069 Yeah, it's probably not going to be throwing more, you know, GPUs or training data at the same architecture that's gonna get us to the next level. 458 00:41:30,069 --> 00:41:37,628 It's, it's probably going to be something in the tech, not in the tech, um, which seems logical. 459 00:41:38,363 --> 00:41:47,771 I like this stuff, like my understanding was that it actually came from translation use cases and someone was kind of tinkering around with it and found that Transformers 460 00:41:47,771 --> 00:41:50,387 actually could have these other potential applications. 461 00:41:50,387 --> 00:41:54,197 So I suspect it would be something like that that we don't even know about right now. 462 00:41:54,197 --> 00:41:55,948 And I don't doubt that it will happen. 463 00:41:55,948 --> 00:42:02,654 And I think people will be a bit more interested when it does, because I think this stuff, the LLMs was such a step forward. 464 00:42:02,922 --> 00:42:08,558 I hope people don't come away from this thinking I'm like some sort of AI skeptic or deny or whatever. 465 00:42:08,558 --> 00:42:09,749 I think the stuff is incredible. 466 00:42:09,749 --> 00:42:10,750 I really do. 467 00:42:10,750 --> 00:42:22,903 It's just that the bias I was talking about earlier means that I'm always slightly trying to dampen hype on things because I don't know if inflating expectations is necessarily the 468 00:42:22,903 --> 00:42:26,636 best thing when they can't deliver on those expectations. 469 00:42:26,846 --> 00:42:28,447 Yeah, no, we share that view. 470 00:42:28,447 --> 00:42:36,970 I use AI probably 10 times a day, very, very frequent for all kinds of stuff. 471 00:42:36,970 --> 00:42:50,494 So I don't, I don't, I'm very bullish on the future of AI, but I'm, I'm, I'm, I'm very bearish on the marketing messaging coming out right now aligning with capabilities. 472 00:42:50,956 --> 00:42:52,847 Yeah, no, agreed, agreed. 473 00:42:52,847 --> 00:43:05,136 And in the moment, it seems to be there's a lot of stuff around the agentic AI as well, like which you could have another conversation and actually the content around the current 474 00:43:05,136 --> 00:43:07,718 use of LLM is all about the quality of the content. 475 00:43:07,718 --> 00:43:17,484 And then with agentic AI, it's probably the quality of the process and the needs to define the process and stop one person doing the same process in a completely different way from 476 00:43:17,484 --> 00:43:18,595 another person. 477 00:43:18,595 --> 00:43:20,350 It all comes back to this kind of 478 00:43:20,350 --> 00:43:25,388 human hygiene element and kind of putting a bit of control around things. 479 00:43:25,610 --> 00:43:38,150 Yeah, I had a Aaron Amadea from relativity on a few episodes back and we talked about agentic AI, super smart dude, PhD in math and I was an undergrad in math and like barely 480 00:43:38,150 --> 00:43:39,230 got through it. 481 00:43:39,230 --> 00:43:54,782 So, uh, he, he, yeah, it is, uh, you know, I was, I'm really good at like, um, I was really good at calculus and, um, differential equations and matrix theory. 482 00:43:54,782 --> 00:44:00,777 where it went off the rails with me was advanced calculus where you have to go back and prove everything in calculus. 483 00:44:00,777 --> 00:44:11,537 So proofs are extremely difficult and really require a lot of abstract thinking and they, they kicked my ass honestly. 484 00:44:11,537 --> 00:44:13,538 So, um, 485 00:44:13,875 --> 00:44:15,516 That is good though. 486 00:44:15,556 --> 00:44:22,339 I really enjoy, like part of the reason, a lot of people say I just, I'm a kind of person that naturally disagrees with things. 487 00:44:22,339 --> 00:44:25,040 And I guess that might be true. 488 00:44:25,040 --> 00:44:28,101 And the kind of the reason I do it is because I want people to tell me I'm wrong. 489 00:44:28,101 --> 00:44:29,202 is I learned. 490 00:44:29,202 --> 00:44:33,684 I think in this age, the one thing I've learned over the last couple of years is don't be precious around how much you know. 491 00:44:33,684 --> 00:44:39,646 And if someone smarter than you tells you that you're not right, go ahead and accept that and learn and move on. 492 00:44:40,170 --> 00:44:41,130 100%. 493 00:44:41,130 --> 00:44:49,850 Yeah, it's understanding what you don't know is as good and valuable as understanding what you do. 494 00:44:49,950 --> 00:44:53,770 All right, last question for you, because I know we're running out of time here. 495 00:44:53,770 --> 00:45:07,810 What do you think about how the capital markets, I mean, we're seeing so much money flow into, if you were to stack rank all the industries where legal has been predicted to be 496 00:45:07,810 --> 00:45:09,150 transformative, 497 00:45:09,974 --> 00:45:17,296 Um, law, you know, legal is at the top of most lists, um, simply because there hasn't been a innovate. 498 00:45:17,296 --> 00:45:23,298 hasn't been a huge innovation around the concept of language like LLMs ever. 499 00:45:23,298 --> 00:45:23,708 Right. 500 00:45:23,708 --> 00:45:30,880 You know, you could argue maybe search, but, um, I think LLMs are just completely transformative with potential. 501 00:45:30,880 --> 00:45:38,248 What do you, do you think there's, uh, you know, is it problematic that we're seeing all this money flow in and how that's impacting 502 00:45:38,248 --> 00:45:42,461 marketing messaging and people's experience with the technology? 503 00:45:42,461 --> 00:45:43,844 What are your thoughts on that? 504 00:45:43,970 --> 00:45:51,933 Well, as somebody that's passionate about driving change in the legal industry, for me, it's always good news that money's flowing into this kind of investment. 505 00:45:51,953 --> 00:45:58,986 I'll never say it's bad news ever for companies trying to improve how lawyers work to have continued investment. 506 00:45:58,986 --> 00:46:02,070 So broadly, think it's a great thing. 507 00:46:02,070 --> 00:46:06,900 I think though, you mentioned that legal's top of these use cases. 508 00:46:06,900 --> 00:46:12,520 And I think that mostly that's because when people think about lawyers, 509 00:46:12,520 --> 00:46:19,816 first thing that tends to come to their head is long documents, long written words, long briefs, that kind of thing. 510 00:46:21,137 --> 00:46:24,340 And they kind of say, well, lawyers write a lot, don't they? 511 00:46:24,340 --> 00:46:27,322 And LLMs also write a lot. 512 00:46:27,322 --> 00:46:31,486 therefore, I'm going to conclude that lawyers are going to get replaced by LLMs. 513 00:46:31,486 --> 00:46:37,020 And therefore, I'm going to invest an awful lot of money in LLMs, because I think it could be disruptive to the legal industry. 514 00:46:37,020 --> 00:46:42,134 And I think it could have some pretty far reaching impacts on the 515 00:46:42,134 --> 00:46:43,474 legal industry. 516 00:46:43,615 --> 00:46:53,020 The one thing I would say though, is that I don't think it's necessarily the right approach to go in with such a superficial understanding of what actually the legal 517 00:46:53,020 --> 00:46:54,461 industry is about. 518 00:46:55,542 --> 00:47:04,127 Again, one of the other things when I speak to you to understand my bias is towards my prior career as a large big law lawyer. 519 00:47:04,127 --> 00:47:09,750 But if I think about the kind of work I did, and my practice was half contentious and half transactional, 520 00:47:09,934 --> 00:47:14,324 much time did I actually spend writing long documents? 521 00:47:14,324 --> 00:47:19,994 How much time did I spend standing up in court, drafting briefs, doing this kind of stuff? 522 00:47:19,994 --> 00:47:23,194 Well, I think back to it, very little really. 523 00:47:23,194 --> 00:47:31,914 Most of it was around managing my inbox, to be honest, trying to project manage, especially when you're junior, a lot of it is project management. 524 00:47:31,914 --> 00:47:37,570 And then you've got the whole ream of menial tasks like producing closing books or 525 00:47:37,570 --> 00:47:39,331 doing these kinds of things. 526 00:47:39,331 --> 00:47:47,773 And I think that it's great that everyone's bought into the idea of disrupting legal industry through generative AI. 527 00:47:47,773 --> 00:47:53,774 But I really wish they wouldn't confine their mission to a method. 528 00:47:53,774 --> 00:47:57,355 By that, mean, don't just look at generative AI. 529 00:47:57,455 --> 00:48:01,196 Try and get bought into the idea of disrupting the legal industry. 530 00:48:01,196 --> 00:48:07,078 But don't just focus on generative AI, because there are so many things lawyers do that are completely backwards. 531 00:48:07,566 --> 00:48:10,888 could and should have been fixed 20 years ago. 532 00:48:11,969 --> 00:48:16,953 So things like that I would kind of encourage people to focus on. 533 00:48:16,953 --> 00:48:22,517 I suspect actually what might happen is that when these investments were made, people had really high hopes. 534 00:48:22,517 --> 00:48:26,639 They thought lawyers were going to get replaced by machines that could draft all their documents for them. 535 00:48:26,639 --> 00:48:31,173 I suspect over time people realize that probably is not actually quite the case. 536 00:48:31,173 --> 00:48:33,785 Like we'll be in some areas, but not across the board. 537 00:48:33,785 --> 00:48:35,926 And I suspect that they probably will. 538 00:48:36,326 --> 00:48:43,114 kind of pivot a little bit towards focusing more on the realities of how people work and intangible use cases. 539 00:48:43,466 --> 00:48:44,471 So yeah, that's my view. 540 00:48:44,471 --> 00:48:46,377 I'm fully in favour of this investment. 541 00:48:46,377 --> 00:48:47,300 I think it's great. 542 00:48:47,300 --> 00:48:50,764 It's getting people excited about technology who weren't excited about it before. 543 00:48:50,764 --> 00:48:53,056 And those are all good things in my view. 544 00:48:53,074 --> 00:48:53,635 Yeah. 545 00:48:53,635 --> 00:49:03,604 I mean, if you look at Harvey, for example, and their cap table, I mean, they've got open AI, they've got Google ventures, they have a 16 Z, which is Andreessen Horowitz. 546 00:49:03,905 --> 00:49:08,418 You've not seen interest at that level in this, in this area ever. 547 00:49:08,418 --> 00:49:11,039 Yeah, I mean, it's great. 548 00:49:11,039 --> 00:49:11,749 It really is great. 549 00:49:11,749 --> 00:49:14,550 And it'll be interesting to see where Harvey goes. 550 00:49:14,550 --> 00:49:18,642 know, I've been really impressed by what I've seen from it. 551 00:49:18,642 --> 00:49:21,323 It's good that they're being a bit more open about what they're building. 552 00:49:21,323 --> 00:49:26,655 And there's lots of other companies in the mix as well that have good kind of pedigree investors as well. 553 00:49:26,655 --> 00:49:29,546 So I think all in all, it's a very, very good thing. 554 00:49:29,566 --> 00:49:36,970 Yeah, I'm actually really happy to see the pivot in Harvey's PR strategy and opening up more. 555 00:49:36,970 --> 00:49:44,484 And I actually met Winston for the first time at TLTF last week and yeah, very sociable guy. 556 00:49:44,484 --> 00:49:51,878 And, um, I was impressed and he needs to get out more so people can engage with him and get to know him. 557 00:49:51,878 --> 00:50:02,266 this is the thing, like, because I think that the problem with that kind of marketing is that I take Figma as an example, right, the UX design program, like people, historically, 558 00:50:02,266 --> 00:50:05,588 people use Sketch or Photoshop, and then Figma came out and it was better. 559 00:50:05,588 --> 00:50:07,297 And so everyone started using Figma. 560 00:50:07,297 --> 00:50:11,612 And I think a lot of these these companies are entering the legal industry afresh. 561 00:50:11,612 --> 00:50:19,800 It's a lesson I learned, actually, which is that just because you are really passionate about technology, that does not mean that all the lawyers that 562 00:50:19,800 --> 00:50:28,734 have billable hour targets and are really, really super busy and don't have time to get lunch, let alone try a new piece of technology, are as excited about it as you are. 563 00:50:28,734 --> 00:50:37,758 So it does not necessarily mean that just because your tech's really good and exciting, just because you can help people do things quicker, that does not necessarily translate 564 00:50:37,818 --> 00:50:40,439 into value and usage of tools. 565 00:50:40,439 --> 00:50:44,781 And that's a lesson I learned really badly as somebody that is passionate about this stuff. 566 00:50:44,781 --> 00:50:46,764 Not everyone shares my passion. 567 00:50:46,764 --> 00:50:52,359 So I had to develop a whole skill set around how I convince people to give up their time to use these things. 568 00:50:52,359 --> 00:50:56,362 And I don't think AI has changed that, if I'm completely honest with you. 569 00:50:56,372 --> 00:50:58,284 mean, I sell intranets and extranets. 570 00:50:58,284 --> 00:50:59,715 You think that excites people? 571 00:50:59,715 --> 00:51:02,146 Yeah. 572 00:51:03,408 --> 00:51:07,286 Well, it doesn't most people, but yeah, we try to get out there. 573 00:51:07,286 --> 00:51:10,443 I did a real quick plug for a previous episode. 574 00:51:10,443 --> 00:51:15,988 It just released what is today, but by the time this episode airs, it'll be several weeks. 575 00:51:15,988 --> 00:51:20,191 It's with Alex Sue, who is the chief revenue officer at Latitude. 576 00:51:20,191 --> 00:51:26,046 And we talked about, he wrote a really good article that inspired the episode about how difficult 577 00:51:26,046 --> 00:51:27,668 legal is to disrupt. 578 00:51:27,668 --> 00:51:33,255 is extremely difficult and history has proven this over and over and over again. 579 00:51:33,255 --> 00:51:41,594 So for these big Silicon Valley investors that think they're just going to write some checks and all the dominoes are going to fall in their direction, they're sadly mistaken. 580 00:51:41,594 --> 00:51:43,190 It's going to be real work. 581 00:51:43,190 --> 00:51:43,990 absolutely. 582 00:51:43,990 --> 00:51:52,870 And I've got friends who've been at firms where they've had these AI tools rolled out, know, I name any names of the tools, but their reaction to me is kind of like, I don't 583 00:51:52,870 --> 00:51:53,790 know what to use this for. 584 00:51:53,790 --> 00:51:58,050 Like this is people tell me it's AI, but I kind of what so what what do do with it? 585 00:51:58,050 --> 00:52:01,370 And that's why that barrier must be bridged. 586 00:52:01,370 --> 00:52:04,470 And lots of people have different views on this whole use case thing. 587 00:52:04,470 --> 00:52:06,562 And lots of people are we shouldn't tell 588 00:52:06,562 --> 00:52:08,753 lawyers about use cases, know, let them experiment. 589 00:52:08,753 --> 00:52:17,827 We don't want to confine them too narrowly, but you do have to do it a little bit because otherwise people who have, you know, only two or three minutes to play around with 590 00:52:17,827 --> 00:52:20,418 something in a day, they're just not going to use it. 591 00:52:20,418 --> 00:52:22,409 So you do have to do a little bit. 592 00:52:23,690 --> 00:52:24,500 Yeah. 593 00:52:24,621 --> 00:52:27,123 Well, this has been a great conversation. 594 00:52:27,123 --> 00:52:34,590 Before we wrap up, how do people find out more about you, iManage, what's the best way to connect? 595 00:52:35,180 --> 00:52:39,634 Well, any legal technology conference, you'll see, I manage there. 596 00:52:39,634 --> 00:52:43,157 If you're lucky or unlucky, depending how you look at it, I might also be there. 597 00:52:43,157 --> 00:52:46,619 So come and speak to me, come and introduce yourself. 598 00:52:46,619 --> 00:52:50,623 Otherwise I'm fairly active on LinkedIn and I post a lot of stuff on there. 599 00:52:50,623 --> 00:52:55,056 And also Blue Sky, which is something that more and more people are coming on now. 600 00:52:55,056 --> 00:52:57,719 So have a look at me on there or reach out to me. 601 00:52:57,719 --> 00:53:02,363 always, my mantra is I never say no to anyone on LinkedIn where they message me. 602 00:53:02,363 --> 00:53:04,584 So yeah, always offer a chat. 603 00:53:04,820 --> 00:53:05,812 Good stuff. 604 00:53:05,812 --> 00:53:10,309 All right, well, I appreciate your time and we went a little bit over, but it was a great conversation. 605 00:53:10,309 --> 00:53:17,370 So, I'll look for you at the next tech conference and hopefully we can connect in person. 606 00:53:17,398 --> 00:53:17,908 Nice one. 607 00:53:17,908 --> 00:53:19,060 Thanks, Ted. 608 00:53:19,060 --> 00:53:24,500 Some really interesting stuff there and look forward to quizzing you on some calculus in the future. 609 00:53:24,500 --> 00:53:24,990 There we go. 610 00:53:24,990 --> 00:53:26,682 I'm a little rusty, but I'll do my best. 611 00:53:26,682 --> 00:53:28,492 All right. 612 00:53:28,492 --> 00:53:29,304 Have a good afternoon. 613 00:53:29,304 --> 00:53:31,075 All right. 614 00:53:31,075 --> 00:53:32,095 Take care. -->