At the backend of 2022 we started to hear a lot of noise about ChatGPT that was outside of the usual circles us legal tech folk move in. A lot of this was due to the ingenious and often hilarious uses people were posting to social media, but also the mainstream press picking up and publishing countless stories about AI.

In ILT we were getting question after question about how it worked, will it change the world, how it applies to the work we do as a law firm, and what our strategy was going to be. These questions came from a place of interest and excitement, but also unfamiliarity, and sometimes fear. 

We have dedicated a lot of time this year to educating our colleagues and clients on the realities of LLMs, machine learning, transformers and prompting techniques. Goldman Sachs throwing fuel on the fire in March with "44% of legal tasks could be automated by generative AI" didn't help – but by and large we have managed to get a good understanding of the realities of AI across the firm.

As a team, we have never been one to be driven by hype and have always set our goals on what is achievable and realistic, but if you work in Legal Tech, you will agree that this level of excitement about technology in a law firm is rare. So naturally we wanted to take advantage of the hype to get help drive forward everything we’d been doing. 

As we quickly realised how important this technology was going to be, our first port of call in 2023 was to make sure that we have the correct mechanisms and stakeholders in place to assess Generative AI as a new and emerging technology.

We needed a clear approach and guidance on how our colleagues should use ChatGPT as well as best practice. We wanted them to engage with the technology in the right way and then understand when and how it should be used. This way we could get ahead of and control some of the initial fear that played out in the market and address any mistrust in the firm.

Several groups were created within the firm, consisting of key stakeholders:

• The Generative AI Decision Group. Consisting of key people from ILT, Office of the General Counsel and IT. This group was responsible for high level decisions and risk sign off around our use of Generative AI.
• Application of Generative AI Group. This is made up of ILT team members focusing on AI and the legal working group and aimed at uncovering use cases and testing the effectiveness of this technology.
• The Legal Advisory Group. This group focuses on supporting clients on the legal aspects of purchasing, using, or developing Generative AI tools. They also look at regulatory and jurisdictional updates and support us internally with working with these tools safely.
• Generative AI Taskforce. To centrally manage engagement across clients, conferences, events and have a clear voice through external content we created a group made up of Business Development and Marketing alongside both the Application and Advisory groups mentioned above. 

Through a combination of these groups' efforts, we drafted and rolled out a Generative AI Policy, decided our risk appetite for the use of third-party solutions and worked with our IT team to set up a safe space within our Microsoft Azure environment to carry out bespoke development. These groups have also helped us have sensible conversations with our clients around the use of AI in the work that we do and have a clear strategy when it comes to negotiating supplier terms both for ourselves and for our clients.

Getting our house in order at such an early stage has helped us stay on top of this technology throughout the year. It put us in a position where we can bring tools in to test, build our own custom solutions and most importantly, engage with our colleagues and our clients in a well-structured and controlled manner.

Once we got to grips with the basics of the tech, we started to speak to people in the market and run some exploratory demoes and sessions. We ended up speaking to around 75 companies that were focusing on using LLMs (mainly GPT-3 and then 4 as time went on) specifically in legal. We were conscious that the best way to figure out new things is to get hands on and play. We wanted to either find a partner to do that with us, a vendor who would work with us, or build a sandbox where we could do this ourselves. 

We quickly narrowed this list down and kicked off some pilots with those we thought had a good approach, strong team and lots of potential. We also of course reached out to our current providers where we thought these newer models may help and kicked off some roadmap related conversations. 

You cannot change anything in a law firm without the buy in of the lawyers, so we reached out to the firm and asked for any volunteers that would like to be involved in the testing and wider application of these new tools. After an overwhelming response, we kicked off a working group made up of 150 people representing all teams from across the firm. We split this group across various tools and gathered feedback on use, as well as continuing to run training sessions and engage with the wider firm. 

Working with the end users of these tools means that we were able to narrow down key features, as well as spotting anything LLMs were particularly good (or bad) at. 

Shiny technology on its own does not move the needle in terms of adoption, but direct examples that can be shown to deliver on a specific use case have always helped us drive engagement. We approached Generative AI like any other technology – learn and identify what it does well, look at the problems we have in the firm, match up ideal use cases, and test our theories. Once we had confidence in the functionality we could bring to our colleagues, we then started spending time with different teams on concrete examples. We also learnt that you can plan and put processes around things but also just putting it in the hands of our colleagues meant that they experimented with different uses for Generative AI within their working day which helped drive forward our understanding of how we could use it across the business.

Our main findings have been that these tools have some clear strengths and are well suited to some specific use cases that we have been trying to deliver for some time. At a base level, these models are impressive but the more structure you put around them, in relation to prompting strategies, structured data outputs, user interface, etc, the better the overall product can become. 

LLMs are good at finding relevant language in documents (extraction) and then using this extraction to answer a question or respond to an instruction. Tools built using LLMs to deliver this need additional features above just a link to the model to make this work, but we are seeing good results on some the work we are delivering. These models are getting better, with token limits increasing and accuracy within those limits improving, we are also starting to see better results using Retrieval Augmented Generation (RAG) as an approach to help with large documents. There are some great explainers on the workings of RAG out there already, so we won’t add to the noise!

One of the predicted use cases – legal drafting – is not something that we are spending much time on or that impressed with. We have a good precedent library and can work with document automation, clause banks and guidance from knowledge lawyers to allow our lawyers to access the relevant drafting quickly, without relying on an AI. However, we have noticed that these tools can be helpful with non-legal drafting such as proposal wording, team engagement, training scripts, and user guidance documentation.

LLMs are usable and working today, but this is still the beginning, and the more we learn and test, the better we can make the wider tools we are using. 

“My biggest takeaway from this year is that the most important thing is to get people using this technology. The way you use it is different to other technologies we are used to, so allowing people to learn, experiment and share is the best engagement technique I can think of.”

Kerry Westland, Partner and Head of the Innovation Group

The key takeaway from this year has been the importance of getting the input right into these models, otherwise known as “Prompt Engineering.” A lot of the initial negative feedback on the performance of LLMs has been due to poor inputs. As we have gotten the hang of how these models work, we are able to get much more accurate responses, focus down the outputs to be more concise and remove a lot of the hallucination risk. 

Consistency and accuracy are obviously two especially important things for a law firm. We are working on prompt libraries and templates for specific use cases to avoid inconsistencies in prompting from individuals resulting in different responses. We are approaching Generative AI as just another tool in a law firm’s wider toolkit, so we still follow the usual processes around validation and feedback. As these models improve, we are making sure we stay on top of developments and continue to roll out and test new features for accuracy above anything else!

Training and educating our colleagues has been vital in getting engagement with tools such as AGPT. This has ranged from sessions about LLMs and Transformers, common misconceptions, general tips, and tricks, to a more detailed Prompt Like a Pro session that saw over half the firm attend. The more people learn about these tools the better the feedback has been.

“Running ‘Prompt like a Pro’ sessions across the firm was a great way of getting the concepts of Prompt Engineering across to our colleagues. I had over 1000 attendees across multiple slots, and we saw a marked increase in positive feedback around AGPT as people got to grips with how to communicate with these tools better.”

Mike Kennedy, Senior Manager, Research and Development

Part of our strategy has always been to focus on the problem, rather than the tech, and if we can fix it ourselves through leveraging tools we have already or building out simple applications then we will. However, we have never shied away from buying the right tool for the job. A tool that solves a specific problem is valuable, as it will be used immediately and relied upon by our colleagues, even if it doesn't go further than one pain point.

Our Generative AI strategy has continued this method, with us looking to the market to bring in specific tools that immediately enable us to deliver on certain work types, looking at general productivity tools such as Microsoft Copilot, but also building out our own internally developed tool, AGPT. 

AGPT was a product of us trying to find a way of testing GPT-4 within our own secure environment, allowing colleagues to properly use it to answer queries or review documents without the worry around data. This was important for engagement, and we have had some great use cases come from across the entire firm since the full release of AGPT in September.

We recognise that as a law firm, we are not a software company and cannot build out fully fledged products suited to the market at large, but due to the knowledge we have access to and the people in the firm, we can build something customised to us. This build strategy has taken a lot of time and luckily, we have had the development capacity to get this done, as well as the imagination to develop and start working through a detailed roadmap.

“Our build strategy for Generative AI this year was to learn, by doing this we have enabled people in roles across the firm to learn. It allowed our developers to understand how to take advantage of the Azure OpenAI services to power a platform. It allowed our Legal Technologists to understand what goes into building an enterprise platform used for legal work. It allowed our colleagues to learn how to use Generative AI in a safe environment.”

Elliot White, Director, Innovation and Legal Technology

Alongside the time spent with our colleagues this year, we have spent a huge amount of time with our clients. We have delivered training sessions and explainers on LLMs and AI more broadly and continued to grow our Legal Technology Consulting offering, often helping clients cut through the shiny thing syndrome and focus on the tools they really need. We have supported on client away days and brainstorming sessions and allowed some of our clients to access these tools through us to allow them to make their own decisions and judgements.

There are lots of learnings that we could share from all these conversations, but broad themes have included: 

• The reality of bringing these tools in house when considering budget and resource;
• AI’s potential impact on fees and pricing models;
• An understanding of the reliability and trust of these models, including questions about hallucinations;
• Continuing to train junior lawyers and build a sustainable future empowered by technology; and
• The potential to explore AG’s ability to support clients with legal self-service tools.

As a firm, we have also been advising clients on the legal implications and risks around AI, with a Client Advisory Group set up at the same time as our Application Group leading the way on a range of legal aspects covering Intellectual Property, Data, Regulations, and contract negotiations with AI vendors. For more information – click here [Generative AI | Addleshaw Goddard LLP].

“I have spoken to over 100 clients this year across our AGenda series, one to one workshops, AI demoes, away day slots and general catch ups. It is clear that there is a lot of excitement about LLMs and their potential in law, but the challenges with resource are still the same as they always have been. I have really enjoyed being able to support our clients as they lean on us not only as their law firm but as an adviser across the world of legal tech.”

Mike Kennedy, Senior Manager, Research and Development

Looking to the next year, we will be focusing on how we bring the hype and promise of Generative AI into reality, working closely with the vendors we have decided to bring in, our current tech stack, and our software development team to make this happen. 

Delivery

This is the key theme to look out for over the next year. It will be a shame if we get towards the summer and the conversations about Generative AI’s ‘potential’ are still ongoing. There are some strong use cases for LLMs in legal, and this next year we will need to deliver on expectations, both to the market at large but for our colleagues as well. There will come a time where interest wanes and cynicism seeps in but the key point for us is to keep our heads down and keep delivering. 

We plan to further embed Generative AI uses cases across the firm with the technologies we have acquired, and we have a whole roadmap laid out to drive our AGPT platform forward.  We are hoping for a year filled with open discussion and we will be looking to share as much as we can, and we hope our peers do also.

Integrations and product development

Across the wider market, we will see continued integrations and potentially acquisitions, we have already seen Thomson Reuters make a big move in 2023, so looking at how they bring CoCounsel into the wider TR suite of products will be an interesting watch. Contract Lifecycle Management providers and current Legal Research and AI Tools are also looking at incorporating LLMs into their products, so we think 2024 will be a year of product development for many. We may also see some of these vendors struggle and then look towards acquisitions to fill gaps.

Learning

As mentioned throughout this piece, we have spent a lot of time on education and training, both of ourselves so we can get to grips with the potential of this tech, but also our colleagues and our clients. We will continue to do this over the next year and believe that as the legal market begins to understand the underlying tech behind some of these products, we will see increased engagement and an element of trust develop. Even if it is related to the fact that we can more clearly define what LLMs do well, and what they do badly.

Beyond AI - Business as usual

A final point, whilst this piece has been entirely about Generative AI, as a team we are continuing to deliver across all our objectives. This covers the new and horizon scanning stuff that we do in R&D, but also the ongoing great work we do with our clients on live matters, the consultancy and product development work we do to support clients, precedent automation and document lifecycle management, and the everlasting challenge of driving adoption of some of our efficiency improving tools. 

Generative AI is capturing a lot of the attention in the space, but don’t let this make you think that teams like ours are just playing with the newest thing!