AI’s development is showing strongly in the legal field
The newest generative language models and related technologies in particular — reasoning models that work through complex legal questions, and Retrieval-Augmented Generation (RAG) tools that draw on legal source material — could change how lawyers and law students do their work. But what concrete results can you expect from using the newest AI models? That question was examined in a recent, comprehensive experimental study assessing AI technologies’ real benefit in legal tasks.
Background and design
The study involved 127 law students from two top US universities. They were given six varied, realistic legal tasks ranging from short client communications and memos to more demanding contract and dispute resolution work. Every task required applying relevant legal material and justifying the conclusions clearly.
The students were randomly divided into three groups:
One group completed the tasks entirely without AI tools.
A second group used a reasoning model developed by OpenAI (called “o1-preview”).
A third group used the Vincent AI system, which combines RAG technology with automated prompting. In other words, Vincent AI sought to retrieve and embed case law, statutes, and other legal source material in the AI-produced text, so as to reduce the risk of incorrect sources or “hallucinations”.
Each student completed six tasks in total: two without AI, two assisted by the reasoning model, and two with the RAG-based Vincent AI. All output was assessed blind: the graders did not know who had used AI and who had not, or to what extent. The assessment took account of content quality, legal analysis, depth of argument, and clarity of writing.
Results: AI models raise the quality of lawyers’ work
Improved quality
Both AI tools significantly raised the quality of work in four of the six tasks, measured by clarity, organisation, and overall professionalism. Participants with AI to hand wrote clearer, better-structured answers. Notably, the reasoning model (o1-preview) produced better results particularly in the depth of legal analysis: it helped students grasp the logical and legal background of problems more precisely than usual.
Speed and productivity
Both AI systems significantly reduced the time taken to complete tasks — by as much as 20–30 per cent at best. When quality and time were considered together, that is, when “productivity” was measured (points per minute used), the improvements were greater still. In time-consuming, complex legal tasks in particular (such as writing a comprehensive memo or analysing a claim), the results showed a clear productivity leap.
Hallucinations vs. source accuracy
Earlier experiments have raised the worry of “hallucinations”, in which AI invents source references or facts that do not exist. In this study, the Vincent AI tool (based on RAG technology) produced fewer hallucinations than general-purpose tools tested previously, such as GPT-4. Fewer “invented sources” still appeared among participants who did not use AI at all — meaning AI cannot be called infallible. In that sense RAG appears to bring stability, but it also underlines the need for careful human checking.
Differences by task and by ability
The study found that certain tasks benefited from AI clearly more than others. Procedural tasks involving writing and the structure of argument seemed to be helped most, whereas in a purely contract-based task (an NDA, or non-disclosure agreement) using AI did not always lead to a significantly better result or greater speed. Students with lower GPAs also benefited on average more from AI’s support than those who already had a high skill level.
What does the study mean in practice?
Practical help for a lawyer’s work
The results show that advanced AI models do not merely speed up routine tasks; they raise the quality of lawyers’ work. That matters, because earlier studies had pointed mainly to gains in speed, with quality remaining variable.
The need to combine different AI techniques
The ways Vincent AI (RAG plus automated prompt structures) and reasoning models (such as o1-preview) were used showed that the two technologies have complementary strengths. RAG tools have the advantage in finding sources and cleaning up at ground level (fewer “invented” references), while advanced reasoning models excel at solving difficult, multi-stage legal problems. When the best of both technologies is combined, productivity can rise further still.
The importance of training and skill
The human role remains important. Critical thinking, legal judgement, and case-by-case adaptation are still the core of a lawyer’s expertise. The newest AI models can support that process by offering drafts, links to sources, or clearer text structures, but a human still has to verify the accuracy of sources and the quality of the result.
The study likewise found that with practice and good guidance, participants learned to use AI more effectively — and got increasingly better results as a consequence.
Anticipate new ways of working
Many workplaces are currently weighing up adopting AI, either with tailored solutions or with public language models. This study shows that the new generation of AI models (law-focused RAG and reasoning models) has real potential to change the effectiveness and efficiency of legal work.
In closing
According to the study, new AI technologies can replace earlier general-purpose language models in supporting legal work, because they offer better efficiency and quality than before. As both RAG and reasoning models develop — and integrate with one another — their impact on legal analysis, drafting, and the whole advocacy process could be genuinely significant.
Although many in the legal field still have learning to do and reservations to work through, the study’s message is quite clear: once you learn to combine the use of AI with precise legal thinking and checking the relevant sources, AI can genuinely extend the capacity of a lawyer and a legal team — and thereby improve access to justice, client service, and the quality of the work. Work in the future will increasingly be a collaboration between human and AI, in which AI shines in speed and organisation, and the human brings their most valuable contribution in strategic judgement, creativity, and ethical responsibility.
Enjoyed this article? Subscribe to get notified about new Tekoälyfoorumi articles straight to your inbox. We’d also hugely appreciate it if you left us a review on Google. Thank you. 💙
