Law Firm Revolutionizes Operations with Cutting-Edge AI Technology

Getty Images

New Research on Legal AI Tools

A recent study by a major British law firm, Linklaters, has found that artificial intelligence (AI) tools have improved in their ability to answer legal questions but still fall short of the capabilities even of junior lawyers. The research involved setting chatbots 50 complex questions about English law and tested various AI models released from 2019 onwards.

The results showed OpenAI’s GPT-2, introduced in 2019, was “hopeless” at providing accurate responses but that the newer model, GPT-o1 released in December 2024, demonstrated significant improvement. Linklaters concluded that while these tools are becoming more useful for real-world legal work under human supervision, they still cannot replace professional judgment.

The law profession is grappling with how AI advancements will affect their practice and whether to view them as a threat or opportunity. Some international firms like Hill Dickinson have restricted access to AI tools after observing increased usage among staff members.

There is also an ongoing debate globally about the risks posed by AI and the level of regulation needed. Recently, the US and UK declined to sign an international agreement on AI, with concerns raised over European emphasis on safety versus innovation.

Evolving Benchmark Tests

This was Linklaters’ second LinksAI benchmark test, following one in October 2023 where they tested OpenAI’s GPT-2 through GPT-4 alongside Google’s Bard. The current round has expanded to include OpenAIs o1 and Googles Gemini 2.0 from the end of 2024.

The test posed questions that would require advice from a “competent mid-level lawyer” with two years’ experience, involving tasks such as legal research support and draft review but not complete legal representation.

Limitations and Future Outlook

Newer AI models performed better than previous versions but still made errors, omitted important information, or invented citations. However, Linklaters noted these tools could be helpful in providing first drafts of documents and checking answers with human oversight.

The firm highlighted that even as AI progresses significantly over recent years, questions remain about whether this trend will continue or if there are inherent limitations to what AI can accomplish. Client relations and interaction were deemed crucial aspects of legal services, suggesting “fleshy bits” would persist in delivering legal work despite future technological advancements.