Thought Leadership  •  May 18, 2022

Start the Conversation

Honeypot Field to Catch Bots
Honeypot Field to Catch Bots

Tips For Evaluating Machine Learning For Contract Review

Advances in machine learning and natural language processing technology are increasingly applied in ways that augment knowledge workers across a wide array of professions.

The practice of law is no exception and, in fact, is more conducive to reaping the benefits of related tools than other areas. Many law firms, audit and consulting firms, corporate legal departments, and alternative legal service providers utilize artificial intelligence, or AI, across myriad applications within their organizations ranging from legal research to e-discovery to contract analysis.

With respect to contracts specifically, use cases include conducting due diligence, regulatory and compliance review, lease abstraction, Libor remediation, post-merger contract migration, and extracting and tracking data from company contracts as part of digitization efforts.

While there is certainly a lot of buzz around AI to maximize value and return on investment, it's important for organizations to think strategically when procuring and deploying this type of technology.

What Are You Looking to Solve?

It's important to identify the business needs of your organization and the related key requirements to meet them. These needs will vary depending on the type of organization you represent and your contemplated use cases.

Once you have determined your needs, the next step is to understand the specific elements of your workflow that new technology can assist with and where it cannot. Identifying these functionality gaps is important.

Let the Implementation Begin.

After taking into account the above and selecting your technology, it's now time for implementation. Rather than just diving in, there are some best practices to consider that will help ensure your effort is a success.


Start by considering where the new system will sit in the business and how it will function within the current workflows. You will want teams throughout the organization to quickly see the value the technology brings because this will help to spur adoption. Typically, the department or group that drove the purchase of the technology serves as a beachhead for adoption within the business.

In a corporation, this might be the legal department or corporate development team. In a law firm, it might be the M&A or private equity practice group. In an audit or consulting firm, it might be the advisory or tax group. Once this group demonstrates some early wins with respect to their use case, internal advocates can assist in expanding the technology into other applications throughout the organization.

User Training

Both at the initial kickoff and throughout the life of the software license, look for opportunities to leverage the support resources at your disposal. These may include training sessions, online guides and technical user support. All new systems have a learning curve, and it is important to make sure your organization is getting as much value out of the tool as possible.

When it comes to training, start with a small set of your core users. This will help to ensure consistency and set expectations. Here are three guidelines to keep in mind throughout the process:

Transparency is key. It is important to be open about the system's limitations as well as its benefits. Be clear that contract analytics software is not an attempt to remove attorneys from the contract review process but rather is designed to augment their work and help them perform their jobs more accurately and efficiently.

When examining metrics to measure the technology's impact on accuracy, do not compare what the system can do versus that of a human. Since the system is meant to be used in tandem with a human, it is best to compare the results of an attorney using the software against those of an attorney performing a traditional manual contract review.

Prepare yourself for the inevitable scenario when two highly qualified attorneys disagree on the output needed for a particular review. At times like these, all parties must keep in mind that the software is not going to satisfy both of their expectations all of the time.


Depending on use cases, some organizations will find significant value in having nontechnical users train the software to meet their specific needs. This custom self-training helps tailor contract extractions to granular, actionable data in line with a given industry or project.

Such custom training can be a great way to scale domain expertise and create a competitive advantage for the organization. Some systems provide functionality, so this custom training can be done as a natural part of the contract review process for a given project, with no additional work required.

When utilizing custom self-training as part of a contract analytics system, it's important to have an organized plan in place. Teams working on the project should follow agreed-upon guidelines and maintain consistency in how they annotate the provisions used to train the system.

Approaches may vary depending on the type and complexity of provisions or data points you're looking to train the system on as well as the variability in the documents you're planning to analyze.

Ned Gannon is president of eBrevia by DFIN. 

The opinions expressed are those of the author(s) and do not necessarily reflect the views of the firm, its clients, or Portfolio Media Inc., or any of its or their respective affiliates. This article is for general information purposes and is not intended to be and should not be taken as legal advice.

Ned Gannon

Ned Gannon

President, eBrevia by DFIN