Empowering Contract Decisions with a Custom LLM
Organization A manages vast quantities of unstructured contract data that make it laborious and costly to efficiently extract key information and to understand their spending patterns. We aimed to streamline contract management by developing an AI-powered solution that reduces labor costs, increases spending awareness, and empowers data-driven acquisition decisions.
Our team developed and deployed a custom large language model (LLM) for users to query contract data for high-value use cases.
The Problem
Organization A lacked an understanding of its spend patterns and a method for efficient contract querying to inform its acquisition decision-making.
My Role
As product leader, I defined the product strategy by conducting user interviews with stakeholders across the contract lifecycle to understand their workflows, pain points, and desired outcomes, focusing on identifying valuable KPIs. Based on these insights, I prioritized high-value use cases to maximize ROI and align development with the product vision, ensuring the team focused on the most impactful features and measurable results.
Stakeholders
Primary
Acquisition Officers
Contract Administrators
Contract Analysts
Leadership
Tertiary
Legal Counsel
Program Management Office
Vendors
Goals
Product
Streamline contract management and improve acquisition decisions with AI, reducing cycle time by 10% and increasing compliance by 5% within a year
Engagement
Achieve 90% monthly active user engagement among acquisition personnel by providing a significantly improved contract querying experience that simplifies their jobs and reduces manual effort
Anti-Goals
Introduce unintended biases into the platform's algorithms or outputs that could inadvertently favor specific vendors
Scoping
During the scoping phase, I collaborated closely with acquisition, contracting, and program management stakeholders across Organization A to define the project's objectives and ensure alignment with their needs. I facilitated workshops and individual interviews to identify high-value use cases for applying AI/ML to contract data, focusing on improving spend transparency, streamlining workflows, and enhancing decision-making. I also established key metrics, including contract cycle time reduction and compliance increase, to measure the project's success and ensure it delivered tangible value. Throughout the process, I maintained consistent communication with stakeholders to gather feedback and ensure the custom LLM addressed their specific requirements.
Potential Solutions
Option 1
Develop a standalone, interactive contract intelligence platform.
Longer development time, higher costs, and potential user workflow disruption were why we opted not to do this.
Option 2
Integrate custom LLM directly into existing contract management systems.
Integration with existing systems meant faster deployment, easier user adoption, and leveraging existing infrastructure.
Findings
Our pilot program demonstrated the significant potential of the custom LLM to streamline contract management at Organization A. We successfully reduced contract querying labor hours by 6.25% (approaching our target of 10%) and saw positive user feedback regarding improved spending transparency. However, while initial results are promising, further fine-tuning of the LLM is needed to fully optimize its accuracy and ensure consistent performance across all contract types. This refinement, along with additional user training, will be crucial to achieving our full compliance increase goal of 5%. We also identified opportunities to expand the LLM's capabilities to include predictive analytics, forecasting future contract needs and identifying potential cost savings.