AI-powered form-checking - Developing an automated AI approach

In traditional procurement operations, manually checking forms is time-consuming and costly. This is particularly true for quotations received from external companies, which come in a mix of paper and PDF formats, lacking standardized configurations and content. For instance, Fujitsu's procurement department uses a checklist with approximately 40 items, conducting multiple layers of checks, which limits the number of verifiable items to only a few dozen per day. To overcome this challenge, many companies are striving to develop AI-powered form checking technology, but there are two key issues that crop up.

Firstly, forms such as quotations often have diverse formats from company to company, as well as often containing complex descriptions, which makes it difficult to structure the data accurately. Secondly, when leveraging AI, engineers are required to generate prompts tailored to the unique rules of each company and business operation, which incurs significant costs. To address these challenges, we have developed a new approach - procurement form check AI. This technology contributes to preventing erroneous orders and reduces the need for multiple manual checks.

── I heard that procurement form check AI is being trialed internally by Fujitsu.

Sachihiro: We have been conducting demonstration experiments in collaboration with our internal procurement department, with the aim of replacing the manual visual confirmation of quotes with this technology. Traditionally, to prevent erroneous orders, the procurement department would check quotes against a checklist of approximately 40 items, and this was done through a double and triple-checking system. So, we collected the external quote data used by the procurement department in their actual operations and analyzed the feasibility of structurization. We also looked at the possibility of checking the content, using the procurement form check AI. The results are very promising - field-based feedback indicates that a labor cost reduction of approximately 50% can be expected. Currently, we are discussing the extraction of issues and operational methods, while making improvements to enhance the AI's checking accuracy.

── Please tell us about the difference between AI OCR and procurement form check AI.

Kengo: A typical AI OCR takes an image of a form as input and recognizes the text written on it, converting it into digital data. However, form layouts are extremely diverse. For example, even for an address, the positional relationship between the field name "address" and the written address can be side-by-side or stacked, varying by company and form. While AI OCR can extract the field name and the written address as text, it often doesn't understand their relationship. Our technology, however, can extract and associate the field name "address" with the written address.

Sachihiro: In addition, typical AI OCR requires precise pre-configuration of the positions of any items to be read, together with the rules for content checks. In contrast, our technology automatically analyzes the rules and generates prompts for checking simply by inputting an existing CSV-format checklist. A key feature is that even if checking rules change, it can be handled simply by updating the CSV-format checklist.

── Please tell us about the usage scenarios for the procurement form check AI.

Sachihiro: For instance, when checking a quote, the user uploads the quote they want to check to the procurement form check AI, along with any related files (such as PDFs) such as the request for quotation, and their existing checking rules (in CSV format). A specialized VLM (Vision Language Model: an AI model that can integrally handle image, video, and text information) reads the quote and creates structured data from it. A prompt generation LLM (Large Language Model: an AI model capable of understanding, generating, summarizing, and translating text) analyzes the rules in the checklist and generates prompts for checking. Finally, a check LLM uses the structured data from the quote and the generated checking prompts to verify the content and output a judgment result. If the user finds an error in the judgment result, they can simply correct it – with a prompt improvement LLM improving future prompts based on this feedback.

── What technologies comprise the procurement form check AI?

Sachihiro: The procurement form check AI is composed of three technologies. The first is a technology that converts image data, such as quotes, into structured data that an LLM can understand. Our specialized VLM can structure complex forms, including those with diverse layouts and multi-header quotes. Instead of merely reading character information from a simple image and converting it to digital data, which is what OCR does, it can output key-value pairs (data where an item and its content are set, like "Recipient": "Fujitsu").

The second is a technology that automatically generates prompts for the LLM to perform checks based on existing checking rules. Simply handing existing checking rules directly to an LLM does not result in accurate checks. This is because checking rules written for human verification may contain ambiguous judgment criteria. Our prompt generation technology doesn't just pass the existing checking rules to the LLM; instead, it organizes instructions, judgment criteria, and reference materials to generate prompts in a format suitable for checking.

The third is a technology that improves prompts based on users' corrections to the judgment results. The prompt improvement LLM learns from user feedback and continuously and automatically modifies the prompts, thereby enhancing checking accuracy.

── How did you enhance the accuracy of AI checks?

Kengo: Forms like quotes provided by various business partners differ in format from company to company, and many also have complex structures, making VLM analysis challenging. Fine-tuning a VLM (additional training and optimization) requires diverse and large quantities of annotated form data (text and image data with tags or labels). However, annotation work incurs significant costs. To address this, we utilized LLMs automatically to generate specialized training data, thereby achieving diverse and large quantities of fine-tuning-specific data at a low cost.

Sachihiro: In prompt generation, we decomposed and clarified instructions and judgment criteria to create prompts that are easier for the LLM to understand. In addition, by restricting reference materials for each checking item, we generated prompts that provide the LLM with only the necessary information.

── Please tell us about the planned future improvements for the procurement form check AI.

Kengo: For the specialized VLM, improving accuracy when structuring complex forms is a challenge. Currently, information from forms is extracted in a single process. However, we plan to improve it so that even forms with more complex structures can be structured with higher accuracy, just by adding a process that involves multiple reviews, much like how humans understand complex forms.

Sachihiro: When verifying quotes, complex reasoning and calculations are often required, such as, for example, 'assuming one person-month equals 170 hours, how many people and total hours of effort are involved during the period from the start date to the end date?' To achieve this, we aim to create an autonomous checking agent and integrate it with external tools that perform complex calculations, thereby enabling more accurate checks.

Noriko: We want to improve the process of inputting data into the LLM to enhance the accuracy of form checking while suppressing the LLM's workload. We plan to proceed with technical verification, using real-world data to improve the content and timing of data input into the LLM.

── Please tell us about other use cases.

Sachihiro: Let me introduce a few application examples:

── What are the future plans for this technology?

Noriko: We want to enhance the AI’s versatility. For example, even for similar tasks, different companies have different customs and judgment criteria. We aim for the AI to be able to adapt flexibly to such cases. Furthermore, it is crucial to design the AI for easy maintenance even if operational rules change over time. We will proceed with research into technology that clarifies implicit checking criteria, allowing for changes and operations based on those criteria.

Kengo: We want to enable customers to handle different types of forms quickly by customizing the procurement form check AI. To that end, we aim to realize technology that can convert complex forms into structured data even when learning data is scarce.

Sachihiro: Currently, we are proceeding with demonstrations limited to certain tasks, such as quotes. In the future, we want to advance the technology development and on-site demonstrations to support form checking across all procurement operations, including ordering, imports and exports. To handle more advanced and broader tasks, a mechanism where humans actively participate in decision-making and judgment processes (Human-in-the-Loop) is indispensable. Through this mechanism, we aim to learn from user feedback and continuously improve checking accuracy.