World-renowned AI researcher reveals breakthroughs accelerating AI agent practical application Fujitsu's Challenge to Revolutionize AI Agent "Collaboration, Memory, and Quality"

For multiple AI agents to work together effectively across diverse tasks, each agent needs strong, well-rounded training. Today, however, the shortage of high-quality supervised fine-tuning data - the kind of data required to teach agents how to collaborate smoothly - remains a major barrier. Without it, improving overall agent performance becomes harder.

The breakthrough addressing this challenge is the Agent Data Protocol[5]. The conceptual diagram below illustrates how it works. First, raw data is collected from a variety of agent datasets. It is then processed using a unified set of Actions and Observations defined by the Agent Data Protocol, and the resulting Trajectory is stored.
By standardizing these diverse datasets and converting them into a form that’s immediately ready for learning, the protocol dramatically reduces the preparation time required before reinforcement learning can begin. A dataset with more than 1.6 million training instances has already been released publicly, allowing anyone to start reinforcement learning with well-trained agents right away.

――We'd like to ask about the challenges and solutions for deploying AI agents in enterprises. Integration with existing systems and overall complexity are major concerns. What technical and organizational barriers do you anticipate?

Professor Neubig: Deploying AI agents in enterprise environments is highly challenging for several reasons.. The primary reason is that the data companies possess is generally not publicly available and is not included in the training data of existing large language models. However, this is where the Agent Data Protocol proves highly valuable. By creating company-specific datasets following this protocol and combining them with existing general-purpose datasets, we can improve agent accuracy without over-biasing toward specific corporate data. This approach allows AI agents to utilize valuable corporate data while preserving broad applicability. Security and other challenges remain, but this marks a significant step forward.

―― It is often noted that multi-agent systems consume up to 15 times more tokens than traditional chat systems. (Tokens refer to units of text processed by AI models.)
In training AI agents using the Agent Data Protocol, are there technical challenges to address from the perspectives of cost efficiency and computational resources?

Professor Neubig: There are two costs: training cost and inference cost. Training costs are not prohibitively high; for instance, a model with 32 billion parameters can be trained on two nodes with NVIDIA H100 GPUs in approximately four days. (H100 refers to NVIDIA’s high-performance AI accelerator).
This isn't particularly expensive compared to many other tasks. More importantly, once an agent is sufficiently trained, its knowledge can be transferred and utilized in smaller models [6]. This has the potential to improve cost efficiency by up to 10 times compared to using API-based models. The Agent Data Protocol provides a crucial foundation for achieving this significant improvement in cost efficiency.

――Please tell us about the necessary conditions and requirements for the Agent Data Protocol to become widely adopted as an industry standard, as well as the challenges the research community and industry should address.

Professor Neubig: First, we believe we must continue to actively promote this protocol. Research projects often end with the publication of a paper, but it's crucial to convert new agent training datasets into the Agent Data Protocol format whenever they emerge and to strengthen collaboration with other institutions. Furthermore, a key challenge we haven't yet addressed is handling multimodal data. We are currently working on this, and if achieved, it will make a significant difference in many use cases. We believe these are the keys to widespread adoption of the protocol.

For AI agents, a memory mechanism is essential for maintaining contextual consistency and preventing information loss. In this chapter, we interviewed Professor Bisk, the paper's author, about the technical innovation behind "Embodied RAG (Retrieval-Augmented Generation)" [7], a groundbreaking approach for efficiently utilizing memory within physical environments.

In this segment, Dr. Kobashi explores the innovations behind "Embodied RAG" with Professor Yonatan Bisk (Carnegie Mellon University).

――First, could you explain how Embodied RAG differs from conventional RAG, particularly regarding the innovation in memory management within physical environments?

Professor Bisk: The most fundamental difference between Embodied RAG and conventional RAG lies in the "definition of semantic units" and the "method for determining relevance." Conventional RAG retrieves information based on word similarity within documents. However, in the physical environment Embodied RAG addresses—such as an office with rugs and plants—it is unclear how to define similarity for physical objects. For example, the experience of "the route taken to work today" is a composite accumulation of multiple elements and does not directly correspond to a specific document like language does. Therefore, the greatest innovation in Embodied RAG lies in how it understands the spatial environment, determines which information is worth retrieving, and judges which pieces of information are relevant to each other.

――How does Embodied RAG update and adapt its memory to environmental changes and unexpected situations? Vulnerable workflows are cited as a cause of failure. Could you elaborate on adaptation to dynamic environments?

Professor Bisk: To handle dynamic environments, "reliability estimation for memory" and "quantifying uncertainty" are essential. For example, the location of buildings or walls is a highly reliable memory, while the position of a coffee cup is prone to change and less reliable. Humans judge this intuitively, but agents similarly need the ability to determine what is still true based on memory decay and uncertainty.This enables agents to act more cautiously—seeking help or shortening plans. Furthermore, since physical mistakes can be irreversible, a "human-in-the-loop" mechanism becomes crucial when risk levels exceed a certain threshold, prompting requests for human intervention. Agents, as collaborators, must also adopt a stance of not acting on ambiguous instructions.

――What do you see as the technical complexities of implementing Embodied RAG in enterprise environments and the key challenges to overcome for practical deployment?

Professor Bisk: Implementation in enterprise environments is extremely complex, as the saying goes, "The devil is in the details." The biggest challenge in physical environments is that we cannot ignore the details of the real world, such as ambiguity, dynamic nature, and multi-agent systems. Practical challenges also include ensuring reliable power supply, overcoming local computing limitations, and mitigating communication failures. . For example, in a factory, there are dozens of cameras and diverse robots. Managing all these components through a single centralized database is impractical. I believe the direction of distributed cooperative memory, where each agent has its own memory and shares information "selectively" as needed, is preferable. This alleviates the requirements for full communication and continuous power supply,and also contributes to privacy protection. We should aim for systems that support human work through a human-centered approach.