The design of virtual agents is undergoing a significant transformation, driven by the advanced capabilities of foundation models (FMs), particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). These FMs now serve as the cognitive core for agents, enabling sophisticated understanding, reasoning, planning, and interaction. This literature review delves into the AI design principles and architectures underpinning modern virtual agents, focusing on their cognitive architectures, reasoning and planning mechanisms, learning and adaptation capabilities, and the pivotal influence of FMs. We explore key innovations such as advanced memory systems (e.g., MemGPT, A-MEM) that address context limitations, sophisticated reasoning frameworks (e.g., ReAct, Reflexion, Tree-of-Thoughts, Case-Based Reasoning integration) that enhance decision-making, and paradigms for agent self-improvement and autonomous evolution (e.g., SICA, ADAS). The review also examines the AI design of multi-agent systems (MAS), including collaborative architectures (e.g., AutoGen, MetaGPT) and the emergence of collective intelligence. A central theme is the trend towards agents that are not only more autonomous but are also capable of evolving their own cognitive processes and architectural blueprints, largely enabled by the meta-cognitive capabilities of recent FMs. However, this rapid progress is accompanied by significant challenges in ensuring robustness, reliability, safety, and ethical alignment, particularly as agents become more integrated into real-world applications. The increasing complexity and resource demands of FM-powered agents also highlight the growing importance of system-level support, such as that proposed by AIOS, to efficiently manage cognitive resources. The field is maturing towards a more holistic approach, integrating AI-aware system design with advanced cognitive capabilities, while emphasizing the critical need for rigorous evaluation and responsible development to harness the full potential of AI-driven virtual agents.

1. Introduction to AI-Driven Virtual Agents

The domain of virtual agents is undergoing a profound transformation, shifting from predominantly rule-based automatons to sophisticated, AI-driven entities. These modern agents are characterized by their capacity for complex reasoning, autonomous planning, and nuanced interaction with their environments and human users [1]. This evolution is not merely an incremental improvement but signifies a paradigm shift towards what is increasingly termed “agentic AI”—systems that perceive, reason, learn, and act with a significant degree of autonomy to achieve specified goals [2]. Such a qualitative leap is fundamentally altering the landscape of artificial intelligence and reshaping the potential of human-machine collaboration [3].

At the heart of this advancement lies the pivotal role of foundation models (FMs), particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). These models are increasingly serving as the “cognitive engine” or the central processing unit for contemporary virtual agents [4]. FMs bestow upon agents unprecedented capabilities in natural language understanding, intricate reasoning, dynamic planning, and even the generation of code necessary for interacting with external tools and environments [5]. The AIOS paper, for instance, underscores the power of LLMs in comprehending instructions, solving problems, and engaging with both human users and external systems [6]. This integration allows agents to undertake complex tasks that traditionally demanded human expertise, spanning domains from scientific research to software development [2]. The development of AI agents, therefore, represents a convergence of previously distinct AI research areas, including natural language processing, automated reasoning, machine learning, and even aspects of robotics for embodied agents. This amalgamation is fostering the emergence of more holistic and versatile agentic systems. Foundation models are the critical enablers of this convergence, providing the core intelligence that allows these disparate capabilities to be woven into a cohesive agent. This trend points towards the development of more “generalist” agents, moving away from the narrow specializations of earlier AI systems.

A significant conceptual development is the emergence of an “agent layer” as a distinct abstraction built upon foundation models [2]. This layer endows the underlying FMs with the necessary scaffolding for perception, memory, planning, tool use, and action execution, thereby transforming them from passive information processors into proactive, goal-oriented entities [7]. The foundation model provides the core intelligence, while the agent architecture furnishes the structure and mechanisms for this intelligence to be purposefully and autonomously applied within an environment. This architectural pattern, characterized by an FM core augmented with agentic scaffolding, is rapidly becoming a standard design principle. It offers modularity, allowing for the underlying FM to be updated or replaced while the overarching agentic framework remains intact. Consequently, advancements in FMs directly translate into enhanced capabilities for the agents built upon them.

This literature review aims to provide an in-depth analysis of the AI design principles and architectures underpinning modern virtual agents. The primary focus will be on their cognitive architectures, reasoning and planning mechanisms, learning capabilities, and the indispensable role of foundation models in shaping these facets. While system-level support, such as that proposed by the AIOS (Agent Operating System) framework [6], is acknowledged for its crucial role in enhancing AI functionalities and managing resources for efficient agent operation, the core of this review will remain centered on the AI aspects of agent creation and functionality. Particular emphasis will be placed on recent publications, especially those from late 2024 and 2025, to ensure the review reflects the current, rapidly advancing state of the art in this dynamic field.

2. Cognitive Architectures for Intelligent Agents

The design of cognitive architectures for intelligent agents is a critical area of research, defining how agents perceive their environment, process information, make decisions, and learn. Modern AI agent architectures are increasingly modular, often centered around powerful foundation models that act as the cognitive core.

2.1 Core Components from an AI Perspective

From an AI design standpoint, contemporary agent architectures typically comprise several key modules. These include perception, for interpreting sensory data and environmental cues; reasoning and planning, for decision-making and strategy formulation; memory, for storing and retrieving relevant information; and action execution, for interacting with the environment [5]. Foundation models often serve as the central processing unit orchestrating these components [3].

The perception module enables agents to ingest and interpret data from diverse modalities, including text, images, audio, and video, as well as inputs from external environments [8]. The advent of MLLMs has been particularly transformative for this component, allowing agents to build a richer understanding of their surroundings [9].

The reasoning and planning module forms the cognitive heart of the agent. Here, perceived information is processed, goals are evaluated, decisions are made, and strategies are formulated. This often involves complex processes like task decomposition, hierarchical planning, and the selection of appropriate actions or tools to achieve objectives [3].

Memory is fundamental for agents to maintain context, learn from past experiences, and inform future decisions. This includes both short-term working memory for ongoing tasks and long-term memory for storing knowledge, skills, and episodic experiences [3]. The AIOS paper, for example, details memory and storage managers that handle agent interaction histories and persistent data, respectively [6].

The action module is responsible for executing the decisions made by the reasoning module. Actions can range from generating textual responses and making API calls to external tools, to controlling actuators in physical or virtual environments [3].

2.2 The Influence of Foundation Models on Architectural Blueprints

Foundation models, including LLMs and MLLMs, are increasingly the central linchpin around which agent architectures are designed and implemented [3]. They provide the core intelligence for sophisticated language understanding, complex reasoning, and even the generation of code required for tool invocation and environmental interaction. The AIOS framework, for instance, conceptualizes LLMs as “cores,” analogous to CPU cores in traditional computing systems, highlighting their central processing role and allowing for the modular integration of diverse LLM instances [6]. The inherent capabilities of the chosen foundation model—such as its context window size (e.g., GPT-3’s 2-4K tokens versus Gemini 1.5’s 1M tokens [1]), reasoning proficiency, and multimodal understanding (as seen in models like GPT-4 [10] and Gemini 2.5 [11])—directly determine the potential complexity and operational effectiveness of the agent’s cognitive functions.

2.3 Innovations in Memory Systems for Long-Term Context and Learning

A significant challenge in designing agents based on foundation models is the inherent limitation of their context windows. This “context wall” restricts the amount of information an LLM can process at any given time, hindering capabilities essential for agents, such as engaging in long-term conversations, continuous learning from extensive histories, or analyzing large documents. Consequently, a substantial body of recent research has focused on developing innovative memory architectures that aim to transcend these limitations, moving towards dynamic, structured, and even self-evolving memory systems.

One notable approach is MemGPT [12]. Drawing inspiration from hierarchical memory systems in traditional operating systems, MemGPT introduces virtual context management. This technique allows the agent to intelligently page information between the LLM’s constrained main context (analogous to RAM) and various external storage tiers (recall storage for message history and archival storage for arbitrary text objects, analogous to disk storage). Crucially, MemGPT empowers the LLM agent to self-direct memory edits and retrieval operations through function calls, effectively extending its working memory and enabling it to manage information flows far exceeding its native context capacity.

Another innovative system is A-MEM (Agentic Memory), proposed by Xu et al. (2025) [13]. Inspired by the Zettelkasten method of knowledge management, A-MEM facilitates the dynamic organization of memories. When new information is encountered, the agent autonomously constructs a structured “note” containing contextual descriptions, keywords, and tags. The system then analyzes historical memories to identify and establish meaningful links with existing notes. Furthermore, A-MEM supports memory evolution: as new experiences are integrated, they can trigger updates to the contextual representations and attributes of historical memories. This allows the agent’s knowledge network to continuously refine its understanding and organization over time, creating a self-evolving memory system.

System-level support, as exemplified by the AIOS memory and storage management components [6], also plays a crucial role. AIOS provides kernel-level services for managing agent interaction histories during runtime (transient memory) and handling persistent data such as knowledge bases (long-term storage). The memory manager employs strategies like LRU-K (Least Recently Used - K times) eviction for swapping data between RAM and disk, enabling agents to effectively manage and access larger volumes of historical interaction data than would otherwise be possible. The storage manager ensures persistent access to critical knowledge resources. These OS-level functionalities directly underpin the AI’s need for robust, scalable, and efficiently managed memory, which is fundamental for learning and long-term reasoning.

The trend towards dual-memory architectures is also prominent in recent LLM agent designs [14]. These architectures typically distinguish between long-term memory, which houses static, pre-trained, or fine-tuned knowledge, and short-term memory, which dynamically stores information from ongoing environmental interactions and dialogue history. Retrieval-Augmented Generation (RAG) is a common technique used in conjunction with these architectures, particularly to augment long-term memory with up-to-date or specialized external knowledge, thereby grounding the agent’s responses and reasoning in factual information.

Furthermore, memory augmentation strategies are being explored to enhance reasoning capabilities by extending the effective context available to LLMs beyond their fixed token limits [15]. This includes research into internal architectural modifications within the FMs themselves (such as augmented attention mechanisms or refined key-value caches, though these are less applicable to black-box API-based models) and, more commonly, the development of sophisticated external memory mechanisms. The AI design of agent memory is thus evolving to incorporate principles from information retrieval, database management, and even cognitive science (as seen with the Zettelkasten inspiration in A-MEM). This suggests a future where an agent’s memory is not merely a passive repository but an active, integral component of its reasoning and learning processes, dynamically shaping and being shaped by its experiences.

2.4 Emerging Paradigms in Cognitive Architectures

Beyond memory, broader innovations are shaping the cognitive architectures of agents. The field is witnessing a move towards more modular and composable designs, departing from monolithic LLM prompting towards engineered systems of specialized AI components.

Factored Agent Architectures, as proposed by Roth et al. (2025) [16], advocate for decomposing agent functionalities into specialized components. For example, a large LLM might serve as a high-level planner and in-context learner, while a smaller, more specialized language model could act as a dedicated “memorizer” for tool formats and outputs. This approach aims to mitigate observed trade-offs where a single agent’s proficiency in in-context learning might diminish as its role in memorization increases, allowing each component to be optimized for its specific cognitive function.

Drawing insights from computer systems, Mi et al. (2025) [17] propose building LLM agents based on established principles from computer architecture, particularly inspired by the von Neumann architecture. This paradigm emphasizes a structured framework with clearly defined, modular components for perception, cognition (processing), memory, tool interaction, action execution, and environmental interface. Such an approach aims to enhance the generality, scalability, and systematic design of LLM agents. Future directions under this paradigm include the exploration of multi-core agent systems (potentially analogous to multi-core CPUs, perhaps with different LLMs or specialized AI modules as “cores”) and the application of parallelization and pipelining techniques to agent workflows for improved efficiency.

The field of Automated Design of Agentic Systems (ADAS), highlighted by Hu et al. (2024) [18], represents another significant frontier. ADAS research focuses on developing methods to automatically create and optimize agentic system designs. This includes inventing novel building blocks—such as new prompting strategies, tool-use mechanisms, or entire workflows—and discovering effective ways to combine them. The “Meta Agent Search” algorithm, for example, employs a foundation model as a “meta agent” that iteratively designs and programs new agent architectures in code. This leverages the advanced coding and reasoning proficiency of modern FMs to explore the vast design space of agentic systems.

The development of these modular and often automatically generated architectures suggests that the AI design of agents is evolving into a form of “cognitive systems engineering.” The focus is increasingly on designing the interactions, information flows, and collaborative workflows between different AI modules, which are orchestrated by a central (or sometimes distributed) planning and reasoning mechanism. This trend also underscores a growing need for standardized interfaces and communication protocols between these diverse agent components to ensure interoperability and facilitate the construction of more complex and capable systems.

The symbiotic relationship between system-level support, like that offered by AIOS [6], and these advanced AI cognitive enhancements is becoming increasingly apparent. While the primary focus of this review is on AI design, it is crucial to recognize that robust system-level services—such as efficient context management (e.g., AIOS’s snapshot/restore capability for uninterrupted reasoning), scalable memory management (e.g., AIOS’s handling of extensive interaction histories for learning), and fair resource scheduling (critical for multi-agent coordination or enabling parallel thought processes within a single agent)—are not merely about optimizing system efficiency. Instead, they directly enable and sustain the complex cognitive functions that define modern AI agents. For instance, the ability of AIOS to snapshot and perfectly restore an agent’s intermediate generation state [6] is a direct enhancement to its reasoning reliability, preventing the AI from losing its “train of thought” and potentially diverging if interrupted. Similarly, AIOS’s memory management with LRU-K eviction allows an agent to learn from a far larger corpus of past experiences than could fit into a standard LLM context window, directly supporting its long-term learning and adaptation [6]. There is a necessary co-evolution: as the cognitive demands of AI agents grow (e.g., requiring longer reasoning chains, more extensive learning from history, or complex multi-agent interactions), the underlying system support must also become more sophisticated to provide the stable, scalable, and resource-managed environment these advanced AI processes require. This points to a future where AI agent research will increasingly need to integrate considerations of “AI-aware” system-level design to unlock further capabilities.

3. Reasoning and Planning in Virtual Agents

Effective reasoning and planning are cornerstone capabilities for intelligent virtual agents, enabling them to understand complex situations, formulate coherent strategies, and achieve goals. The advent of powerful foundation models has revolutionized these aspects of AI design, providing new mechanisms for agents to think and act.

3.1 Foundation Model-Based Reasoning

Large Language Models (LLMs) serve as the primary reasoning engine in a vast majority of modern agent architectures. They leverage their extensive pre-trained knowledge and sophisticated in-context learning abilities to interpret instructions, infer intentions, and generate logical steps towards a solution [3]. A key aspect of utilizing FMs for reasoning involves sophisticated prompting strategies designed to elicit and structure their cognitive processes.

Chain-of-Thought (CoT) prompting [3] is a foundational technique where the LLM is guided to generate a series of intermediate reasoning steps before producing a final answer. This explicit articulation of the “thought process” has been shown to significantly improve performance on complex reasoning tasks that require multiple logical deductions or calculations. Building upon CoT, Self-Consistency [19] further enhances robustness by sampling multiple diverse reasoning paths for the same problem and then selecting the most frequently occurring answer through a majority vote. Other advanced prompting techniques, such as self-refine (where the LLM critiques and improves its own previous outputs), selection-inference, Tree-of-Thoughts (ToT) [50], and Graph-of-Thoughts (GoT) [51], provide even more structured ways to guide and explore the LLM’s reasoning space, allowing for more deliberate and often more accurate problem-solving.

A significant trend in FM-based reasoning is the evolution from relying purely on inference-time prompting strategies (often termed inference scaling) towards dedicated training regimes designed to imbue models with better intrinsic reasoning abilities (learning-to-reason) [21]. This involves techniques like supervised fine-tuning (SFT) on datasets of reasoning trajectories and applying reinforcement learning (RL) to optimize reasoning processes based on feedback. Models like DeepSeek-R1 [22] exemplify this shift, showcasing advanced reasoning capabilities derived from such specialized training.

3.2 Advanced Reasoning Frameworks

Beyond prompting individual FMs, several encompassing frameworks have been developed to structure and enhance agent reasoning:

The ReAct framework (Reasoning and Acting), proposed by Yao et al. (2023) [20], offers a powerful paradigm by synergizing reasoning and acting. In ReAct, the LLM generates interleaved sequences of thought traces and actions. Thoughts involve reasoning about the current task state, analyzing information, and planning the next steps. Actions involve interacting with an external environment, such as querying a Wikipedia API or using a calculator. The observations resulting from these actions are then fed back into the agent’s context, informing subsequent reasoning steps. This iterative loop of reasoning, acting, and observing allows the agent to ground its reasoning in factual, up-to-date information from external sources, dynamically adjust its plans, and effectively handle exceptions or unexpected outcomes. System-level support, such as the tool manager and context manager in AIOS [6], could provide crucial infrastructure for ReAct-style agents by facilitating reliable API interactions and maintaining the agent’s state across these interleaved reasoning and action steps. The active interaction with external environments or knowledge sources in ReAct helps to mitigate issues like hallucination, which can occur when reasoning relies solely on an LLM’s internal, static knowledge.

The Reflexion framework by Shinn et al. (2023) [34] introduces a novel approach called verbal reinforcement learning. Instead of updating model weights, Reflexion agents learn by linguistically reflecting on task feedback. This feedback can be scalar (e.g., a success/failure signal), free-form language (e.g., an error message from a compiler), and can originate from external sources or be internally simulated by a critic model. The agent generates a textual self-reflection on this feedback, which is then stored in an episodic memory buffer. In subsequent trials for similar tasks, the agent uses these stored reflections to inform its decision-making and improve its strategy. This mechanism allows agents to learn from trial-and-error through in-context learning from their textualized experiences, without requiring expensive fine-tuning. The memory manager within a system like AIOS [6] could be instrumental in storing and managing the reflective texts for Reflexion agents, while its context manager could help preserve the agent’s state, including its reflections, if the process is interrupted.

The ability of agents to evaluate their own reasoning processes and outputs, learn from mistakes, and adapt—as seen in Reflexion and self-refinement techniques—is crucial for robustness. In multi-step tasks, initial errors can propagate and lead to complete failure. Frameworks that allow agents to reflect on feedback, critique their own outputs, or revise solutions based on past experiences are essential for building reliable autonomous systems. This is pushing AI agent design towards more human-like learning, where trial-and-error, reflection, and iterative improvement are central.

Other structured reasoning approaches like Tree-of-Thoughts (ToT) [50] enable LLMs to explore multiple parallel reasoning paths, akin to branches of a tree. The agent can then use self-evaluation heuristics or search algorithms (like breadth-first or depth-first search) to decide which paths to pursue further, which to prune, or when to backtrack. This allows for a more deliberate and systematic exploration of the problem space compared to the linear generation of CoT. Graph-of-Thoughts (GoT) [51] further generalizes this by modeling reasoning processes as arbitrary graphs, allowing for more complex operations like merging information from different reasoning branches and aggregating thoughts, leading to potentially more powerful and flexible reasoning.

3.3 Planning with LLMs

Planning, the process of devising a sequence of actions to achieve a goal, is a critical function for autonomous agents. LLMs are increasingly being used for this purpose, tasked with breaking down high-level objectives into manageable, actionable steps [3]. This often involves sophisticated task decomposition strategies, where a complex goal is divided into sub-goals. These strategies can range from single-path chaining, where sub-tasks are addressed sequentially (with dynamic adjustments based on feedback, as in ReAct), to multi-path tree expansion approaches (as in ToT), where multiple potential plans are explored. Effective planning also requires robust state tracking to maintain an understanding of the current situation and the ability to adapt plans based on new information or feedback.

However, planning with LLMs is not without its challenges [23]. Long-horizon planning, where a large number of steps are required to reach a goal, remains difficult due to issues like error propagation and the difficulty of maintaining logical consistency over extended sequences. Reasoning under uncertainty or with incomplete information is another significant hurdle, as many real-world scenarios do not offer the full observability often assumed in simpler planning problems. Multimodal planning, which requires integrating information from different sensory inputs (e.g., vision and language) to inform the planning process, is also an area of active development.

To address these challenges and drive progress, the field is seeing an evolution in benchmarks for LLM-based planning [24]. Early benchmarks often focused on text-based reasoning or simple PDDL (Planning Domain Definition Language) style problems. However, newer benchmarks are becoming increasingly complex and realistic. Examples include:

  • PlanBench [24]: Evaluates diverse aspects of planning, including generation, cost optimization, and replanning.
  • WebArena [24]: Tests agents on navigating and performing tasks on self-hosted websites, requiring long-horizon planning and interaction.
  • TravelPlanner [24]: Focuses on itinerary planning with constraints and the need for external tool use.
  • AgentBench [24]: A suite of environments evaluating LLM agents in multi-turn, open-ended contexts across diverse domains like operating systems and web shopping.

These evolving benchmarks are crucial for identifying the limitations of current LLM planners and guiding the development of more capable planning architectures.

3.4 Case-Based Reasoning (CBR) Integration

A promising direction for enhancing LLM agent reasoning, particularly in domain-specific contexts, is the integration of Case-Based Reasoning (CBR) [25]. CBR is an AI paradigm that solves new problems by retrieving and adapting solutions from similar past problems (cases) stored in a case library. Integrating CBR with LLM agents aims to combine the strong language comprehension and generative capabilities of LLMs with the explicit, experience-based knowledge of CBR. This hybrid approach can address several limitations of standalone LLMs, such as difficulties in handling specific, structured knowledge, retaining contextual memory across numerous interactions, and reducing the likelihood of hallucinations by grounding reasoning in concrete past examples.

The architectural components of a CBR-LLM agent typically include:

  • Case Representation: Defining how past problems and solutions are structured.
  • Case Library: A database or memory storing these past cases.
  • Retrieval Mechanisms: LLMs can be used to understand the new problem and query the case library for the most relevant past cases.
  • Reuse/Adaptation Mechanisms: The retrieved solution(s) from past cases are then adapted to fit the specifics of the current problem, a process where LLMs can assist in interpreting and modifying the solution components.
  • Revision: The newly proposed solution is evaluated for its effectiveness.
  • Retention: If successful (or even if it’s a notable failure providing learning), the new problem-solution pair is stored in the case library for future use.

The cognitive dimensions of CBR, such as self-reflection (evaluating the applicability of a past case or the quality of an adapted solution), introspection, and curiosity (proactively seeking new information or cases when existing ones are insufficient), can be further enhanced through integration with goal-driven autonomy (GDA) mechanisms. This allows agents to dynamically select their goals and monitor their performance against expectations, further refining their CBR processes. The synergy between the sub-symbolic pattern recognition of LLMs and the more symbolic, structured knowledge of CBR represents a significant step towards neuro-symbolic AI. This combination holds the potential to create agents that are not only more robust and adaptable but also more transparent in their decision-making, as their reasoning can be traced back to specific past experiences.

4. Learning and Adaptation in Virtual Agents

The ability of virtual agents to learn and adapt is paramount for their long-term utility and autonomy. Modern AI design focuses on enabling agents to improve their performance, refine their behaviors, and acquire new skills through various forms of feedback and experience. Foundation models play a crucial role in these learning processes, often acting as the core engine for interpreting feedback, generating reflective insights, or even modifying the agent’s own operational parameters.

4.1 Learning from Diverse Feedback Mechanisms

Agents learn and adapt by processing feedback from multiple sources, which helps them to refine their internal models and improve future decision-making.

Human feedback remains a cornerstone for aligning agent behavior with user expectations and societal norms [8]. This can range from explicit corrections and ratings provided by users to more implicit signals derived from observing user choices and interactions. LLM-based Human-Agent Systems (LLM-HAS) are an emerging area specifically focused on designing effective human-AI collaboration, where human input is integral to the agent’s learning and operational loop [8]. For instance, Seßler et al. (2025) explored the quality of LLM-generated feedback compared to teachers, finding comparable overall quality but limitations in contextual error explanation, suggesting that combining LLM efficiency with human nuance is beneficial [26].

Environmental feedback is another critical learning channel, especially for agents operating in dynamic settings [3]. Agents observe the outcomes of their actions—whether in simulated environments or the real world—and use this information to adjust their policies. This is the fundamental principle behind Reinforcement Learning (RL), where agents learn to maximize a reward signal obtained from environmental interactions.

AI-generated feedback and self-reflection represent a significant advancement, enabling agents to learn more autonomously. The Reflexion framework [34] allows agents to verbally reflect on task feedback signals (which can be from external sources or internally simulated by a critic model). These linguistic reflections are stored in an episodic memory buffer and used to guide decision-making in subsequent trials, effectively enabling a form of reinforcement learning without direct weight updates. Similarly, Reinforcement Learning from AI Feedback (RLAIF), a core component of Constitutional AI [27], uses an AI model to provide preference judgments on an agent’s responses, thereby guiding the RL process towards desired behaviors (e.g., harmlessness) with reduced reliance on human labeling.

The concept of a co-evolving world model, as proposed by Fang et al. (2024) [28], involves an auxiliary LLM that simulates the agent’s environment. This world model generates diverse training trajectories and enables look-ahead planning, allowing the primary agent to learn from these simulated experiences. A specific application of feedback-driven learning is Reinforcement Learning from Task Feedback (RLTF), central to the OpenAGI platform [29]. RLTF utilizes the overall success or failure of a task, or intermediate outcomes, as a reward signal to iteratively improve the LLM’s task-solving capabilities. This creates a self-improving AI feedback loop, particularly suited for open-ended and dynamically extending tasks where predefined reward functions may be difficult to specify.

The increasing sophistication of these feedback mechanisms, particularly those that are internalized or AI-generated, points to a trend where agents become more self-sufficient learners. While this enhances autonomy and potentially accelerates learning, it also underscores the critical importance of ensuring that the internal feedback mechanisms and any guiding “values” (as in Constitutional AI) are robustly aligned with overarching human objectives. System-level components, such as the context manager in AIOS [6], could play a vital role in reliably managing the complex internal states associated with these advanced learning and reflection processes.

4.2 Self-Improving Agents: Towards Autonomous Evolution

A frontier in agent AI design is the development of systems that can autonomously improve their own performance, operational strategies, or even their underlying architecture over time. This represents a deeper form of learning that goes beyond mere parameter updates or prompt refinement.

The Self-Improving Coding Agent (SICA), described by Robeyns et al. (2025) [30], exemplifies this trend. SICA is an agent system equipped with basic coding tools that can autonomously edit its own codebase. This includes modifying its prompts, the implementation of its tools, or its control logic. The self-modification process is driven by an LLM reflecting on the agent’s performance on benchmark tasks. This approach has demonstrated significant performance gains on coding benchmarks and showcases a data-efficient, non-gradient-based learning mechanism where the agent learns by directly changing its own structure and behavior.

The Automated Design of Agentic Systems (ADAS) framework, proposed by Hu et al. (2024) [18], takes this a step further. In ADAS, a “meta-agent” (itself an FM) is tasked with designing and programming new agent architectures. This meta-agent explores a search space of possible agent designs (represented as code) and iteratively discovers novel and potentially more performant agentic systems. This leverages the advanced reasoning and code generation capabilities of modern FMs to automate aspects of AI research and development itself.

The use of co-evolving world models [28] also contributes to autonomous evolution. As the agent learns and improves its policy for interacting with the (real or simulated) environment, the world model concurrently improves its ability to simulate that environment accurately. This synergistic improvement allows the agent to train on increasingly diverse and realistic synthetic data generated by the world model, and to perform more effective look-ahead planning, thus enabling sustained adaptability and helping to overcome performance plateaus often seen in simpler self-improvement cycles.

These approaches—where agents modify their own internal structure or generate new architectures—blur the line between the agent and the agent-designer. The advanced reasoning and code-generation capabilities of foundation models are the primary enablers for this level of self-modification. This implies a future where AI development might involve creating “seed” agents that then autonomously evolve and specialize their own cognitive architectures for specific tasks or domains, a trajectory with profound implications for both the pace of AI advancement and the challenges of ensuring safety and control.

4.3 Tool Learning and Adaptation

Modern virtual agents are not merely passive users of pre-defined tools; they are increasingly capable of learning how and when to use tools effectively, and in some cases, even adapting or creating tools.

Toolformer, by Schick et al. (2023) [3], demonstrates how LLMs can teach themselves to use external tools via APIs in a self-supervised fashion. The process involves the LLM sampling potential API calls within a given text, executing these calls, and then filtering them based on whether the inclusion of the API call and its result helps the model to better predict subsequent tokens in the text. The LLM is then fine-tuned on this augmented dataset, learning to invoke tools where they are most beneficial.

System-level support like the AIOS Tool Manager [6] can facilitate more robust tool use by AI agents. It provides functionalities such as standardized tool loading procedures, pre-execution parameter validation (to prevent tool crashes or erroneous calls), and mechanisms for resolving conflicts when multiple agents attempt to access tools with concurrency limitations. Such infrastructure simplifies the development of tool-using agents and enhances their reliability.

The “Control Plane as a Tool” design pattern [31] offers a modular approach to tool orchestration. It decouples the complex logic of tool management and routing from the agent’s core reasoning module. The agent interacts with a single, unified “control plane” tool, which then intelligently selects and invokes the appropriate specific tool from a larger library based on the agent’s intent, task requirements, or other contextual factors. This allows for dynamic tool selection, better governance over tool usage, and easier adaptation or expansion of the agent’s toolset. Furthermore, agents can learn and adapt their tool selection policies within such a control plane based on feedback or observed task success [4].

4.4 Continual and Lifelong Learning for Sustained Adaptation

For virtual agents to be truly intelligent and autonomous, they must be capable of continual or lifelong learning—adapting to new tasks, environments, and information over extended periods without catastrophically forgetting previously acquired knowledge [32]. This presents the classic stability-plasticity dilemma: the agent must be stable enough to retain existing knowledge and skills, yet plastic enough to acquire new ones.

The memory module is of paramount importance for lifelong learning agents [32]. It is responsible for storing and retrieving evolving knowledge, distinguishing between rapidly changing short-term context and more stable long-term knowledge. Sophisticated memory architectures, as discussed earlier (e.g., MemGPT, A-MEM), are crucial for managing the vast amounts of information an agent might accumulate over its lifetime and for retrieving relevant past experiences to inform current decisions.

Self-evolution mechanisms, where agents autonomously explore, learn from their environment, and adjust their behaviors based on feedback, are also key to lifelong learning [33]. This allows agents to provide increasingly personalized and effective responses as they gain more experience.

The optimization of LLM-based agents, as surveyed by Du et al. (2025) [31], encompasses various techniques that contribute to sustained adaptation. Parameter-driven methods, such as fine-tuning on new data or using reinforcement learning to adapt to new reward structures, directly modify the agent’s underlying model. Parameter-free methods, like evolving prompt engineering strategies or dynamically updating knowledge through Retrieval-Augmented Generation (RAG), allow agents to adapt their behavior without altering their core parameters. Both types of optimization are vital for enabling agents to handle long-term planning, interact effectively with dynamic environments, and make complex decisions over time—all hallmarks of lifelong learning.

The development of agents that are not just learning specific tasks but are learning how to learn or how to improve themselves (meta-learning agents) is a significant indicator of progress in this area. ADAS [18], which features a meta-agent designing better agents, and Reflexion [34], where agents learn a reflection process, are examples of this trend. This hierarchical learning approach could lead to more sample-efficient adaptation in the long run, as agents become better at generalizing their learning strategies to new and unseen tasks. This points towards a future where AI systems might autonomously drive their own research and development, a prospect that brings both immense potential and significant new challenges for oversight and alignment.

5. Foundation Models as Enablers of Advanced Agent AI

Foundation models (FMs) are the bedrock upon which modern advanced AI agents are constructed. Their inherent capabilities in language comprehension, reasoning, generation, and increasingly, multimodal processing, provide the raw cognitive power that agent architectures then channel towards purposeful, autonomous behavior. The continuous evolution of FMs directly translates into new possibilities for agent design and functionality. The capabilities of the chosen FM—be it context window length, reasoning acuity, or multimodal support—effectively define the performance ceiling for agents built upon them. This tight coupling means that breakthroughs in FM research often herald subsequent leaps in agent capabilities, establishing a synergistic relationship where progress in one domain fuels advancement in the other. However, this dependence also implies that inherent limitations or biases within FMs will inevitably propagate to the agents, making the responsible development and alignment of FMs an even more critical concern for the agentics field.

5.1 Deep Dive into Capabilities of Recent Foundation Models

Several recent foundation models have introduced capabilities particularly relevant to the design of sophisticated AI agents:

GPT-4 and its successors (e.g., GPT-4o) by OpenAI (2023) [10] set a high bar with human-level performance on numerous professional and academic benchmarks. Key strengths include robust multilingual proficiency, significantly improved instruction-following capabilities, and native multimodality (accepting both image and text inputs to produce text outputs). For agent design, GPT-4’s advanced reasoning is fundamental for planning and decision-making modules, while its multimodality allows agents to perceive and interact with richer, more complex environments, such as those encountered by GUI agents [9]. Efforts to reduce hallucinations and enhance safety properties are also crucial for deploying reliable agents. Despite these strengths, limitations such as a fixed context window (though significantly larger than previous models), the lack of continuous learning from experience post-training, and the potential for embedded biases remain areas of ongoing research and concern.

Llama 3 from Meta (2024) [35] and its subsequent iterations like Llama 3.1 [40], specialized versions such as Llama-Nemotron [36] and FoundationAI-SecurityLLM-Base-8B [37] represent signicant strides in open foundation models. Llama 3 boasts improved reasoning, enhanced coding capabilities (attributed to a training dataset with four times more code than Llama 2), and better instruction following. Its 8K token context window (with stated plans for expansion) and initial multilingual support (over 5% of its pretraining data being high-quality non-English text covering over 30 languages) are vital for versatile agent development. The focus on coding is particularly relevant for agents that need to generate or interact with software tools, such as the Self-Improving Coding Agent (SICA) [30]. The Llama-Nemotron series, specifically designed for efficient reasoning and featuring a dynamic reasoning toggle, further underscores the trend towards FMs optimized for agentic tasks. Meta’s introduction of safety tools like Code Shield also addresses the responsible deployment of code-generating agents.

Gemini 2.5 by Google (2025) [11] is marketed as a “thinking model,” emphasizing its capability to reason through complex problems before responding. It demonstrates state-of-the-art performance on challenging reasoning benchmarks (e.g., GPQA, AIME 2025) and advanced coding skills. Gemini 2.5’s native multimodality, allowing it to process text, audio, images, video, and even entire code repositories, combined with an exceptionally long context window (1 million tokens, with 2 million planned), makes it a powerful candidate for building sophisticated agents. Features like Deep Research within Gemini [11], which autonomously browses websites and synthesizes information into reports, exemplify direct FM application in agent-like workflows requiring multi-step planning and information integration.

General trends observable across these and other recent foundation models (as noted in [3]) include rapidly expanding context windows (e.g., Claude’s 200K tokens, Gemini 1.5’s 1M tokens [1]), continuous improvements in reasoning and planning capabilities (with models like Gemini explicitly designed as “thinking models” and DeepSeek-R1 focusing on reasoning [19]), increasingly proficient tool use and function calling [38], the standardization of multimodality, and the growing practice of specialization via fine-tuning for specific domains or agentic functions [1].

5.2 Foundation Models in Agent Modules

Foundation models are not just generic power sources; they are being integrated into specific functional modules within agent architectures:

  • Perception: MLLMs are crucial for processing and interpreting multimodal inputs, enabling agents to understand complex scenes or documents containing text, images, and other data types [9].
  • Planning & Reasoning: LLMs typically form the core of the planning and reasoning modules, responsible for task decomposition, strategy generation, logical inference, and decision-making under uncertainty [3].
  • Memory Control: In advanced memory systems like MemGPT [12] and A-MEM [13], LLMs can be actively involved in managing memory. This includes deciding what information to store, how to index or link it, when to retrieve it, and what to forget. For example, an LLM might generate contextual descriptions for new memories or determine relevant connections between existing memory nodes.
  • Action Generation: LLMs are used to generate a wide range of actions, from natural language responses in dialogues to executable code for invoking tools or APIs, and even sequences of commands for controlling robotic systems [3].
  • Self-Improvement & Reflection: A particularly advanced use of FMs within agents involves leveraging them for meta-cognitive tasks. LLMs can be prompted to reflect on the agent’s past performance, critique its own outputs or plans, identify areas for improvement, and even generate code to modify the agent’s own structure or behavior, as seen in frameworks like SICA [30] and Reflexion [34].

The paradoxical nature of foundation models is that their broad, general-purpose capabilities in language understanding, basic reasoning, and code generation are precisely what enable the creation of highly specialized and diverse agent architectures [18]. FMs act as a kind of “universal constructor” for agent cognition. The innovation in agent design is increasingly shifting towards how to best orchestrate these generalist FMs and augment them with specialized modules (memory systems, tool libraries, feedback loops) and structured workflows to achieve specific, advanced agent capabilities.

5.3 Impact of Scaling Laws and Emergent Abilities

The development of foundation models is heavily influenced by scaling laws, which describe how model performance on certain tasks improves predictably with increases in model size, dataset size, and computational resources used for training [10]. These scaling laws have been instrumental in guiding the development of increasingly large and capable FMs.

As FMs scale, they often exhibit emergent abilities—capabilities that are not present or predictable in smaller models but appear once a certain scale threshold is crossed [39]. Examples include advanced multi-step reasoning, in-context learning, and proficient code generation. These emergent capabilities are directly harnessed by agent architectures, forming the basis for many of their advanced cognitive functions. For instance, an agent’s ability to perform complex planning or learn from few-shot demonstrations is often a direct consequence of the emergent reasoning and in-context learning abilities of its underlying FM.

The development of Large Reasoning Models (LRMs) [39] further exemplifies this, where techniques like reinforcement learning and inference-time search are applied to already large FMs to specifically amplify their reasoning and self-reflection capabilities, which are critical for sophisticated agents. However, it is crucial to note that emergent abilities are not always beneficial. Undesirable or harmful emergent behaviors, such as increased tendencies for deception, manipulation, or developing misaligned goals, are also a concern as models become more powerful and autonomous [39]. This underscores the critical need for robust safety protocols, alignment techniques, and continuous monitoring in the design and deployment of agents built upon these highly capable FMs.

The increasing complexity and resource demands of FM-powered agents, especially in multi-agent scenarios or long-running tasks, also highlight the growing importance of system-level support. An “agent-aware” operating system, like the one envisioned by AIOS [6], becomes critical. By managing access to LLM “cores,” handling context efficiently, and providing scalable memory and storage solutions, such a system can ensure that the cognitive processes of advanced AI agents are not bottlenecked by underlying resource constraints. This suggests an intertwined future where the design of agent AI and the design of the systems hosting them co-evolve to support increasingly sophisticated intelligent behaviors.

6. AI Design in Multi-Agent Systems (MAS)

Multi-Agent Systems (MAS) represent a significant frontier in AI, where multiple intelligent agents, often powered by LLMs, interact to solve problems or simulate complex dynamics that may be intractable for a single agent [3]. The AI design of MAS focuses on enabling effective collaboration, coordination, and collective intelligence.

6.1 Architectures for Collaboration and Coordination

The architecture of an LLM-based MAS dictates how agents interact, share information, and coordinate their actions. Tran et al. (2025) provide a framework for characterizing collaboration mechanisms along several key dimensions [41]:

  • Actors: The individual agents involved, each potentially with unique capabilities, roles, and underlying foundation models.
  • Types of Interaction:
    • Cooperation: Agents share common goals and work synergistically, often sharing rewards [42].
    • Competition: Agents have conflicting goals, operating in a zero-sum or near-zero-sum environment.
    • Coopetition: A mix where agents may cooperate on some aspects while competing on others, reflecting many real-world scenarios.
  • Structures:
    • Centralized: A single orchestrator or controller agent manages task allocation, information flow, and decision integration for other agents [3]. This “meta-agent” or “planner LLM” often requires advanced reasoning to understand the overall goal, decompose it effectively, and assign sub-tasks to the most suitable worker agents, managing dependencies and integrating results.
    • Decentralized/Distributed: Agents interact directly in a peer-to-peer fashion, often self-organizing without a central authority.
    • Hierarchical: Agents are organized in layers, with higher-level agents managing or coordinating teams of lower-level agents.
  • Strategies:
    • Role-Based: Agents are assigned specific roles (e.g., planner, executor, critic, domain expert) based on their specialized capabilities or prompted personas [3]. This division of labor is a common trend, moving from homogeneous MAS to heterogeneous teams of specialized agents.
    • Model-Based: Strategies might be learned or dynamically adapted based on the state of the environment and the behavior of other agents.
  • Coordination Protocols/Communication: The mechanisms agents use to exchange information, negotiate, reach consensus, and synchronize their actions. This can involve structured messages, natural language dialogue, or shared memory/knowledge bases [41].

System-level support, such as that provided by AIOS [6], can be particularly beneficial for MAS. The AIOS scheduler can help orchestrate resource access (e.g., to LLM cores) for multiple contending agents, while its access manager can regulate inter-agent data sharing based on predefined privileges. Such functionalities contribute to more efficient, secure, and predictable coordination within a MAS.

6.2 Frameworks for MAS Development

Several frameworks have emerged to facilitate the development and deployment of LLM-based MAS:

  • AutoGen (Wu et al., 2023) [43] provides a versatile platform for creating multi-agent conversation systems. Agents in AutoGen can be LLM-backed, human-backed (allowing for human-in-the-loop interaction), or tool-backed. They collaborate by exchanging messages to achieve tasks such as code generation, data analysis, or complex question answering. AutoGen supports flexible control flows, programmable via both natural language instructions and code, and features an auto-reply mechanism that drives the conversation forward.
  • MetaGPT (Hong et al., 2023) [44] introduces a meta-programming approach that encodes human-like Standardized Operating Procedures (SOPs) into prompt sequences for multi-agent collaboration. It employs an “assembly line” paradigm, assigning distinct roles (e.g., Product Manager, Architect, Engineer, QA Tester) to different agents. This structured workflow, with clearly defined responsibilities and handovers, is particularly effective for complex, multi-stage tasks like software development, aiming to improve coherence and reduce errors through systematic verification of intermediate outputs. The specialization of roles in MetaGPT reflects a broader trend: complex tasks often benefit from a division of labor, where different FMs or agent configurations optimized for specific sub-tasks contribute to the overall solution.
  • Other notable frameworks include CAMEL (Communicative Agents for “Mind” Exploration of Large Language Model Society) by Li et al. (2023) [48], which focuses on simulating complex social interactions and exploring emergent behaviors in LLM societies, and AgentVerse by Chen et al. (2023) [49], designed to facilitate broader multi-agent collaboration.

6.3 Task Decomposition, Role Assignment, and Coordination Strategies

Effective functioning of MAS hinges on intelligent task decomposition, appropriate role assignment, and robust coordination strategies. LLMs themselves are increasingly used as the orchestrators or planners responsible for these critical functions [42].

In centralized orchestration, a dedicated LLM agent acts as a controller, breaking down the main task into sub-tasks and assigning them to specialized worker agents based on their capabilities [3]. The orchestrator then monitors progress and integrates the results. A semi-decentralized planner method involves an LLM generating a high-level plan, which is then disseminated to individual executor LLMs that independently generate actions to fulfill their part of the plan [42].

Role-playing, where LLMs are prompted to adopt specific personas or expert roles (e.g., “you are a senior software architect” or “you are a skeptical reviewer”), is a widely used technique to elicit specialized contributions and diverse perspectives within the MAS [3]. The ability of LLMs to convincingly adopt these roles through prompting is a key enabler of this strategy. Communication protocols are vital for coordination. These can range from highly structured message formats and API calls to more flexible natural language dialogues between agents or interactions mediated through shared knowledge bases or memory systems [41]. The design of these protocols significantly impacts the efficiency and effectiveness of collaboration.

6.4 Emergent Behavior and Collective Intelligence

One of the most fascinating and challenging aspects of LLM-based MAS is the potential for emergent behavior and collective intelligence [25]. When multiple autonomous agents interact within a shared environment, complex group dynamics, social norms, and large-scale patterns can arise that were not explicitly programmed into the individual agents. Researchers are using LLM-based MAS to simulate and study phenomena such as cultural evolution, the formation of social norms, opinion dynamics in populations, and organizational decision-making [25].

The concept of an “Economy of Minds” (EOM), as theorized by Zhuge et al. (2023) [42], suggests that future MAS could feature LLM agents that self-manage and allocate resources (such as computational power or access to specialized tools) based on an internal monetary or utility system, aiming to optimize collective task performance and maximize overall rewards.

However, emergent behavior is a double-edged sword. While it holds the promise of novel problem-solving capabilities and insights into complex systems, it also introduces significant challenges in terms of predictability, control, and alignment [39]. Unintended collective behaviors, such as harmful collusion, amplification of biases present in the underlying LLMs, or actions that deviate from overall human goals, are serious concerns. This necessitates robust governance mechanisms, ethical guidelines encoded into agent objectives (extending concepts like Constitutional AI [27] to multi-agent contexts), and sophisticated methods for monitoring, understanding, and steering collective agent behavior. System-level controls, like the AIOS access manager [6], which manages resource access between agents, represent a very basic but important step in this direction, ensuring that inter-agent interactions occur within defined boundaries. The design of truly effective and safe MAS will require a deep understanding of not just individual agent capabilities but also the “social laws” and interaction protocols that govern the collective, drawing insights from fields like game theory, sociology, and complex systems theory.

6.5 Alignment in Multi-Agent Systems

Achieving alignment in MAS is a multifaceted challenge that extends beyond aligning individual agents with human values. Duque et al. (2024) and Li et al. (2024b) emphasize that alignment in MAS is a dynamic, interaction-dependent process, profoundly shaped by the social context—be it collaborative, cooperative, or competitive—in which the agents operate [45]. Key challenges include reconciling individual agent objectives with collective goals, ensuring that the MAS as a whole adheres to human values, and aligning system behavior with the nuanced preferences and intentions of users. Social dynamics within the MAS can lead to emergent misalignments, such as power-seeking behaviors, divergence of values among agents, manipulation, or harmful collusion, even if individual agents appear aligned in isolation. Addressing these issues requires holistic approaches that consider the interplay of all incentives and constraints within the MAS, alongside the development of dedicated simulation environments and metrics to study and foster robust multi-agent alignment.

7. Challenges and Future Directions in Virtual Agent AI Design

Despite rapid advancements, the AI design of virtual agents faces significant challenges that outline the trajectory for future research. These challenges span enhancing core cognitive capabilities, ensuring safety and alignment, and scaling agent intelligence effectively.

7.1 Enhancing Robustness, Reliability, and Safety

A primary concern is the robustness and reliability of LLM-based agents. Hallucination mitigation remains a persistent issue, where agents generate factually incorrect or nonsensical information. Current approaches to combat this include Retrieval-Augmented Generation (RAG) to ground responses in external knowledge [14], integrating fact-verification modules, and employing structured reasoning frameworks like Case-Based Reasoning (CBR) which can provide solutions based on verified past experiences [25].

In multi-step tasks, errors made in early stages of a plan can propagate and derail the entire process. Mechanisms for self-correction [7] and more resilient planning algorithms are crucial. The AIOS framework’s context management, with its snapshot and restoration capabilities [6], can contribute to robustness by allowing an agent to revert to a previous correct state if an error is detected in its reasoning or action sequence.

Safety in action execution is paramount, especially as agents interact more directly with digital systems (e.g., GUI agents [9]) or physical environments. Ensuring that an agent’s actions are safe, predictable, and aligned with human intentions is a critical research area [10]. Constitutional AI [27] offers a promising approach by training AI assistants to be harmless through self-improvement guided by an explicit set of ethical principles or rules (a “constitution”). This involves the AI critiquing its own responses against these principles and using Reinforcement Learning from AI Feedback (RLAIF) where an AI model provides preference feedback for harmlessness, directly shaping the agent’s decision-making to adhere to these encoded guidelines.

Furthermore, alignment in Multi-Agent Systems (MAS) presents unique challenges, requiring strategies to ensure that both individual agent objectives and emergent collective behaviors align with human values and overall system goals [45]. The increasing autonomy and capability of agents necessitate a greater focus on what can be termed the “alignment tax”—the significant and growing effort required to ensure these sophisticated systems operate safely and in accordance with human values. This is evident in the dedicated research streams focusing on Constitutional AI, safety protocols for MAS, and the development of ethical guidelines. As agents become capable of more complex, long-horizon actions with real-world impact, the design process must incorporate rigorous safety engineering and alignment verification. This may lead to specialized roles within development teams, such as “AI ethicists” or “AI safety engineers,” and could mean that the deployment of raw capabilities is paced by the development of corresponding safety and alignment measures.

7.2 Achieving True Long-Term Reasoning and Scalable Context Management

While foundation models boast increasingly large context windows, achieving true long-term reasoning and managing context scalably over extended interactions or across vast knowledge bases remains a frontier [46]. Current FMs can still struggle with very long-range dependencies. Advanced memory architectures like MemGPT [12], which employs hierarchical storage and LLM-directed paging, and A-MEM [13], which allows for agentic construction and evolution of a linked knowledge network, represent significant steps forward. However, scaling these systems to support truly lifelong learning and the seamless integration of vast, ever-changing knowledge remains an active area of research.

Techniques like Fine-Grained Optimization (FGO) [46] for LLM-based optimizers are also being developed to address context window overflow issues when training or optimizing agents using large datasets of execution trajectories, by dividing tasks into manageable subsets and progressively merging optimized components. The AIOS paper [6], by proposing OS-level management of “cognitive resources” such as LLM core access, context, and memory, implicitly addresses this scalability. As agents become more complex, perhaps evolving into multi-agent systems or requiring persistent, very long-term memory, the efficient allocation and management of these underlying cognitive resources will be critical for both performance and the ability to execute advanced AI functions. This suggests a future where agent architectures might explicitly include “cognitive resource managers” to optimize the use of available FM compute, context space, and memory bandwidth, moving beyond just designing the cognitive functions themselves into designing how these functions are efficiently orchestrated under constraints.

7.3 Improving Learning Efficiency and Generalization

Enhancing the efficiency and generalization of learning in virtual agents is crucial for their practical deployment. This includes improving sample efficiency, reducing the amount of data or interactive experience an agent needs to learn a new task or adapt its behavior. The Self-Improving Coding Agent (SICA) [30], which learns by modifying its own code based on performance feedback, is an example of a data-efficient, non-gradient-based learning approach. Similarly, verbal reinforcement learning frameworks like Reflexion [34] aim for efficient learning from linguistic feedback.

Transfer learning and generalization—enabling agents to effectively apply knowledge and skills learned in one context to novel tasks and environments—is another key objective. Overcoming stagnation in self-improvement cycles, where agents reach a performance plateau despite continued autonomous learning, is also important. Co-evolving world models [28], which provide diverse simulated experiences and enable look-ahead planning, have been proposed to address this.

7.4 The Path Towards More Autonomous and Generally Capable Agents

The long-term vision for virtual agents involves increasing their autonomy and general capabilities. The concept of NGENT (Next-Generation AI Agents) [47] envisions future agents that integrate abilities across multiple domains—text, vision, robotics, reinforcement learning, and even emotional intelligence—within unified frameworks, moving closer to Artificial General Intelligence (AGI).

Automated Agent Design (ADAS) [18], where AI systems are used to design and program new, potentially more powerful agent architectures, could significantly accelerate progress in this direction. Furthermore, enhancing tool creation and use capabilities, where agents not only use existing tools but can also dynamically create or compose new tools to solve novel problems (e.g., the WorldAPIs approach by Liu et al. (2024) mentioned in [24]), will be vital for increasing agent versatility.

7.5 Ethical Considerations in Advanced AI Agent Design

As AI agents become more autonomous, capable, and integrated into various aspects of life, ethical considerations become increasingly critical [10]. Issues such as the potential for bias amplification (where agents perpetuate or even magnify biases present in their training data or underlying FMs), misuse for malicious purposes, ensuring accountability for agent actions (especially complex in MAS [45]), the societal impact of job displacement, and the necessity of maintaining meaningful human oversight are paramount.

The development of robust evaluation frameworks that assess not only performance but also safety, fairness, transparency, and reproducibility is essential [38]. The increasing sophistication of AI agents naturally leads to a co-evolution of evaluation methodologies. As agents develop new capabilities, existing benchmarks may become saturated or fail to capture the nuances of these advanced skills. This, in turn, drives the creation of new, more challenging benchmarks designed to test specific aspects like long-horizon planning (e.g., PlanBench [24]), complex web navigation (e.g., WebArena [24]), or nuanced self-reflection (e.g., Reflection-Bench [38]). Success on these new benchmarks then often spurs further research focused on enhancing the targeted capabilities. This iterative feedback loop between agent capability development and evaluation methodology refinement is a crucial driver of progress in the field, ensuring that research remains focused on pushing the boundaries of what AI agents can achieve.

8. Conclusion

The AI design of virtual agents is in a period of dynamic evolution, largely propelled by the rapid advancements in foundation models. This review has traced the trajectory from earlier agent concepts to the sophisticated, AI-centric systems of today, highlighting key architectural trends, reasoning and planning mechanisms, learning paradigms, and the integral role of FMs.

Modern agents are increasingly characterized by their modular cognitive architectures, with FMs serving as powerful cognitive engines augmented by specialized modules for perception, memory, and action. Innovations in memory systems, such as MemGPT and A-MEM, are tackling the critical challenge of limited context windows, enabling agents to maintain long-term coherence and learn from extended histories. Reasoning frameworks like ReAct and Reflexion, along with structured approaches such as Tree-of-Thoughts and Case-Based Reasoning integration, are endowing agents with more robust, grounded, and adaptable thinking processes.

A significant shift is observable towards agents that can learn and adapt more autonomously. Self-improving agents that can modify their own code or architectures, like SICA and those emerging from ADAS research, alongside agents that learn from diverse forms of feedback (human, environmental, self-generated, AI-generated), are pushing the boundaries of machine learning. The ability to learn to use tools effectively, as seen in Toolformer, and the development of systems for lifelong learning, are further testaments to this trend. In the realm of Multi-Agent Systems, AI design is focusing on sophisticated collaboration and coordination strategies, with frameworks like AutoGen and MetaGPT enabling complex group behaviors and task execution through specialized roles and structured communication.

The collective evidence from recent research, particularly from late 2024 and 2025, points towards a paradigm shift. The field is moving beyond designing agents with fixed cognitive architectures towards creating systems capable of autonomously evolving their own cognitive processes, knowledge representations, and even their fundamental architectural blueprints. This transformative potential is primarily unlocked by the advanced generative, reasoning, and meta-cognitive capabilities of the latest foundation models. As agents themselves begin to contribute to their own advancement, the pace of AI development could accelerate further.

However, this progress is intrinsically linked with significant challenges. Ensuring the robustness, reliability, safety, and ethical alignment of these increasingly autonomous and capable agents is paramount. The “alignment tax”—the substantial effort required to ensure that advanced agents operate in accordance with human values—will likely become a more prominent factor in the development lifecycle. The co-evolution of agent capabilities and the methodologies to evaluate them will continue to be a critical driver of progress, pushing researchers to develop more nuanced and comprehensive benchmarks that assess not only performance but also safety, fairness, and real-world applicability.

Furthermore, as agents become more complex and are deployed at scale, particularly in multi-agent configurations, the efficient management of their “cognitive resources”—such as access to foundation models, context space, and memory bandwidth—will necessitate more sophisticated, “agent-aware” system-level support, as hinted at by frameworks like AIOS. The journey towards truly intelligent virtual agents is ongoing. The path forward will require continued interdisciplinary research, fostering a deeper understanding of both the immense potential and the inherent complexities of designing AI that can reason, learn, and act autonomously and responsibly in an increasingly intricate world.

References

  1. Anthropic. (2024). The Claude 3 model family: Opus, Sonnet, Haiku [Technical report]. Anthropic. https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf
  2. Krishnan, N., et al. (2025). AI agents: Evolution, architecture, and real‑world applications (arXiv:2503.12687). arXiv. https://arxiv.org/abs/2503.12687
  3. Schick, T., et al. (2023). Toolformer: Language models can teach themselves to use tools (arXiv:2302.04761). arXiv. https://arxiv.org/abs/2302.04761
  4. Liang, G., & Tong, Q. (2025). LLM‑powered AI‑agent systems and their applications in industry (arXiv:2505.16120). arXiv. https://arxiv.org/abs/2505.16120
  5. Luo, J., et al. (2025). Large Language‑Model agent: A survey on methodology, applications and challenges (arXiv:2503.21460). arXiv. https://arxiv.org/abs/2503.21460
  6. Mei, K., et al. (2025). AIOS: LLM‑Agent operating system (arXiv:2403.16971). arXiv. https://arxiv.org/abs/2403.16971
  7. Casper, S., et al. (2025). The AI‑Agent Index (arXiv:2502.01635). arXiv. https://arxiv.org/abs/2502.01635
  8. Zou, H. P., et al. (2025). A survey on Large‑Language‑Model‑based human‑agent systems (arXiv:2505.00753). arXiv. https://arxiv.org/abs/2505.00753
  9. Wang, S., et al. (2024). GUI agents with foundation models: A comprehensive survey (arXiv:2411.04890). arXiv. https://arxiv.org/abs/2411.04890
  10. OpenAI. (2023). GPT‑4 technical report (arXiv:2303.08774). arXiv. https://arxiv.org/abs/2303.08774
  11. Google DeepMind. (2025, March 25). Gemini 2.5: Our most intelligent AI model [Blog post]. Google AI Blog. https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/
  12. Packer, C., et al. (2023). MemGPT: Towards LLMs as operating systems (arXiv:2310.08560). arXiv. https://arxiv.org/abs/2310.08560
  13. Xu, W., et al. (2025). A‑MEM: Agentic memory for LLM agents (arXiv:2502.12110). arXiv. https://arxiv.org/abs/2502.12110
  14. Xu, M., et al. (2025). Forewarned is fore‑armed: A survey on LLM‑based agents in autonomous cyber‑attacks (arXiv:2505.12786). arXiv. https://arxiv.org/abs/2505.12786
  15. Chen, Z., et al. (2025). A survey of scaling in Large‑Language‑Model reasoning (arXiv:2504.02181). arXiv. https://arxiv.org/abs/2504.02181
  16. Roth, N., et al. (2025). Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Use(arXiv:2503.22931). arXiv. https://arxiv.org/abs/2503.22931
  17. Mi, Y., et al. (2025). Building LLM agents by incorporating insights from computer systems (arXiv:2504.04485). arXiv. https://arxiv.org/abs/2504.04485
  18. Hu, S., Lu, C., & Clune, J. (2024). Automated design of agentic systems (arXiv:2408.08435). arXiv. https://arxiv.org/abs/2408.08435
  19. Wang, X., et al. (2022). Self‑consistency improves chain‑of‑thought reasoning in language models (arXiv:2203.11171). arXiv. https://arxiv.org/abs/2203.11171
  20. Yao, S., et al. (2023). ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR 2023). https://openreview.net/forum?id=WE_vluYUL-X
  21. Ke, Z., et al. (2025). A survey of frontiers in LLM reasoning: Inference scaling, learning to reason, and agentic systems (arXiv:2504.09037). arXiv. https://arxiv.org/abs/2504.09037
  22. DeepSeek‑AI Team. (2025). DeepSeek‑R1: Incentivizing reasoning capability in LLMs via reinforcement learning (arXiv:2501.12948). arXiv. https://arxiv.org/abs/2501.12948
  23. Xi, Z., et al. (2023). The rise and potential of Large‑Language‑Model‑based agents: A survey (arXiv:2309.07864). arXiv. https://arxiv.org/abs/2309.07864
  24. Liu, X., et al. (2023). AgentBench: Evaluating LLMs as agents (arXiv:2308.03688). arXiv. https://arxiv.org/abs/2308.03688
  25. Hatalis, K., Christou, D., & Kondapalli, V. (2025). Review of case‑based reasoning for LLM agents: Theoretical foundations, architectural components, and cognitive integration (arXiv:2504.06943). arXiv. https://arxiv.org/abs/2504.06943
  26. Seßler, K., et al. (2025). Towards adaptive feedback with AI: Comparing the feedback quality of LLMs and teachers on experimentation protocols (arXiv:2502.12842). arXiv. https://arxiv.org/abs/2502.12842
  27. Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI feedback (arXiv:2212.08073). arXiv. https://arxiv.org/abs/2212.08073
  28. Fang, T., et al. (2025). WebEvolver: A co‑evolving world model for self‑improving web agents (arXiv:2504.21024). arXiv. https://arxiv.org/abs/2504.21024
  29. Ge, Y., et al. (2023). OpenAGI: When LLM meets domain experts (arXiv:2304.04370). Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2304.04370
  30. Robeyns, M., et al. (2025). Self‑improving coding agents through autonomous code edits (arXiv:2504.15228). arXiv. https://arxiv.org/abs/2504.15228
  31. Du, S., et al. (2025). A survey on the optimization of Large‑Language‑Model‑based agents (arXiv:2503.12434). arXiv. https://arxiv.org/abs/2503.12434
  32. Biesialska, M., et al. (2020). Continual lifelong learning in Natural Language Processing: A survey (arXiv:2012.09823). arXiv. https://arxiv.org/abs/2012.09823
  33. Zhang, Z., et al. (2024). A survey on the memory mechanism of Large‑Language‑Model‑based agents (arXiv:2404.13501). arXiv. https://arxiv.org/abs/2404.13501
  34. Shinn, N., et al. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2303.11366
  35. Meta AI. (2024, April 18). Introducing Meta Llama 3: The most capable openly available LLM to date [Blog post]. Meta AI Blog. https://ai.meta.com/blog/meta-llama-3/
  36. Bercovich, A., et al. (2025). Llama‑Nemotron: Efficient reasoning models (arXiv:2505.00949). arXiv. https://arxiv.org/abs/2505.00949
  37. Robeyns, P., et al. (2025). Llama‑3.1‑FoundationAI‑SecurityLLM‑Base‑8B technical report (arXiv:2504.21039). arXiv. https://arxiv.org/abs/2504.21039
  38. Yehudai, A., et al. (2025). Survey on evaluation of LLM‑based agents (arXiv:2503.16416). arXiv. https://arxiv.org/abs/2503.16416
  39. Berti, L., et al. (2025). Emergent abilities in Large‑Language Models: A survey (arXiv:2503.05788). arXiv. https://arxiv.org/abs/2503.05788
  40. Grattafiori, A., et al. (2024). The Llama 3 herd of models (arXiv:2407.21783). arXiv. https://arxiv.org/abs/2407.21783
  41. Tran, K.‑T., et al. (2025). Multi‑agent collaboration mechanisms: A survey of LLMs (arXiv:2501.06322). arXiv. https://arxiv.org/abs/2501.06322
  42. Amayuelas, A., et al. (2025). Self‑resource allocation in multi‑agent LLM systems (arXiv:2504.02051). arXiv. https://arxiv.org/abs/2504.02051
  43. Wu, Q., et al. (2023). AutoGen: Enabling next‑gen LLM applications via a multi‑agent conversation framework (arXiv:2308.08155). arXiv. https://arxiv.org/abs/2308.08155
  44. Hong, S., et al. (2025). MetaGPT: Meta‑programming for a multi‑agent collaborative framework. International Conference on Learning Representations (ICLR 2024) (arXiv:2308.00352). https://arxiv.org/abs/2308.00352
  45. Carichon, F., et al. (2024). The coming crisis of multi‑agent misalignment: AI alignment must be a dynamic and social process (arXiv:2506.01080). arXiv. https://arxiv.org/abs/2506.01080
  46. Liu, J., et al. (2025). Divide, optimize, merge: Fine‑grained LLM‑agent optimization at scale (arXiv:2505.03973). arXiv. https://arxiv.org/abs/2505.03973
  47. Li, Z., et al. (2025). NGENT: Next‑generation AI agents must integrate multi‑domain abilities to achieve artificial general intelligence (arXiv:2504.21433). arXiv. https://arxiv.org/abs/2504.21433
  48. Li, G., et al. (2023). CAMEL: Communicative agents for “mind” exploration of Large Language Model society (arXiv:2303.17760). Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2303.17760
  49. Chen, W., et al. (2023). AgentVerse: Facilitating multi‑agent collaboration and exploring emergent behaviors (arXiv:2308.10848). arXiv. https://arxiv.org/abs/2308.10848
  50. Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models (arXiv:2305.10601). Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2305.10601
  51. Besta, M., et al. (2023). Graph of Thoughts: Solving Elaborate Problems with Large Language Models (arXiv:2308.09687). arXiv. https://arxiv.org/abs/2308.09687