PhD Topic: A Methodology for the Analysis and Design of Multi-Agent Systems Enhanced by Large Language Models (LLM)

Context

LLMs are models that have been trained on large datasets and have learned the subtleties of language. They demonstrate high performance in natural language processing (NLP) and comprehension, demonstrating human-like language production and understanding capabilities. LLMs have excelled not only in language understanding and production, but also in more complex cognitive tasks such as reasoning, information extraction and problem solving. These capabilities demonstrate the potential of LLMs to achieve human-like intelligence. These developments have opened up many new opportunities to improve and enrich the workflow in autonomous systems (Naveed et al., 2023).

Thanks to their strong language understanding, complex task reasoning and common sense understanding capabilities, LLM-based autonomous agents have the potential to impact many fields. In the field of psychology, LLM-based agents can be used to run simulation experiments, provide mental health support ((Aher et al., 2023), (Akata et al., 2023), (Ma et al., 2023), (Ziems et al., 2024)). For example, Aher et al. (Aher et al., 2023) assigned different profiles to LLMs and had them complete psychology experiments. From the results, it was found that LLMs were able to produce results consistent with those obtained from studies involving human participants.

There are also studies where LLM-based agents have the ability to conduct experiments independently and support scientists in their research projects ((Boiko et al., 2023), (Bran et al., 2023)). Boiko et al. (Boiko et al., 2023) introduced an innovative agent system that automates the design, planning and execution of scientific experiments using LLMs. Once the experimental objectives are entered, the tool, called ChemCrow, provides recommendations for experimental procedures and points out potential safety risks associated with the proposed experiments.

This potential of LLMs offers the opportunity to significantly increase the capabilities of agents. In particular, LLMs can strengthen agents’ capacities to reason, acquire knowledge and adapt to new observations. For example, LLMs can guide agents through complex decision-making processes, provide more creative and flexible solutions to problems faced by agents, and enable agents to communicate more effectively with each other. In this context, LLMs are becoming an important building block in building powerful and flexible agents ((Yao et al., 2022), (Xi et al., 2023), (Wang et al., 2023b)).

Considering the tasks that LLMs can be used for and the increasing task complexity, one way to increase the power of agents is to have multiple agents cooperating with each other (Wu et al., 2023). Studies have shown that agents can encourage each other to think differently (Lian et al., 2023), increase factual and logical accuracy (Du et al., 2023), and provide validation (Wu et al., 2023). In addition, many studies have been conducted both in academia and industry to utilize the power of LLMs for agents, and various methods, analyses and tools have emerged ((Wu et al., 2023), (Qian et al., 2023), (Hong et al., 2023)). It is understood that LLM-supported agents can create much more effective systems by collaborating, sharing information and working together to achieve common goals. In this context, it is clear that the integration of MASs with LLM will play a key role in the development of future autonomous systems.

Problematic

Although there is a growing body of work for LLM-enhanced MAS applications, it has been found that these studies do not provide MAS development methodologies within the scope of AOSE. Researchers have mostly determined and applied their own analysis methods. Wu et al. (Wu et al., 2023) introduced the AutoGen framework, which aims to simplify multi-agent workflows using LLM. This approach aims to maximize the reusability of agents. In addition, the AutoGen-based application presented in the study is evaluated according to various criteria such as Ease of Use, Modularity, Programmability, Allowing Human Participation and Agent Interactions and it is stated how it provides advantages in line with these criteria.

In the analysis and evaluation phases of existing frameworks and tools, they acted independently from standard processes and put forward various criteria in line with their own knowledge. However, it was found that these criteria did not cover all MAS development processes defined in the field of AOSE. In addition, the common point emphasized in the studies is the lack of a general methodology that covers both MAS and LLM evaluation processes. While the methods available in the literature are effective for specific scenarios and use cases, they do not provide a comprehensive and standardized methodology. The work of Ricci et al. (Ricci et al., 2024) addresses this issue and discusses the lack of usable engineering abstractions, the gap between cognitive agent-based concepts and behavioral models of LLMs, and the resulting inability to methodically engineer complex agent-based applications, and presents an approach that they believe would be useful for people to understand, design and control agents and multi-agent systems. The lack of abstraction at the engineering level and the absence of a specific methodology leads to inconsistencies and integration difficulties in the research and implementation processes.

Subject

In this dissertation, it is aimed to evaluate the existing studies on LLM-enhanced MAS in detail, to identify their shortcomings in the analysis and design phase, and to present a new AOSE methodology that covers all LLM-based MAS development processes. Since there are no studies in this direction yet, this study is considered to be a pioneering study that can guide future research and applications.

This methodology will examine the effects of LLM integration on MAS from both theoretical and practical perspectives, and by benchmarking with existing studies, it will reveal its advantages and disadvantages and provide recommendations to guide future research and practice.

This study aims to provide a comprehensive roadmap on how LLM-enhanced MAS can be designed, implemented and evaluated as more efficient, scalable and flexible systems. It will also focus on the applications of such systems in different domains, their performance metrics and the challenges faced in real-world scenarios. In this way, the potential benefits and limitations of the combination of MAS and LLM technologies will be better understood and a solid foundation for future research will be established.

References