Large language models (LLMs) have rapidly gained tremendous popularity since their inception, with remarkable performance in natural language processing (NLP) tasks, as well as in many other domains beyond NLP. However, whether the application of LLMs in complex numerical optimization problems is genuinely beneficial remains unexplored.
In an article published in IEEE Computational Intelligence Magazine, researchers provide a comprehensive evaluation of LLMs in optimization, encompassing both discrete and continuous optimization problems to evaluate their effectiveness and unique contributions in this field. This study provides a thorough review to explore the full capabilities of LLMs in various optimization tasks.
LLMs and Optimization
In the field of optimization, LLMs are applied incrementally. With a limited scope and depth, as well as a small number of explored tasks, it remains unclear whether the current trend of applying LLMs to non-linguistic optimization problems is driven by their application or if they genuinely offer distinct advantages in solving these problems.
In this study, LLMs are examined across a range of classic complex optimization challenges, spanning both discrete and continuous domains. In addition, the mechanisms underlying the use of LLMs for solving optimization problems are investigated by analyzing their fundamental properties in relation to traditional algorithms and by exploring novel approaches.
Illustration of evaluation on applying popular LLMs for complex optimization.
To assess the effectiveness of these models in addressing optimization problems, the researchers’ analysis begins with the properties common to traditional optimizers, followed by an exploration of those unique to LLM-based optimizers.
Investigation and Analysis
The study begins with a baseline evaluation of several candidate LLMs using a series of straightforward tasks that include both discrete and continuous optimization challenges. The term baseline evaluation refers to the use of the simplest settings to assess the fundamental optimization capabilities of the models.
The researchers go on to investigate whether LLMs exhibit properties commonly associated with traditional optimization algorithms, including:
- Understanding of Numerical Values
- Scalability on Problem Dimensions
- Adaptability to Diverse Problem Scenarios
- Balancing Exploration and Exploitation
It can be observed that LLMs do not consistently benefit from increased input precision; in some cases, performance deteriorates as the number of decimal digits increases.
Evaluation results of LLMs’ capacity in handling string-represented numbers on a 2-dimensional sphere function.
The researchers present evidence that as the number of dimensions increases exponentially, there is a consistent decline in performance. Such a limit of context length poses a significant constraint on the practical use of LLMs in optimization, directly impacting their scalability.
Evaluation results of LLMs’ scalability with respect to problem dimensionality using the Sphere function.
Performance fluctuates drastically when the given input undergoes a shift. Notably, the tested models exhibit distinct responses to these shifts.
Evaluation results of LLMs’ adaptability to shifted continuous optimization problems.
The behavior of LLMs is found to be significantly influenced by the structure and content of the prompt, which can have a substantial impact on optimization outcomes. The researchers posit that this may be because these LLMs are primarily designed to process general texts, rather than being optimized for complex optimization tasks, in which the ability to distinguish the input order is a desirable feature rather than a limitation. Consequently, their performance is heavily dependent on the specific formulation of the prompt, which can lead to inconsistent results and undermine their reliability in certain applications.
Conclusion and Future Research
In summary, although LLMs can engage with various numerical optimization tasks, they lack the distinctive features of traditional algorithms, while incurring significantly higher computational costs, which leaves them far behind conventional methods. The researchers found LLMs lack basic properties essential for effective optimization, such as the ability to understand and handle numbers in the string format. Furthermore, their ability to balance exploration and exploitation lags behind that of traditional algorithms.
As a result, relying solely on LLMs to tackle complex optimization tasks, which require numerical comprehension and assume little prior knowledge, is unreliable. The promising performance reflected in current research is often attributed to small problem sizes or the optimal solution being proximal to a special value. Therefore, caution and rigorous validation of LLMs’ effectiveness are indispensable when applying LLMs to related fields.
Despite their limitations, according to the researchers, LLMs have still demonstrated some effectiveness in solving optimization problems. This behavior can be attributed to the use of a genetic algorithm framework, where the LLM is responsible only for generating individual solutions rather than controlling the entire optimization process. LLMs offer the advantage of requiring less domain knowledge than traditional optimization algorithms. They can autonomously distinguish between discrete and continuous problems without any human input, while also leveraging information embedded in the prompt—unlocking new possibilities for heuristic approaches and advanced optimization techniques.
Enhancing the competency of LLMs in optimization is likely to benefit from increased research efforts on integrating external tools to assist with numerical operations and addressing the limitations imposed by context length. Moreover, future applications of LLMs are expected to encompass more complex scenarios, such as evolutionary multitasking and evolutionary transfer optimization. The findings from this article lay the groundwork for a deeper understanding of LLMs’ role in optimization and inform future applications of LLMs in a wide range of scenarios.
Interested in learning more about Large Language Models? The IEEE Xplore Digital Library offers over 39,000 publications on Large Language Models.
Interested in acquiring full-text access to this collection for your entire organization? Request a free demo and trial subscription for your organization.
