The AI revolution is here and has started transforming our personal and professional lives through automation. As we venture into the world of AI agents, we find ourselves at the cusp of unprecedented automation and efficiency.
In the second half of 2024 and again near the end of the year, we evaluated three of the main AI agentic frameworks: CrewAI, Autogen and LangGraph. The evaluation was based on a task that asked a set up of three agents to compare a CV against job posts and advise on suitability of the candidate for the role, alongside suggestions for CV improvements, to better suit the role.
Everyone is talking about AI agents and their promise to change the game with respect to how AI will be applied, their contribution to accelerating process automation and their potential for a significant impact how software is constructed and consumed.
At AI-on-Cloud, we ascribe to these beliefs and consider AI agents a major driver to increased automation and rapid application development and deployment. We envision a future situation where developers create some basic tools to interact with data stores and legacy environments in a deterministic manner. These tools will be used by AI Agents who will, in turn, do the heavy lifting of processing based on prompting rather than having to explicitly code the logic behind this processing.
In other words, with AI agents, business processing could become as simple as giving a few prompts, with AI doing the heavy lifting. This will significantly change the architecture, speed of development and deployment of business solution, leading to far shorter time to market.
Given that AI agents, driven by AI models, provide their outputs in a non-determinist manner, this will require an adjustment of attitudes on the use of computers as well as the way we define and test solutions. We have been so far, conditioned on very deterministic outputs when interacting with computers (although, we do not have such demands from intelligent humans, it would appear).
On the back of such a great promise for the future, our evaluation on the current state of affairs was a ‘mixed bag’.
Perhaps as a testament to being at the early stages of this technology, we discovered that none of the current agentic AI frameworks would work with local small AI models. They all needed large commercial heavy duty AI models to power them up, even for simple actions, with all that this entails for cost and latency. This was especially the case for CrewAI and Autogen, which prefer OpenAI’s API and models. Our attempts in using emulated OpenAI APIs (from LM-Studio) and local models were, on the main not very successful.
A key issue with using small local models, was that the models would ‘harmfully’ hallucinate and not helping the agents execute their actions.
We also ascertained that, in their then current formats, the frameworks fell into two distinct categories:
- Purely ‘descriptive’: where all the logic is being described in action statements and prompts, utilising tools (that would contain functions) when needed.. This was the case for CrewAI and Autogen.
- Mainly ‘prescriptive’ and function based, strung together via graph-like definition of the logic flow, which was the case for LangGraph. We found that, in the evaluated release, LangGraph needed a lot of very low level coding, especially for the routing function, that negated some of the expected benefits of using AI agents.
All of the tested frameworks preferred and worked best with their own pre-defined tools. Defining and using new tools was a hit and miss operation. This was especially the case with CrewAI.
On a positive note, in the right conditions (mainly through using their own tools and large commercial models), the technology did work and we did get glimpses of the future, which was indeed, very promising.
It was also the case that AI agents defined and operated in the AWS and Azure platforms, using their respective solutions (eg AWS Bedrock) and supported AI models, worked flawlessly and were far easier to define and deploy, than coding and deploying locally. This points the way to Cloud platforms being the main and key environments in achieving the much-promised agentic transformation.
We strongly believe that it is simply a case of ‘when’, not an ‘if’, for AI agents to have the impact we envision, as the frameworks and the AI models keep developing and improving. We have started seeing releases of small AI models that have been trained to perform best in agentic flows.
We expect the frameworks to mature and offer the full gamma of functionality from the very abstract (for ease of construction and deployment in relatively easy use cases), to the low level (where fine-tuned control and guidance of the agent activity is required for complex scenarios).
All three frameworks had new releases late in 2024, although Autogen’s was still in beta. The new release of Autogen provides a more granular approach to managing agents, in the same type of approach as LangGraph’s, whilst updating the more abstract layer. In a parallel approach, LangGraph now offers new additional functionality that abstracts some of the detailed routing coding needs. In their updated release, CrewAI offers additional capabilities to direct agent actions, without going to a lower-level of (detailed) coding.
There has also been a release of an agentic framework from Pydantic, which is very promising, given that Pydantic has been a forerunner in creating a strict typing framework for Python, something that is essential for guiding AI agent behaviour.
At AI-on-Cloud, we’re passionate about harnessing the power of AI agents to drive innovation and efficiency. Are you ready to transform your business with the latest in AI technology?
