CrewAI Proxy example connecting to Sap Generative AI Hub via LiteLLM

How CrewAI works

LLM access via LiteLLM Proxy

CrewAI also supports the LiteLLM Proxy for access via OpenAI API calls.

ٔٔFollow the details in LiteLLM Proxy set up for SAP models

Installation

%pip install crewai

Set env variables

Add the following variables from the service key in a file called “.env” and put it in the same folder where you run the notebook:

LITELLM_PROXY_API_KEY=sk-1234
PROXY_BASE_URL=http://localhost:4000

Run the CrewAI with LiteLLM and SAP LLMs

import os

import litellm
from crewai import LLM, Agent, Crew, Task
from crewai.tools import tool
from dotenv import load_dotenv

Load your credentials as environment variables.

litellm.use_litellm_proxy = True
load_dotenv()
api_base = os.getenv("PROXY_BASE_URL")
api_key = os.getenv("LITELLM_PROXY_API_KEY")

Set up the model with your proxy params

proxy_llm = LLM(
    model="sap/gpt-4o", api_base=api_base, base_url=api_base, api_key=api_key
)

Define the agent tools.

@tool("get_weather")
def get_weather(city: str) -> str:
    """Moke function"""
    city_normalized = city.lower().replace(" ", "")

    mock_weather_db = {
        "newyork": "The weather in New York is sunny with a temperature of 25°C.",
        "london": "It's cloudy in London with a temperature of 15°C.",
        "tokyo": "Tokyo is experiencing light rain and a temperature of 18°C.",
    }

    if city_normalized in mock_weather_db:
        return mock_weather_db[city_normalized]
    else:
        return f"The weather in {city} is sunny with a temperature of 20°C."

User can select a city.

city = "Tbilisi"

Define the Agent and the SAP LLM to be used by CrewAI.

agent = Agent(
    role="Weather presenter",
    goal=f"Prepare a couple of sentences TV speach about weather in the {city}, "
         f"using information from run the get_weather tool",
    backstory="You are the weather presenter on TV",
    llm=proxy_llm,
    tools=[get_weather],
    allow_delegation=False,
)

Define tasks for the agents.

agent_task = Task(
    description=(
        f"Write a couple of sentences for TV weather report in {city}, that will be include small jok"
    ),
    expected_output=(
        "Good quality text of two sentences about weather and with small jok"
    ),
    agent=agent,
)

Create the crew with the Agent and task.

crew = Crew(
    agents=[agent],
    tasks=[agent_task],
    verbose=True,
)

Run the crew

result = crew.kickoff()
print("\n📘 Result:\n", result)