Minion(s): A Simple Protocol for Communicating with Local and Cloud LLMs

Recently, HazyResearch introduced a simple communication protocol for integrating local and cloud LLMs.

The core idea behind the protocol is to maximize the use of local LLM models with local data, minimizing cloud API costs while maintaining high-quality outputs. The protocol comes in two flavors:

• Minion: A local LLM model (such as Llama 3.2 with Ollama) running on a laptop or desktop processes local documents. It communicates with a cloud model like GPT-4o, deciding which portion of the information to share and when to terminate the conversation. This selective communication helps reduce cloud API usage costs without compromising the overall result.

• Minions: The cloud model receives the documents, decomposes the task into smaller subtasks, and distributes them to the local LLM model. The local model executes the subtasks in parallel, and the cloud model aggregates the final result. This approach leverages local resources for faster execution while reducing cloud API costs.

The author’s blog post provides an in-depth explanation of the protocol, covering all its nuts and bolts. You can also explore the source code and examples shared by the author.

In this blog post, I will run the example from the source code to evaluate how the protocol works in practice.

Get started.

To use the protocol, you need Ollama running locally on your device along with access to a cloud LLM like ChatGPT.

First, install Ollama on your local machine. Then, obtain your OpenAI API key to enable cloud-based model interaction.

Step 1. Clone the repository:

git clone https://github.com/HazyResearch/minions.git 
cd minions

Step 2. Active your conda environment:

conda activate YOUR_ENV
jupyter notebook # assume that, you already have the jupyter installed on conda

Step 3. Pull and run your LLM model on Ollama

ollama pull deepseek-r1:32b  # you can use any model like llama3.2

If you're having trouble installing Ollama, please refer to my previous post, How to Install Ollama, or check the sample chapter from the book Generative AI with Local LLM.

Step 4. Install the Python package.

pip install -e . 

The above command will install the necessary libraries in editable mode.

If you encounter an error like:

Ignoring numpy: markers 'python_version < "3.9"' don't match your environment.  

Edit the setup.py file and make the spacy library optional by adding the following section:

extras_require={
        "pii": ["spacy"],  # Optional PII extraction functionality
    },

Step 5. Run the application with Streamlit

streamlit run app.py

You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501

Enter your OpenAI API key, select a local model, provide the context, a one-sentence description, and your query.

For demonstration purposes, I have used the following context and parameters.

Context: "When you arrive at a hotel, you must check-in at the reception or front desk.

The check-in process can take a while since the receptionist has to find your

reservation, request payment for the room, and then inform you about the

hotel's policies and procedures. You are also given a key to your room at this

time.

Hotels often distinguish themselves by the services they offer. Fancy hotels

often have a porter to help you with a variety of tasks. They can help you get a

taxi, make reservations at restaurants or plays for you, and give you advice

about the city.

In smaller and cheaper hotels, the job of porter is done by the receptionist and

a doorman, who opens the hotel doors and car doors for you.

These are nice services, especially after you've been traveling, but they're not

free. It is common courtesy to tip the porter and bellboys each time they help

you.

In your room, there may be a single or double bed, depending on how many

people are staying there. There may even be two single beds, or twin beds.

Also in the room are a desk, a dresser to store your clothes in, a nightstand

with a lamp beside the bed, a television with cable/satellite TV, heating and air

conditioning.

If you're lucky, there may even be a mini-bar in your room. This is a small

refrigerator that has tiny bottles of alcohol, as well as snacks. But beware:

these are not free. In fact, they typically cost two to three times as much as

they do in a grocery store. But many people pay the exuberant price for the

convenience of not having to leave the hotel.

Another convenience that hotels offer is room service. To order room service,

you call down to the reception and ask for a food item listed on the hotel's

menu. The food is then brought to your room for you to enjoy. Remember, this

is another service that deserves a tip.

Many hotels also have restaurants attached where breakfast is served in the

morning. A hotel breakfast can range from a Continental buffet, which consists

of you helping yourself to food that has been laid out for you and other hotel

guests. The Continental breakfast is not very elaborate and is often included in

the price of the room. However, you can also order prepared food from the

restaurant's menu.

At the proper check-out time, which is often early, you must vacate the room

so that the cleaning staff, can clean the rooms and make the beds. If you don't

leave on time, charges may apply and you will have to pay extra money.

Fortunately, you can ask for a wake-up call from the front desk so you won't

sleep through check-out

To get to your next destination, you can take an airport shuttle, which will take

you directly to the airport. If a hotel doesn't have its own shuttle, it can usually

arrange for one to pick you up at the hotel.

"Making Reservations

Reception: Good morning. The Big Grand Hotel. How may I help you?

Guest: Hi, good morning. I'd like to make a reservation for the first weekend in

September. Do you have any vacancies?

Reception: Yes sir, we have several rooms available for that particular

weekend. And what is the exact date of your arrival?

Guest: The 10th.

Reception: How long will you be staying?

Guest: I'll be staying for three nights.

Reception: How many people is the reservation for?

Guest: There will be two of us.

Reception: And would you like a room with twin beds or a double bed?

Guest: A double bed, please.

Reception: Great. And would you prefer to have a room with a view of the

lake?

Guest: If that type of room is available, I would love to have a lake view.

What's the rate for the room?

Reception: Your room is six hundred and fifty dollars per night. Now what

name will the reservation be listed under?

Guest: Simon Moore.

Reception: Could you spell your last name for me, please?

Guest: Sure M-O-O-R-E

Reception: And is there a phone number where you can be contacted?

Guest: Yes, my mobile phone number is 123 456 7890.

Reception: Great. Now I'll need your credit card information to reserve the

room for you. What type of card do you have?

Guest: MasterCard. The number is 987654321.

Reception: And what is the name of the cardholder?

Guest: Simon T Moore.

Reception: Alright, Mr. Moore, your reservation has been made for the tenth

of September for a room with a double bed and view of the lake. Check-in is

after 2 pm. If you have any other questions, please do not hesitate to call us.

Guest: Great, thank you so much.

Reception: My pleasure. We'll see you in September, Mr. Moore."

"Checking-In

Reception: Good afternoon. Welcome to The Big Grand Hotel. How may I

help you?

Guest: I have a reservation for today. It's under the name of Moore.

Reception: Can you please spell that for me, sir?

Guest: Sure. M-O-O-R-E.

Reception: Yes, Mr. Moore, we've reserved a double room for you with a view

of the lake for three nights. Is that correct?

Guest: Yes, it is.

Reception: Excellent. We already have your credit card information on file. If

you'll just sign the receipt along the bottom, please.

Guest: Whoa! Six hundred and fifty dollars a night!

Reception: Yes, sir. We are a five star hotel after all.

Guest: Well, fine. I'm here on business anyway, so at least I'm staying on the

company's dime. What's included in this cost anyway?

Reception: A full Continental buffet every morning, free airport shuttle service,

and use of the hotel's safe are all included.

Guest: So what's not included in the price?

Reception: Well, you will find a mini-bar in your room. Use of it will be

charged to your account. Also, the hotel provides room service, at an

additional charge of course.

Guest: Hmm. Ok, so what room am I in?

Reception: Room 721 sir. Here is your key. To get to your room, take the

elevator on the right up to the seventh floor. Turn right once you exit the

elevator and your room will be on the left hand side. A bellboy will bring your

bags up shortly.

Guest: Great. Thanks.

Reception: Should you have any questions or requests, please dial 'O' from

your room. Also, there is internet available in the room 24 hours a day.

Guest: Ok, and what time is check-out?

Reception: Before midday, sir.

Guest: Ok, thanks.

Reception: My pleasure, sir. Have a wonderful stay at The Big Grand Hotel."

"Check-out / Getting to the airport

Reception: Did you enjoy your stay with us?

Guest: Yes, very much so. However, I now need to get to the airport. I have a

flight that leaves in about two and a half hours, so what is the quickest way to

get there?

Reception: We do have a free airport shuttle service.

Guest: That sounds great, but will it get me to the airport on time?

Reception: Yes, it should. The next shuttle leaves in 10 minutes, and it takes

approximately 15 minutes to get to the airport.

Guest: Fantastic. I'll just wait in the lounge area. Will you please let me know

when it will be leaving?

Reception: Of course, sir. Oh, before you go would you be able to settle the

mini-bar bill?

Guest: Oh yes certainly. How much will that be?

Reception: Let's see. The bill comes to 37.50. How would you like to pay for

that?

Guest: I'll pay with my Visa thanks, but I'll need a receipt so I can charge it to

my company.

Reception: Absolutely. Here we are sir. If you like you can leave your bags

with the porter and he can load them onto the shuttle for you when it arrives.

Guest: That would be great thank you.

Hotel: Would you like to sign the hotel guestbook too while you wait?

Guest: Sure, I had a really good stay here and I'll recommend you to other

people too.

Hotel: That's good to hear. Thank you again for staying at The Big Grand

Hotel.""

One-sentence description of the context (Required): dealing with hotels

Query: Using the dealing with hotels dialog, find out how much the mini bar bill

Note that the final answer is incorrect ($14.50) when using the Minion communication protocol.

However, the total tokens sent to the remote LLM is only 1,774, resulting in a total cost of $0.0068.

Let's run the Minions communication protocol with the same context and query.

First of all, the answer is correct. However, the total cost is higher at $0.0418 because the entire document was sent to the remote model.

Even more interesting things happen under the hood. To solve the task, the remote model performs the following steps:

1. To determine the mini bar bill from the provided text, focus on identifying any sections or dialogues related to hotel billing or charges. Look for keywords such as "mini bar," "charges," "bill," "invoice," or "expenses." Pay attention to any numerical values or currency symbols that might indicate costs associated with the mini bar. Additionally, consider any context or conversations that mention room services or amenities, as these might include references to the mini bar usage. It's important to differentiate between actual charges and any hypothetical or discussed scenarios that do not reflect the final bill.

2. To address the task of extracting the mini bar bill from a large text document, we will break the document into manageable chunks and assign tasks to extract relevant information from each chunk. Here's how we can implement the prepare_jobs and transform_outputsfunctions:

   def prepare_jobs(
       context: List[str],
       prev_job_manifests: Optional[List[JobManifest]] = None,
       prev_job_outputs: Optional[List[JobOutput]] = None,
   ) -> List[JobManifest]:
       job_manifests = []
   
       # Define the maximum chunk size for processing
       max_chunk_size = 1000  # Smaller chunks for precise information extraction
       chunks = chunk_by_section(context[0], max_chunk_size=max_chunk_size, overlap=0)
   
       # Define the task to extract mini bar charges
       task = "Extract any information related to mini bar charges, including any numerical values or currency symbols."
   
       # Create a job manifest for each chunk
       for chunk in chunks:
           job_manifest = JobManifest(
               chunk=chunk,
               task=task,
               advice="Focus on identifying any mentions of mini bar charges, costs, or related billing information.",
           )
           job_manifests.append(job_manifest)
   
       return job_manifests
   
   def transform_outputs(
       jobs: List[Job],
   ) -> str:
       # Collect all relevant outputs
       mini_bar_info = []
   
       for job in jobs:
           output = job.output
           if output.answer:
               mini_bar_info.append(output.answer)
   
       # Aggregate the collected information
       aggregated_info = " ".join(mini_bar_info)
   
       return aggregated_info

Explanation

• prepare_jobs: This function breaks the document into smaller chunks of 1000 characters each, which is suitable for precise information extraction tasks. Each chunk is assigned a task to extract any information related to mini bar charges, including numerical values or currency symbols. The task is designed to be atomic and focused on a single chunk.

• transform_outputs: This function processes the outputs from the worker model. It filters out any empty or irrelevant results and aggregates the relevant information about mini bar charges into a single string. This aggregated string will be used for further analysis or decision-making.

Here are the outputs from all the minions! (Local model)

Note: 0 jobs did not have relevant information.

Jobs with relevant information:

✅ Job 1 (Chunk 1)

Answer: $650-$975 per night

✅ Job 2 (Chunk 2)

Answer: $37.50

During the aggregation phase, the remote model uses the gathered information to generate the final conclusion.

In conclusion, based on the available information, the mini bar bill is likely $37.50. This figure is supported by the context provided in the collected outputs.

Here's a table summarizing the performance differences and cost savings between the Minion and Minions communication protocols:

As an alternative, you can run the minions.ipynb notebook to explore the protocol's functionality.

Step 6. Open the minions.ipynb on jupyter and add the following lines

!pip install pandas
!pip install pydantic
!pip install tiktoken
!pip install openai
!pip install anthropic
!pip install together
!pip install requests
!pip install pymupdf
!pip install st-theme
!pip install mcp
!pip install spacy
!pip install streamlit
!pip install ollama

The above lines will install all the necessary libraries.

Step 7. Imports:

Add or let's unchanged the following codes on the notebook.

# Misc imports
import os
import json
import pandas as pd

# Import Minions + Minions
from minions.minions import Minions
from minions.minion import Minion

# Import Minion Clients
from minions.clients.ollama import OllamaClient
from minions.clients.tokasaurus import TokasaurusClient
from minions.clients.openai import OpenAIClient
from minions.clients.anthropic import AnthropicClient
from minions.clients.together import TogetherClient

# Import Pydantic
from pydantic import BaseModel

Step 8. Specify structured output schema for the OllamaClient.

class StructuredLocalOutput(BaseModel):
    explanation: str
    citation: str | None
    answer: str | None

Step 9. Initialized the local and cloud clients:

remote_client = OpenAIClient(model_name="gpt-4o", temperature=0.0)

# Option 1: Ollama
local_client = OllamaClient(
    model_name="deepseek-r1:32b", 
    temperature=0.0, 
    structured_output_schema=StructuredLocalOutput
)

Step 10. Setup the communication protocol:

First of all, we will use the Minion Protocol as shown below.

# Option 1: Minions
#protocol = Minions(local_client=local_client, remote_client=remote_client)

# Option 2: Minion
protocol = Minion(local_client=local_client, remote_client=remote_client)

Step 11. Configure the context.

Here is the context that the Minions need to reason over. I slightly modified the context by using hotel reservation dialogs to make the concept clearer.

context = """\
When you arrive at a hotel, you must check-in at the reception or front desk.
The check-in process can take a while since the receptionist has to find your
reservation, request payment for the room, and then inform you about the
hotel's policies and procedures. You are also given a key to your room at this
time.
Hotels often distinguish themselves by the services they offer. Fancy hotels
often have a porter to help you with a variety of tasks. They can help you get a
taxi, make reservations at restaurants or plays for you, and give you advice
about the city.
In smaller and cheaper hotels, the job of porter is done by the receptionist and
a doorman, who opens the hotel doors and car doors for you.
These are nice services, especially after you've been traveling, but they're not
free. It is common courtesy to tip the porter and bellboys each time they help
you.
In your room, there may be a single or double bed, depending on how many
people are staying there. There may even be two single beds, or twin beds.
Also in the room are a desk, a dresser to store your clothes in, a nightstand
with a lamp beside the bed, a television with cable/satellite TV, heating and air
conditioning.
If you're lucky, there may even be a mini-bar in your room. This is a small
refrigerator that has tiny bottles of alcohol, as well as snacks. But beware:
these are not free. In fact, they typically cost two to three times as much as
they do in a grocery store. But many people pay the exuberant price for the
convenience of not having to leave the hotel.
Another convenience that hotels offer is room service. To order room service,
you call down to the reception and ask for a food item listed on the hotel's
menu. The food is then brought to your room for you to enjoy. Remember, this
is another service that deserves a tip.
Many hotels also have restaurants attached where breakfast is served in the
morning. A hotel breakfast can range from a Continental buffet, which consists
of you helping yourself to food that has been laid out for you and other hotel
guests. The Continental breakfast is not very elaborate and is often included in
the price of the room. However, you can also order prepared food from the
restaurant's menu.
At the proper check-out time, which is often early, you must vacate the room
so that the cleaning staff, can clean the rooms and make the beds. If you don't
leave on time, charges may apply and you will have to pay extra money.
Fortunately, you can ask for a wake-up call from the front desk so you won't
sleep through check-out
To get to your next destination, you can take an airport shuttle, which will take
you directly to the airport. If a hotel doesn't have its own shuttle, it can usually
arrange for one to pick you up at the hotel.

"Making Reservations
 Reception: Good morning. The Big Grand Hotel. How may I help you?
 Guest: Hi, good morning. I'd like to make a reservation for the first weekend in
September. Do you have any vacancies?
 Reception: Yes sir, we have several rooms available for that particular
weekend. And what is the exact date of your arrival?
 Guest: The 10th.
 Reception: How long will you be staying?
 Guest: I'll be staying for three nights.
 Reception: How many people is the reservation for?
 Guest: There will be two of us.
 Reception: And would you like a room with twin beds or a double bed?
 Guest: A double bed, please.
 Reception: Great. And would you prefer to have a room with a view of the
lake?
 Guest: If that type of room is available, I would love to have a lake view.
What's the rate for the room?
 Reception: Your room is six hundred and fifty dollars per night. Now what
name will the reservation be listed under?
 Guest: Simon Moore.
 Reception: Could you spell your last name for me, please?
 Guest: Sure M-O-O-R-E
 Reception: And is there a phone number where you can be contacted?
 Guest: Yes, my mobile phone number is 123 456 7890.
 Reception: Great. Now I'll need your credit card information to reserve the
room for you. What type of card do you have?
 Guest: MasterCard. The number is 987654321.
 Reception: And what is the name of the cardholder?
 Guest: Simon T Moore.
 Reception: Alright, Mr. Moore, your reservation has been made for the tenth
of September for a room with a double bed and view of the lake. Check-in is
after 2 pm. If you have any other questions, please do not hesitate to call us.
 Guest: Great, thank you so much.
 Reception: My pleasure. We'll see you in September, Mr. Moore."

"Checking-In
 Reception: Good afternoon. Welcome to The Big Grand Hotel. How may I
help you?
 Guest: I have a reservation for today. It's under the name of Moore.
 Reception: Can you please spell that for me, sir?
 Guest: Sure. M-O-O-R-E.
 Reception: Yes, Mr. Moore, we've reserved a double room for you with a view
of the lake for three nights. Is that correct?
 Guest: Yes, it is.
 Reception: Excellent. We already have your credit card information on file. If
you'll just sign the receipt along the bottom, please.
 Guest: Whoa! Six hundred and fifty dollars a night!
 Reception: Yes, sir. We are a five star hotel after all.
 Guest: Well, fine. I'm here on business anyway, so at least I'm staying on the
company's dime. What's included in this cost anyway?
 Reception: A full Continental buffet every morning, free airport shuttle service,
and use of the hotel's safe are all included.
 Guest: So what's not included in the price?
 Reception: Well, you will find a mini-bar in your room. Use of it will be
charged to your account. Also, the hotel provides room service, at an
additional charge of course.
 Guest: Hmm. Ok, so what room am I in?
 Reception: Room 721 sir. Here is your key. To get to your room, take the
elevator on the right up to the seventh floor. Turn right once you exit the
elevator and your room will be on the left hand side. A bellboy will bring your
bags up shortly.
 Guest: Great. Thanks.
 Reception: Should you have any questions or requests, please dial 'O' from
your room. Also, there is internet available in the room 24 hours a day.
 Guest: Ok, and what time is check-out?
 Reception: Before midday, sir.
 Guest: Ok, thanks.
 Reception: My pleasure, sir. Have a wonderful stay at The Big Grand Hotel."

"Check-out / Getting to the airport
 Reception: Did you enjoy your stay with us?
 Guest: Yes, very much so. However, I now need to get to the airport. I have a
flight that leaves in about two and a half hours, so what is the quickest way to
get there?
 Reception: We do have a free airport shuttle service.
 Guest: That sounds great, but will it get me to the airport on time?
 Reception: Yes, it should. The next shuttle leaves in 10 minutes, and it takes
approximately 15 minutes to get to the airport.
 Guest: Fantastic. I'll just wait in the lounge area. Will you please let me know
when it will be leaving?
 Reception: Of course, sir. Oh, before you go would you be able to settle the
mini-bar bill?
 Guest: Oh yes certainly. How much will that be?
 Reception: Let's see. The bill comes to 37.50. How would you like to pay for
that?
 Guest: I'll pay with my Visa thanks, but I'll need a receipt so I can charge it to
my company.
 Reception: Absolutely. Here we are sir. If you like you can leave your bags
with the porter and he can load them onto the shuttle for you when it arrives.
 Guest: That would be great thank you.
 Hotel: Would you like to sign the hotel guestbook too while you wait?
 Guest: Sure, I had a really good stay here and I'll recommend you to other
people too.
 Hotel: That's good to hear. Thank you again for staying at The Big Grand
Hotel."
"""

Step 12. Specify meta-data and task description:

doc_metadata = "dealing with hotels"

# Define the task without explicitly providing the key values.
task = (
    "Using the dealing with hotels dialog "
    "Find out how much the mini bar bill"
)

# For testing purposes, we suggest that the correct answer is:
suggested_final_answer = (
    "The bill comes to 37.50"
)

Step 13. Run the protocol:

output = protocol(
        task=task,
        doc_metadata=doc_metadata,
        context=[context],
        max_rounds=5,  # you can adjust rounds as needed for testing
    )
print(output['final_answer'])

If everything works as expected, you should see a result that closely matches the original example.

Privacy is enabled:  False
entering loop with max_rounds:  5
getting worker's response
Supervisor_json: {'decision': 'request_additional_info', 'message': 'Can you provide a list of items taken from the mini bar, their prices, and any additional charges or discounts mentioned in the hotel dialog?'}
getting worker's response
Supervisor_json: {'decision': 'provide_final_answer', 'answer': 'The mini bar bill totaled $37.50. The dialog does not provide a detailed list of items, their prices, or additional charges or discounts.'}

The mini bar bill totaled $37.50. The dialog does not provide a detailed list of items, their prices, or additional charges or discounts.

Step 14. Switch to the Minions protocol in step 10 and run the notebook again:

This time, the execution may take a few moments as the tasks are processed in parallel before returning the final result.

Round 1/5
Attempt 1/10
Created 2 job manifests (2 chunks, apriori requested 1 samples per chunk, 1 tasks)
Total number of job_manifests: 2
Sending 2 worker chats to the worker client
$37.50
Attempt 1/5 response: {
  "explanation": "The extracted mini bar charge of $37.50 directly answers the question.",
  "feedback": null,
  "decision": "provide_final_answer",
  "answer": "$37.50",
  "scratchpad": "Extracted mini bar charge is $37.50 from Chunk ID 0, which is sufficient to answer the question."
}

$37.50

The result is quite promising. According to OpenAI statistics, 33 requests were made across the two examples.

This protocol could be highly beneficial for those looking to reduce cloud LLM costs while leveraging a local LLM with minimal degradation in output quality.