6. Bulk prompts¶
Our reusable prompting function is pretty cool. But requesting answers one by one across a big dataset could take forever. And it could cost us money to hit the Groq API so many times.
One solution is to submit your requests in batches and then ask the LLM to return its responses in bulk.
A common way to do that is to prompt the LLM to return its responses in JSON, a JavaScript data format that is easy to work with in Python.
To try that, we start by adding the built-in json library to our imports in classifier.py
{emphasize-lines=”2”}
import os
import json
from retry import retry
from groq import Groq
Next, we make a series of changes to our function to adapt it to work with a batch of inputs. Get ready. It’s a lot.
We tweak the name of the function.
We change our input argument to a list.
We expand our prompt to explain that we will provide a list of team names.
We ask the LLM to classify them individually, returning its answers in a JSON list.
We insist on getting one answer for each input.
We tweak our few-shot training to reflect this new approach.
We submit our input as a single string with new lines separating each team name.
We convert the LLM’s response from a string to a list using the
json.loadsfunction.We check that the LLM’s answers are in our list of acceptable answers with a loop through the list.
We merge the team names and the LLM’s answers into a dictionary returned by the function.
{emphasize-lines=”2,17-27,36-43,46,53-54,62-66”}
@retry(ValueError, tries=2, delay=2)
def classify_teams(name_list):
prompt = """
You are an AI model trained to classify text.
I will provide list of professional sports team names separated by new lines
You will reply with the sports league in which they compete.
Your responses must come from the following list:
- Major League Baseball (MLB)
- National Football League (NFL)
- National Basketball Association (NBA)
If the team's league is not on the list, you should label them as "Other".
Your answers should be returned as a flat JSON list.
It is very important that the length of JSON list you return is exactly the same as the number of names you receive.
If I were to submit:
"Los Angeles Rams\nLos Angeles Dodgers\nLos Angeles Lakers\nLos Angeles Kings"
You should return the following:
["National Football League (NFL)", "Major League Baseball (MLB)", "National Basketball Association (NBA)", "Other"]
"""
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": prompt,
},
{
"role": "user",
"content": "Chicago Bears\nChicago Cubs\nChicago Bulls\nChicago Blackhawks",
},
{
"role": "assistant",
"content": '["National Football League (NFL)", "Major League Baseball (MLB)", "National Basketball Association (NBA)", "Other"]',
},
{
"role": "user",
"content": "\n".join(name_list),
}
],
model="llama-3.3-70b-versatile",
temperature=0,
)
answer_str = response.choices[0].message.content
answer_list = json.loads(answer_str)
acceptable_answers = [
"Major League Baseball (MLB)",
"National Football League (NFL)",
"National Basketball Association (NBA)",
"Other",
]
for answer in answer_list:
if answer not in acceptable_answers:
raise ValueError(f"{answer} not in list of acceptable answers")
return dict(zip(name_list, answer_list))
Try that with our team list. At the bottom, instead of the loop, just put this:
classify_teams(team_list)
And re-run the classifier. You’ll see that it works with only a single API call. The same technique will work for a batch of any size.
{'Minnesota Twins': 'Major League Baseball (MLB)',
'Minnesota Vikings': 'National Football League (NFL)',
'Minnesota Timberwolves': 'National Basketball Association (NBA)',
'Minnesota Wild': 'Other'}
Though, as you batches get bigger, one common problem is that the number of outputs from the LLM can fail to match the number of inputs you provide. This problem may lessen as LLMs improve, but for now it’s a good idea to limit to batches to a few dozen inputs and to verify that you’re getting the right number back.
{emphasize-lines=”66-69”}
@retry(ValueError, tries=2, delay=2)
def classify_teams(name_list):
prompt = """
You are an AI model trained to classify text.
I will provide list of professional sports team names separated by new lines
You will reply with the sports league in which they compete.
Your responses must come from the following list:
- Major League Baseball (MLB)
- National Football League (NFL)
- National Basketball Association (NBA)
If the team's league is not on the list, you should label them as "Other".
Your answers should be returned as a flat JSON list.
It is very important that the length of JSON list you return is exactly the same as the number of names you receive.
If I were to submit:
"Los Angeles Rams\nLos Angeles Dodgers\nLos Angeles Lakers\nLos Angeles Kings"
You should return the following:
["National Football League (NFL)", "Major League Baseball (MLB)", "National Basketball Association (NBA)", "Other"]
"""
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": prompt,
},
{
"role": "user",
"content": "Chicago Bears,Chicago Cubs,Chicago Bulls,Chicago Blackhawks",
},
{
"role": "assistant",
"content": '["National Football League (NFL)", "Major League Baseball (MLB)", "National Basketball Association (NBA)", "Other"]',
},
{
"role": "user",
"content": "\n".join(name_list),
}
],
model="llama-3.3-70b-versatile",
temperature=0,
)
answer_str = response.choices[0].message.content
answer_list = json.loads(answer_str)
acceptable_answers = [
"Major League Baseball (MLB)",
"National Football League (NFL)",
"National Basketball Association (NBA)",
"Other",
]
for answer in answer_list:
if answer not in acceptable_answers:
raise ValueError(f"{answer} not in list of acceptable answers")
try:
assert len(name_list) == len(answer_list)
except AssertionError:
raise ValueError(f"Number of outputs ({len(name_list)}) does not equal the number of inputs ({len(answer_list)})")
return dict(zip(name_list, answer_list))
Okay. Naming sports teams is a cute trick, but what about something hard?
We’ll tackle that by pulling in our example dataset using pandas, a popular data manipulation library in Python.
First, we need to install it. Back to the terminal:
pip install pandas
Let’s create a new file for our specific classifier, and call it classifier_md.py:
touch classifier_md.py
Let’s edit that file, adding the imports and Groq setup:
{emphasize-lines=”5”}
import os
import json
from groq import Groq
from retry import retry
import pandas as pd
api_key = os.environ.get('GROQ_API_KEY')
client = Groq(api_key=api_key)
Now we’re ready to load the Maryland grantees data prepared for the class. It contains the distinct list of all recipients listed as grantees in this data. Let’s take a look by starting a Python shell in the terminal and putting these lines in:
import pandas as pd
df = pd.read_csv("https://raw.githubusercontent.com/dwillis/datajournalismbook-maryland/refs/heads/main/data/grantees.csv")
Have a look at a random sample to get a taste of what’s in there.
df.sample(10)
Grantee
742 PHILLIPS WARFT ENVIRONMENTAL CENTER INC
4848 HOUSING AUTHORITY ST MARYS CO MD
29 TURNER FARMS INC
4425 ANNE ARUNDEL COUNTY MARYLAND
195 ONE DAY AT A TIME INC
5818 FAMILY LEAGUE OF BALTIMORE CITY INC LMB
3571 UNIVERSITY OF MARYLAND COLLEGE PARK FOUNDATIONSTA
5036 HABITAT FOR HUMANITY OF WICOMICO COUNTY INC
6017 CHOPTANK COMMUNITY HEALTH SYSTEM CAROLINE
5433 FAMILY AND MEDICAL COUNSELING SERVICE
Now let’s adapt what we have to fit. Instead of asking for a sports league back, we will ask the LLM to classify each payee as a higher education institution, a museum, local government agency or other establishment. Add this function to your new classifier script.
{emphasize-lines=”2-26,33-48,61-66”}
@retry(ValueError, tries=2, delay=2)
def classify_grantees(name_list):
prompt = """You are an AI model trained to categorize organizations based on their names.
You will be given a list of organization names, each separated by a new line.
Your task is to analyze each name and classify it into one of the following categories: Higher Education, Museum, Local Government, or Other.
It is extremely critical that there is a corresponding category output for each organization name provided as an input.
If a organization does not clearly fall into Higher Education, Museum, or Local Government categories, you should classify it as "Other".
Even if the type of organization is not immediately clear from the name, it is essential that you provide your best guess based on the information available to you. If you can't make a good guess, classify it as Other.
For example, if given the following input:
"ANNE ARUNDEL COUNTY MARYLAND\nJOHN HOPKINS UNIVERSITY\nLEDOS PIZZA\nFREDERICK MEMORIAL HOSPITAL\nANNAPOLIS MARITIME MUSEUM\nCOLLEGE OF NOTRE DAME OF MARYLAND"
Your output should be a JSON list in the following format:
["Local Government", "Higher Education", "Other", "Other", "Museum", "Higher Education"]
This means that you have classified "ANNE ARUNDEL COUNTY MARYLAND" as a Local Government, "JOHN HOPKINS UNIVERSITY" as Higher Educatio, "LEDOS PIZZA" as Other, "FREDERICK MEMORIAL HOSPITAL" as Other, "ANNAPOLIS MARITIME MUSEUM" as Museum and "COLLEGE OF NOTRE DAME OF MARYLAND" as Higher Education.
Ensure that the number of classifications in your output matches the number of organization names in the input. It is very important that the length of JSON list you return is exactly the same as the number of organization names you receive.
"""
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": prompt,
},
{
"role": "user",
"content": "ANNE ARUNDEL COUNTY MARYLAND\nJOHN HOPKINS UNIVERSITY\nLEDOS PIZZA\nFREDERICK MEMORIAL HOSPITAL\nANNAPOLIS MARITIME MUSEUM\nCOLLEGE OF NOTRE DAME OF MARYLAND",
},
{
"role": "assistant",
"content": '["Local Government", "Higher Education", "Other", "Other", "Museum", "Higher Education"]',
},
{
"role": "user",
"content": "MD CENTER AT BOWIE STATE UNIVERSITY\nBALTIMORE CHILDRENS MUSEUM INCPORT DISCOVERY\nWMATA\nANNE ARUNDEL COUNTY COMMUNITY ACTION AGENCY",
},
{
"role": "assistant",
"content": '["Higher Education", "Museum", "Other", "Local Government"]',
},
{
"role": "user",
"content": "\n".join(name_list),
}
],
model="llama-3.3-70b-versatile",
temperature=0,
)
answer_str = response.choices[0].message.content
answer_list = json.loads(answer_str)
acceptable_answers = [
"Local Government",
"Higher Education",
"Museum",
"Other",
]
for answer in answer_list:
if answer not in acceptable_answers:
raise ValueError(f"{answer} not in list of acceptable answers")
try:
assert len(name_list) == len(answer_list)
except AssertionError:
raise ValueError(f"Number of outputs ({len(name_list)}) does not equal the number of inputs ({len(answer_list)})")
return dict(zip(name_list, answer_list))
Now pull out a random sample of grantees as a list and put that at the bottom of the script.
df = pd.read_csv("https://raw.githubusercontent.com/dwillis/datajournalismbook-maryland/refs/heads/main/data/grantees.csv")
sample_list = list(df.sample(10).Grantee)
print(classify_grantees(sample_list))
Save classifier_umd.py and see how it does.
python classifier_umd.py
{'SALISBURY STATE UNIVERSITY': 'Higher Education', 'LOCAL MANAGEMENT BOARD OF ALLEGANY CO INC': 'Local Government', 'WILLIS REDDEN': 'Other', 'ELB AUTOMOTIVE INC': 'Other', 'CATHOLIC CHARITIES ARCHDIOCESE OF WASHINGTON': 'Other', 'MARYLAND PARTNERSHIP FOR PREVENTION INC': 'Other', 'UNIVERSITY OF MARYLAND BALTIMORE COUNTY CENTER FOR WOMEN INFORMATION TECH': 'Higher Education', 'THE ARC OF HOWARD COUNTY INC': 'Other', 'METRO LAUNDRY CORPORATION': 'Other', 'EUI J CHOI': 'Other'}
That’s nice for a sample. But how do you loop through the entire dataset and code them.
One way to start is to write a function that will split up a list into batches of a certain size.
def get_batch_list(li, n=10):
"""Split the provided list into batches of size `n`."""
batch_list = []
for i in range(0, len(li), n):
batch_list.append(li[i : i + n])
return batch_list
Before we loop through our payees, let’s add a couple libraries that will let us avoid hammering Groq and keep tabs on our progress. In the terminal, do this:
pip install rich
{emphasize-lines=”4”}
import os
import time
import json
from rich.progress import track
from retry import retry
from groq import Groq
import pandas as pd
That batching trick can then be fit into a new function that will accept a big list of grantees and classify them batch by batch. Add this to the script just after the get_batch_list function:
def classify_batches(name_list, batch_size=10, wait=2):
"""Split the provided list of names into batches and classify with our LLM them one by one."""
# Create a place to store the results
all_results = {}
# Batch up the list
batch_list = get_batch_list(name_list, n=batch_size)
# Loop through the list in batches
for batch in track(batch_list):
# Classify it with the LLM
batch_results = classify_grantees(batch)
# Add what we get back to the results
all_results.update(batch_results)
# Tap the brakes to avoid overloading groq's API
time.sleep(wait)
# Return the results
return all_results
Now, let’s take out a bigger sample. Update the script to use a bigger sample.
bigger_sample = list(df.sample(100).Grantee)
And then replace the last line with this and run the script:
print(classify_batches(bigger_sample))
Printing out to the console is interesting, but eventually you’ll want to be able to work with the results in a more structured way. So let’s convert the results into a pandas DataFrame by modifying our classify_batches function.
{emphasize-lines=”20-23”}
def classify_batches(name_list, batch_size=10, wait=2):
# Store the results
all_results = {}
# Batch up the list
batch_list = get_batch_list(name_list, n=batch_size)
# Loop through the list in batches
for batch in track(batch_list):
# Classify it
batch_results = classify_grantees(batch)
# Add it to the results
all_results.update(batch_results)
# Tap the brakes
time.sleep(wait)
# Return the results
return pd.DataFrame(
all_results.items(),
columns=["grantee", "category"]
)
Results can now be stored as a DataFrame. Replace your last line with these:
results_df = classify_batches(bigger_sample)
print(results_df.head())
grantee category
0 MINH VINH Other
1 WALNUT HILL FARMS INC Other
2 THE BENEDICTINE SCHOOL Higher Education
3 THE UNIVERSITY OF KANSAS Higher Education
4 GARRETT CO CAC Local Government
Or a sum of all the categories.
print(results_df.category.value_counts())
category
Other 81
Higher Education 11
Local Government 8
Name: count, dtype: int64