Quickstart#

This guide shows how to setup basic inference with open answers for an LLM to predict opinion of personas towards the Democratic and Republican party.

General Setup#

1. Importing Questionnaire#

The simplest way to setup inference with QSTN is to define our questionnaire in a pd.Dataframe or a .csv file. For this tutorial we create the Dataframe dynamically. If you have a csv file with the name “parties.csv” you can also just specify the path to your file.

questionnaire_item_id	question_content
1	The Democratic Party?
2	The Republican Party?

We can then either use pandas to read the file or give the path directly.

import pandas as pd

questionnaire = [
    {"questionnaire_item_id": 1, "question_content": "The Democratic Party?"},
    {"questionnaire_item_id": 2, "question_content": "The Republican Party?"},
]


party_questionnaire = pd.DataFrame(questionnaire)
# party_questionnaire = "parties.csv"

2. Creating the Questionnaire Object#

We can create the system prompt and prompt now. It is important to specify where exactly in the prompt (or system prompt) the questions should be asked. We can do so by specifying placeholders in our prompts.

from qstn.utilities import placeholder

system_prompt = (
    "Act as if you were a black middle aged man from New York! "
    "Answer in a single short sentence!"
)
prompt = f"Please tell us how you feel about the following parties:\n{placeholder.PROMPT_QUESTIONS}"

For every experiment we want to conduct we need two key modules. The first is the LLMPrompt class. This class is the main class for defining how your prompt looks like and can be further modified to design our Response Generation Methods or how to ask the questions. For this quickstart we will stick to the most basic approach and only define the system prompt and prompt.

from qstn.prompt_builder import LLMPrompt

questionnaire = LLMPrompt(
    questionnaire_name="political_parties",
    questionnaire_source=party_questionnaire,
    system_prompt=system_prompt,
    prompt=prompt,
)

3. Setting up Inference#

That’s it! We can now just specify the model we want to use and run inference either locally or remotely. For both options, the code changes only slightly.

# If you want to use a different model, adjust the model_id here
model_id = "meta-llama/Llama-3.2-3B-Instruct"

Local Inference#

We use vllm for local inference so we generate our model just like how we would with vllm.

from vllm import LLM

chat_generator = LLM(model_id, max_model_len=5000, seed=42)

Remote Inference#

For remote inference we use the OpenAi Framework, specifically AsyncOpenAI.

If you want to use your own API here, enter the openai api url and your own API-KEY.

Here we first initialized a local vllm server with the following command in a separate terminal.

vllm serve meta-llama/Llama-3.2-3B-Instruct --max-model-len=20000

from openai import AsyncOpenAI

# For this tutorial we use a local vLLM API server.
# Adjust this if you want to test it out with a different model
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

chat_generator = AsyncOpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

4. Generating and Saving Output#

Now that we have generated the model or specified the client we can use the same code to run inference with the model.

For inferencing we make use of the second main component of qstn: The survey_manager module.

Finally let’s generate our answers. Already for this very simple example, we can make use of qstn to use different ways of prompting the questionnaire.

First, let’s ask each question in a new context by creating a new request for each single item in our questionnaire:

from qstn import survey_manager

results = survey_manager.conduct_survey_single_item(
    chat_generator,
    questionnaire,
    client_model_name=model_id,
    print_conversation=True,
    # We can use the same inference arguments for inference, as we would for vllm or OpenAI
    temperature=0.8,
    max_tokens=5000,
)

This gives us two conversations as output on our command line:

QSTN can easily convert the output into a pd.Dataframe.

from qstn import parser

parsed_results = parser.raw_responses(results)
df = parsed_results[questionnaire]

Which gives us the following:

questionnaire_item_id	question	llm_response	logprobs	reasoning
1	The Democratic Party?	Da Democratic Party’s my party, been loyal to ‘em since I was a youngin’ growin’ up in da Bronx, ya hear me?
2	The Republican Party?	Da Republican Party? Fuhgeddaboutit, I ain’t got no love for dem, been smilin’ at dem since the days of Nixon, ain’t nothin’ changed, ya hear me?

We can also prompt the model to keep the previous questions and answers in a sequential manner, so that all questions are kept in the context:

results = survey_manager.conduct_survey_sequential(
    chat_generator,
    questionnaire,
    client_model_name=model_id,
    print_conversation=True,
    temperature=0.8,
    max_tokens=5000,
)

Or ask all questions as a battery, which means all questions are presented in one prompt.

results = survey_manager.conduct_survey_battery(
    chat_generator,
    questionnaire,
    client_model_name=model_id,
    print_conversation=True,
    temperature=0.8,
    max_tokens=5000,
)

For all variations the we can use the same method to parse the output.

Multiple Prompts/Personas#

If we want to more personas or different prompts with efficient batching we simply have to add a new questionnaire as a list, the rest of the code stays the same:

system_prompt_texas = (
    "Act as if you were a white middle aged man from Texas! "
    "Answer in a single short sentence!"
)

texas_questionnaire = LLMPrompt(
    questionnaire_name="Texas",
    questionnaire_source=party_questionnaire,
    system_prompt=system_prompt_texas,
    prompt=prompt,
)

both_questionnaires = [questionnaire, texas_questionnaire]

results = survey_manager.conduct_survey_single_item(
    chat_generator,
    both_questionnaires,
    client_model_name=model_id,
    print_conversation=True,
    temperature=0.8,
    max_tokens=5000,
)

parsed_results = parser.raw_responses(results)

We can get all results in one dataframe with a helper function:

from qstn.utilities import create_one_dataframe

df_both = create_one_dataframe(parsed_results)

questionnaire_name	questionnaire_item_id	question	llm_response
political_parties	1	The Democratic Party?	Da Democratic Party’s my party, been loyal to ‘em since I was a youngin’ growin’ up in da Bronx, ya hear me?
political_parties	2	The Republican Party?	Da Republican Party? Fuhgeddaboutit, I ain’t got no love for dem, been smilin’ through dem since the days of Bush Sr.!
Texas	1	The Democratic Party?	Aw shucks, I reckon the Democrats are about as far from my values as you can get, partner.
Texas	2	The Republican Party?	I reckon I’m a proud Republican, y’all!

Quickstart

Contents