QSTN

QSTN is a Python framework designed to facilitate the creation of robust inference experiments with Large Language Models based around questionnaires. It provides a full pipeline from perturbation of prompts, to choosing Response Generation Methods, inferencing and finally parsing of the output. QSTN supports both local inference with vllm and remote inference via the OpenAI API.

Tutorials

We show a number of tutorials here, starting with an installation and a minimum usage example.

We also show how to use QSTN for real experiments, including how to use it to get answers from the German General Personas Dataset.

Getting Started

• Installation
• Quickstart
  • General Setup
    • 1. Importing Questionnaire
    • 2. Creating the Questionnaire Object
    • 3. Setting up Inference
    • 4. Generating and Saving Output
  • Multiple Prompts/Personas

Tutorials

• Foundations
  • LLMPrompt Tutorial
  • Answer Options Tutorial
  • Response Generation & Parsing Tutorial (Without Inference)
  • Inference Foundations: Batching, Modes, and Fast Runs
• Guides
  • Tutorial 1: Multiple Choice Questionnaires
  • Tutorial 2: Verbalized Distribution
• Resources
  • German General Personas

API Reference

The full documentation can be found below.

API Reference

• Detailed Documentation
  • QSTN
    • Prompt Generation
      • LLMPrompt
        • LLMPrompt.DEFAULT_JSON_STRUCTURE
        • LLMPrompt.DEFAULT_PROMPT_STRUCTURE
        • LLMPrompt.DEFAULT_QUESTIONNAIRE_ID
        • LLMPrompt.DEFAULT_SYSTEM_PROMPT
        • LLMPrompt.DEFAULT_TASK_INSTRUCTION
        • LLMPrompt.duplicate()
        • LLMPrompt.generate_question_prompt()
        • LLMPrompt.get_prompt_for_questionnaire_type()
        • LLMPrompt.get_question()
        • LLMPrompt.get_question_item_id()
        • LLMPrompt.get_questions()
        • LLMPrompt.insert_questions()
        • LLMPrompt.load_questionnaire_format()
        • LLMPrompt.prepare_prompt()
        • LLMPrompt.questions
        • LLMPrompt.remove_question()
        • LLMPrompt.replace_question()
      • generate_likert_options()
    • Survey Manager
      • Usage example:
      • SurveyCreator
        • SurveyCreator.from_dataframe()
        • SurveyCreator.from_path()
      • conduct_survey_battery()
      • conduct_survey_sequential()
      • conduct_survey_single_item()
    • Subpackages
      • Inference & Guided Decoding
        • Dynamic Pydantic
        • Response Generation Methods
        • Inference
      • Parser
        • parse_json()
        • parse_json_battery()
        • parse_json_str()
        • parse_logprobs()
        • parse_with_llm()
        • parse_with_llm_battery()
        • raw_responses()
      • Utilities
        • Constants
        • Placeholder
        • Prompt Perturbation
        • Prompt Templates
        • Survey Objects
        • Util Functions