promptfoo
Choose promptfoo for repeatable test configurations whose model combinations and assertions you can inspect.
- Starting price
- MIT CLI software; inference billed separately
- Free access
- MIT CLI and library; model API costs separate
- Billing unit
- Local software plus configured provider usage
- Platform
- CLI and CI/CD; maintainer image shown
- Verified
- 1 Nov 2026
Overview
What you provide
- Provide prompt configurations, model connections, test inputs and explicit assertions for systems you are authorised to test.
What you get
- promptfoo produces comparison results, assertion outcomes and security findings that can be reviewed locally or in CI.
Choosing this tool
The tool can compare prompts and providers, evaluate assertions and run authorised red-team checks as part of development. Phoenix complements a trace-first investigation of an application, while LangWatch adds managed observability and conversation simulations. Start with fixed test cases and inspect the generated requests; a local CLI does not establish that every model invocation remains local.
A decision worked through
Twenty cases evaluated against three model configurations imply sixty base model runs, before repetitions, grading calls or red-team generation. Budget those additional calls rather than treating twenty cases as twenty requests.
promptfoo alternatives for local cli evaluations and ai red teaming
Compare promptfoo with Arize Phoenix, LangWatch for this workflow, including the billing unit and the work left to your team.
| Product | Price / unit | Free allowance | Input → output | Key limit |
|---|---|---|---|---|
| promptfooRead profileOfficial source ↗ | MIT CLI software; inference billed separately | No core CLI licence fee | Test cases and model configurations yield comparative evaluations. | Remote providers can receive prompts despite local orchestration. |
| Arize PhoenixRead profileOfficial source ↗ | ELv2 self-hosted software; operating costs separate | No listed self-hosted subscription charge | Instrumented spans become traces and experiment datasets. | ELv2 restricts offering substantial features as a managed service. |
| LangWatchRead profileOfficial source ↗ | Growth EUR 29/core seat/month | Developer includes 50,000 monthly events | Agent traces and datasets yield evaluation results. | Calls and evaluation steps can count as separate events. |
Setup & workflow
- Initialise a small example configuration and replace it with approved prompts.
- Run the evaluation and inspect the local comparison view.
- Add meaningful assertions before using results as a CI gate.
Requirements & installation
- Prepare a supported CLI runtime, an approved test target and restricted provider credentials for the chosen configuration.
Limits & review
- Provider keys and remote model calls are separate dependencies; automated scores and red-team findings require human interpretation.
Your part
- Review failed assertions and false positives and keep attack testing within the approved target scope.
Sources & verification5
This profile is based on official sources, not a hands-on product test.
Vendor descriptions and demos document advertised features. Editorial guidance is based on these sources.
- Official promptfoo pricinggithub.com
- Official promptfoo productpromptfoo.dev
- promptfoo official referencegithub.com
- Arize Phoenix official referencegithub.com
- LangWatch official referencelangwatch.ai
