iFixAi專做智能體獨立審計

mainBranchesTagsGo to fileCodeOpen more actions menuLatest commit History285 Commits285 CommitsFolders and filesNameNameLast commit messageLast commit date.agents/plugins.agents/plugins .claude-plugin.claude-plugin .github.

github casestudiescasestudies docsdocs examplesexamples ifixaiifixai pluginplugin scriptsscripts .gitattributes.gitattributes .gitignore.gitignore .pre-commit-config.yaml.pre-commit-config.yaml CONTRIBUTING.mdCONTRIBUTING.md LICENSELICENSE README.ja.mdREADME.ja.md README.ko.mdREADME.ko.md README.

mdREADME.md README.zh-CN.mdREADME.zh-CN.md SECURITY.mdSECURITY.md conftest.pyconftest.py countfixturefields.pycountfixturefields.py pyproject.tomlpyproject.

toml View all filesRepository files navigation English · 簡體中文 · 日本語 · 한국어 Catch your agent's mistakes and blind spots before the shit hits the fan.

Quick start • Three ways to run • Test your agent • Scoring • Audit recorded actions • Docs • Contributing What it is The existing Eval, Red-teaming, and Observability Tools are evaluating the agent mainly based on tech capability (token efficiency, latency, prompt injections).

They cannot answer the most crucial question.Is the agent doing the job it is supposed to do based on the business KPIs and Organizational Structure?iFixAi gives you this answer in less than 120 seconds by striking the right balance between AI-Red Teaming and Operational Assurance.Adversarial depth.

Assurance discipline.All-in-one auditing process.Three ways to run All three run the same diagnostic underneath.The difference is how you configure and drive it.

CLI: guided wizard CLI: explicit flags Plugin or Skill How you drive it ifixai setup once → ifixai run zero-flag every time; config saved to ifixai.

yaml pass every option as a CLI flag; fully scriptable the agent is the operator: discovers your setup, builds the fixture, runs it, and explains the scorecard Best for first-time users, fast repeatable runs, team onboarding CI, automation, audit-ready scripted batches a guided, explained run with an interactive scorecard, inside the agent you already use Setup pip install "ifixai[]" + ifixai setup pip install "ifixai[]" + export keys Claude Code or Codex: install the plugin (self-provisions).

Any agent: uvx ifixai install scaffolds /ifixai-skill Keys auto-detected by wizard; stored as env-var name in ifixai.

yaml, never the secret itself --api-key flag or env var each provider's key from its environment variable, never on the command line What you test any provider, or your agent's real endpoint same same Who grades it self, one independent vendor, or a multi-judge ensemble same same Output JSON + Markdown reports + rich terminal scorecard same interactive results artifact (+ JSON source of truth; static-report fallback) Suite pick with arrow keys in the wizard 10 inspections by default; --full-run for all 60, or --suite smoke|strategic|essential|core|extended|all the agent picks --mode/--suite, same engine as the CLI Works in any terminal any terminal / CI Claude Code, Cursor, Codex, VS Code, Windsurf, Cline, Continue, Gemini, Zed Quick start Explore business-policy compositions: Policy Lab finds shortest harmful sequences of permitted actions and checks permission repairs against legitimate completion goals.

Includes an offline refund-store example and explicit local-callable replay.Now try it yourself.Pick a path from the table above; full walkthrough: docs/get-started.md.Guided wizard (recommended) pip install "ifixai[openai]" # or anthropic, gemini, etc.

: install the provider extra you'll test ifixai setup # arrow-key wizard: pick provider, model, judge, suite → writes ifixai.yaml ifixai run # no flags needed; reports land in ./ifixai-results/ ifixai setup detects API keys already in your environment and surfaces them at the top of each prompt.

No key found?The wizard tells you which env var to export; if it's still missing when you run, you'll be prompted for it before the first API call.Windows note: if PowerShell can't find ifixai after pip install, add Python's Scripts\ folder to your PATH, or run it as python -m ifixai.

This is the usual Python-on-Windows PATH gap, not an iFixAi issue.Plugin (Claude Code and Codex) The recommended way to run from an agent: a one-time native install with an auto-provisioning hook, so there is nothing to set up per run.

Ask in plain English ("run iFixAi on my setup") and the agent discovers your config, builds the fixture, names the cost before anything is billed, runs the diagnostic on the model(s) and judge(s) you pick, then walks you through the scorecard.

Claude Code, from inside Claude Code: /plugin marketplace add ifixai-ai/iFixAi /plugin install ifixai@ifixai-community Then ask "run iFixAi on my setup", or type /ifixai:ifixai.(Restart Claude Code or run /reload-plugins if it doesn't appear.) Already on ifixai@ifixai-ai?

That keeps working, and to move to the new marketplace name you run /plugin marketplace remove ifixai-ai first, then the two commands above.

Codex, in your terminal: codex plugin marketplace add ifixai-ai/iFixAi codex plugin add ifixai@ifixai-community Then start Codex and ask "run iFixAi on my setup".Codex asks once to trust the plugin's hook, then provisions the engine on the first session.Already on ifixai@ifixai-ai?

codex plugin marketplace upgrade fails on the renamed marketplace, so run codex plugin marketplace remove ifixai-ai first, then the two commands above.Skill (every agent) Prefer a single scaffolded file, or use an agent without a plugin?

One zero-install command writes a native /ifixai-skill slash command into any agent: Claude Code, Codex, Cursor, VS Code / Copilot, Windsurf, Cline, Continue, Gemini, or Zed (plus an AGENTS.md bridge).Only uv and Python 3.

10+ are needed; no API key or provider extra to scaffold: uvx ifixai install --agents cursor # any slug: claude, codex, vscode, windsurf, cline, continue, gemini, zed uvx ifixai install --agents all # scaffold every agent at once uvx ifixai install --list # every supported agent and where its file lands Then run /ifixai-skill in that agent.

It reads your setup, builds the fixture, shows the cost via a free --dry-run, and runs only after you say yes (the run is zero-install too, driving uvx --from "ifixai[]" ifixai run).On a new project, name the agent with --agents (auto-detect only finds agents whose folder already exists).

Already have the CLI on your PATH?Drop the uvx prefix.The command is named ifixai-skill so it never collides with the Claude Code plugin's /ifixai; pass --name ifixai for the bare name.Explicit flags # 1.Install the CLI + the extra for the provider you'll test pip install "ifixai[anthropic]" # 2.

Prove the pipeline runs: built-in mock, no keys, no network, a few seconds.# Runs the default 10 inspections.Expect a FAILING scorecard (B01 and P01 # fail) — the bundled default fixture ships seeded defects on purpose so you # see what failures look like.Defect map: ifixai/fixtures/default/README.

md ifixai run --provider mock --api-key not-used --eval-mode self # 3.Get a citable grade: your model graded by a different vendor's judge.# Pass --fixture : without it the seeded-defect default # is used and its failures land on YOUR scorecard.

pip install "ifixai[anthropic,openai]" # SUT's + judge's SDKs (or ifixai[all]) export ANTHROPICAPIKEY=sk-ant-...# the SUT, graded export OPENAIAPIKEY=sk-...# the judge, auto-paired from the environment ifixai run --provider anthropic --api-key "$ANTHROPICAPIKEY" --fixture ./my-fixture.

yaml A grade is citable when a second, independent provider graded your agent, not the agent grading itself.

Every run has two roles, so a citable run needs two keys, one per role, from different vendors: Role What it is How you set it SUT (system under test) the agent/model being graded --provider + --api-key; the SUT key is always passed explicitly, never read from the environment Judge who grades it auto-paired from a different provider whose key is in your environment (the SUT's own vendor is excluded, so it never grades itself) Reports land in .

/ifixai-results/ as JSON and Markdown.Without a second key, add --eval-mode self to run as a smoke test (the grade still prints, but it's flagged as self-judged, not a result you can cite).Pinning the judge, Full-mode ensembles, and the eval modes: docs/cli.md.

Other providers (OpenAI, Atlas Cloud, OpenRouter, OrcaRouter, Requesty, Cloudflare AI Gateway, Vercel AI Gateway, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; the HTTP and LangChain adapters need no provider extra: docs/testing-your-agent.md.

Recommended judge setups The judge grades your agent's answers.Two reliable setups: Setup Judge model(s) Est.cost, full suite Single judge: Sonnet anthropic/claude-sonnet-4.6 ~$12–18 More affordable: two judges google/gemini-2.5-pro + openai/gpt-5.4-mini ~$10–14 combined Both are reliable.

Sonnet is the simplest, highest-quality single grader.Gemini 2.5 Pro and GPT-5.

4-mini are strong, capable models from two different vendors; running them as a pair still comes in under a single Sonnet run and adds cross-vendor robustness, so no one model or vendor decides your grade (ties break conservatively, fail > partial > pass).

# Single judge (Standard mode): Sonnet grades your agent --eval-mode single --judge-provider openrouter --judge-model anthropic/claude-sonnet-4.

6 # Two affordable judges (Full mode; needs a hand-built --fixture), both on one OpenRouter key --mode full --eval-mode full \ --judge-provider openrouter --judge-model google/gemini-2.5-pro \ --judge-provider openrouter --judge-model openai/gpt-5.

4-mini Rough total for one full-suite run at OpenRouter list prices (mid-2026), based on the ~2,000 judge calls a full run makes (the suite generates far more probes than its 50-test count, so the figure is fairly stable across fixtures).The agent under test is billed separately.

Full mode needs a hand-built fixture: docs/fixture_authoring.md.These are full-suite figures (--full-run).The default 10-inspection run makes about a sixth of the calls, so expect roughly a sixth of the cost and time.What a default run covers ifixai run runs 10 inspections unless you ask for more.

They are the essential suite: six high-impact checks plus four medium-impact ones, picked so a first run is quick and still ends in a grade.Add --full-run to run all 60.

Impact ID Checks that your agent… High B01 only lets a role use the tools it is allowed to use · mandatory minimum High B08 cannot be talked into more access than the user has · mandatory minimum High P01 never runs a destructive action for someone not allowed to · mandatory minimum High B12 resists prompt injection High B09 spots and blocks requests that break your policies High B27 does not hand one user's private data to another Medium B06 says when it is unsure instead of guessing with confidence Medium B16 reports a failure instead of pretending it worked Medium B18 sticks to the task it was given Medium B20 follows exact instructions on format, length, and content It still produces a grade.

The three mandatory minimums are always included (a run without them has its overall score withheld), and the ten cover all five graded pillars.It runs on any agent.Eight of the ten only need your agent to answer chat messages.

B01 and P01 read its tool-authorization layer instead; what they report when none is exposed: docs/scoring.md.About 6x less work.On the bundled default fixture a default run makes about 850 calls to your agent and about 820 judge calls; --full-run makes about 5,500 and 4,700.

ifixai run --provider openai # the default: 10 inspections ifixai run --provider openai --full-run # all 60: slower, costs more --full-run is the old default behaviour, unchanged.

It cannot be combined with --suite, --test, --category, or --strategic, and it has nothing to do with --mode full, which sets how a run is