Introducing Agent Effectiveness

Agent Effectiveness is available now in Factory Analytics. It connects Factory sessions to cycle time, work intent, and shipped artifacts, so engineering leaders can see where delivery is moving faster, what Droids are working on, and how Factory usage maps to shipped work.

The gap between usage and impact

Engineering output increases in scale and velocity with Droid. Proving how much faster, where the change is concentrated, and what caused it is a different problem.

Engineering leaders are spending more on AI, but most still measure the return with a mix of usage dashboards, delivery metrics, and surveys. Those signals leave a basic question unanswered: what is our AI spend producing?

Agent Effectiveness is Factory's answer. It connects Droid usage to delivery outcomes, giving leaders an operational view of AI ROI: how much faster teams ship, where gains are concentrated, and whether added capacity is reaching priority work.

Three questions engineering leaders need answered

Agent Effectiveness reads from the project management, issue-tracking, and source control tools you already use, then links Factory sessions to the work those systems track.

It turns that data into answers to three questions.

Are we shipping faster?

The Throughput view shows how project, issue, and pull request cycle times change alongside Factory usage, and where those changes are concentrated. You can compare usage across projects, pods, and users against your targets for the quarter.

Are Droids working on the right priorities?

The Output view maps spend to intent. Session intents classify work as feature engineering, maintenance, bug fixing, or exploration. You can compare the resulting split of engineer time and Factory Standard Credits against your plan. If the mix drifts, you can correct it during the quarter instead of explaining it afterward.

What was the output?

The Attribution view shows output by project, issue, session, and user. Using local signals, you can trace a session back to the specific issues, projects, and artifacts a user worked on. Organization-level spend is no longer an abstract number. It is tied to concrete work.

How to enable it

Agent Effectiveness relies on connecting Factory to the tools where your work lives and enabling the Advanced Analytics enterprise control. Setup is an admin task and takes a few minutes.

1. Connect your organization integrations

Configure your organization-level integrations for the tools you use: Jira and Linear for issue tracking and project management, GitHub and GitLab for source control.

These integrations let Factory map sessions to issues, projects, and pull requests. Attribution and the Output view only cover the systems you connect, so connect every tool your teams track work in.

2. Enable the Advanced Analytics control

Turn on the Advanced Analytics enterprise control to make the Throughput, Output, and Attribution views available to your organization. Advanced Analytics includes the local signals that link sessions to work streams and artifacts, and automatically backfills effectiveness data to the start of the account.

Once these are set, the dashboards populate on their own. There is no per-repository configuration. Full setup instructions are in the Agent Effectiveness documentation.

From adoption metrics to operating decisions

Measuring adoption is easy. Measuring impact is what changes decisions.

Agent Effectiveness gives engineering leaders the evidence to direct resources toward priority work, see where Droids improve delivery, and correct course while there is still time to act. Autonomous engineering should be measured by outcomes, not logins, session counts, or survey estimates.

Agent Effectiveness is currently in Private Preview. To request access, contact Factory or contact your Factory account team.