Factory.ai

Strong Testing

Build Service reaches Level 3 with 67% Testing pass rate. Currently becoming autonomous-capable with 22/54 criteria passing (41%). Key areas for improvement include the opportunities listed below.

Strengths

01
Dead Code Detection
Rust compiler warns about unused/dead code with dead_code lint
02
Formatter
cargo fmt documented in AGENTS.md as pre-commit requirement (Rust standard formatter)
03
Lint Config
cargo clippy documented in AGENTS.md as pre-commit requirement (Rust standard linter)

Opportunities

01
Cyclomatic Complexity
Add complexity analysis to identify and refactor overly complex functions.
02
Feature Flag Infrastructure
Add feature flags to enable safer deployments and gradual rollouts.
03
Test Coverage Thresholds
Establish minimum test coverage thresholds to prevent regression and maintain code quality.

All Criteria

Style & Validation6/11 (55%)
—code_modularizationSkipped: Small project (17 .rs files), Rust compiler enforces module visibility boundaries
✗cyclomatic_complexityNo complexity analysis configured (would require cargo-geiger, clippy complexity rules, or similar)
✓dead_code_detectionRust compiler warns about unused/dead code with dead_code lint
✗duplicate_code_detectionNo duplicate code detection configured (would need jscpd, PMD CPD, or similar)
✓formattercargo fmt documented in AGENTS.md as pre-commit requirement (Rust standard formatter)
✗large_file_detectionNo git hooks, CI checks, LFS configuration, or linter rules for large file detection found
✓lint_configcargo clippy documented in AGENTS.md as pre-commit requirement (Rust standard linter)
—n_plus_one_detectionSkipped: No database or ORM usage in this build service application
✓naming_consistencyRust compiler enforces naming conventions (snake_case, CamelCase) with warnings on violations
✗pre_commit_hooksNo pre-commit hooks configured (no .pre-commit-config.yaml, husky, or git hooks)
✓strict_typingRust is strictly typed by default with compile-time type checking
✗tech_debt_trackingNo TODO/FIXME scanner in CI, linter rules enforcing issue links, or tech debt tracking tools configured
✓type_checkRust has built-in strict type checking via rustc compiler
Build System5/9 (56%)
✗agentic_developmentNo evidence of agent co-authorship in git log (checked 100 commits) or agent-invoking CI workflows
—automated_pr_reviewSkipped: gh CLI available but no automated review generation found in PR comments or CI workflows
✓build_cmd_docREADME documents build command: cargo build --release
—build_performance_trackingSkipped: No CI/CD configured, no build caching or metrics export to monitoring found
—dead_feature_flag_detectionSkipped: Prerequisite not met (feature_flag_infrastructure must pass first)
—deployment_frequencySkipped: 7 releases exist but created manually via release.mjs script, not automated CD pipeline
✓deps_pinnedCargo.lock is committed providing dependency pinning for Rust project
—fast_ci_feedbackSkipped: No CI/CD configured (PRs have no status checks, empty statusCheckRollup)
✗feature_flag_infrastructureNo feature flag system (LaunchDarkly, Statsig, Unleash, GrowthBook, or custom) configured
—heavy_dependency_detectionSkipped: Backend service, not a bundled frontend application requiring bundle size analysis
—monorepo_toolingSkipped: Single-application repository with one Cargo.toml at root
—progressive_rolloutSkipped: Infrastructure tool without user-facing deployment requiring canary or percentage-based rollouts
✓release_automationrelease.mjs automates version bumps, CHANGELOG updates, git tags, and GitHub release creation
✓release_notes_automationrelease.mjs script automates release note generation from CHANGELOG.md sections
—rollback_automationSkipped: Infrastructure tool without deployment infrastructure requiring rollback capability
✗single_command_setupNo single command from clone to running dev environment; requires cargo build, manual installation, and config setup
✗unused_dependencies_detectionNo cargo-udeps or similar unused dependency detection configured
✓vcs_cli_toolsgh CLI is available and authenticated (verified with gh auth status)
—version_drift_detectionSkipped: Single-application repository, no multiple packages to detect version drift across
Testing4/6 (67%)
—flaky_test_detectionSkipped: No CI/CD configured to check for test retries or flaky test tracking
✓integration_tests_existtests/build_cli_connection.rs contains integration tests for client connection behavior
✗test_coverage_thresholdsNo coverage thresholds enforced (no tarpaulin config, codecov, or CI coverage gates)
✓test_isolationcargo test runs tests in parallel by default providing test isolation
✓test_naming_conventionsRust conventions followed: *_test.rs files and #[test] attributes for test functions
✗test_performance_trackingNo test timing output configuration or test analytics platform integration
✓unit_tests_existTests present in tests/ directory and embedded #[cfg(test)] modules in source files
—unit_tests_runnableSkipped: OSS evaluation mode without fully configured local development environment
Documentation3/8 (38%)
✓agents_mdAGENTS.md exists at repo root with development guidelines, workflow, and changelog conventions (>100 chars)
✗agents_md_validationNo CI job, automated AGENTS.md generation, pre-commit hook, or doc testing validating AGENTS.md commands work
✗api_schema_docsNo OpenAPI/Swagger spec for HTTP API (protocol defined in Rust structs with serde only)
✗automated_doc_generationNo automated doc generation configured (no rustdoc workflow, API doc generators, or changelog automation beyond manual script)
✓documentation_freshnessAGENTS.md was modified within last 180 days (git log confirms recent update)
✓readmeREADME.md exists with comprehensive setup, usage, architecture, and configuration instructions
✗service_flow_documentedASCII art diagrams in README but no architecture diagram files (*.mermaid, *.puml, images) or formal service dependency docs
✗skillsNo skills directory found in standard locations (.factory/skills/, .skills/, .claude/skills/)
Dev Environment1/2 (50%)
—database_schemaSkipped: No database usage in application
✗devcontainerNo .devcontainer/devcontainer.json configuration found
—devcontainer_runnableSkipped: No devcontainer configured to test runnability
✓env_templateEnvironment variables documented in README (BUILD_SERVICE_ENDPOINT, BUILD_SERVICE_TOKEN, etc.)
—local_services_setupSkipped: Standalone service with no external dependencies (no database, Redis, etc. required)
Debugging & Observability2/8 (25%)
✗alerting_configuredNo PagerDuty, OpsGenie, or custom alerting rules configured
—circuit_breakersSkipped: No external service dependencies requiring circuit breaker pattern
—code_quality_metricsSkipped: 403 from code-scanning API (requires admin access not available for OSS evaluation)
✗deployment_observabilityNo monitoring dashboard links or deployment notification integrations in documentation
✓distributed_tracingrequest_id field propagated through build requests for tracing across components
✗error_tracking_contextualizedNo Sentry, Bugsnag, or Rollbar error tracking with source maps and context configured
✗health_checksNo /health, /healthz, or /ready endpoints implemented (service has HTTP API but no health checks)
✗metrics_collectionNo metrics/telemetry instrumentation (Datadog, Prometheus, CloudWatch, etc.) configured
—profiling_instrumentationSkipped: Infrastructure tool where performance profiling is not critical
✗runbooks_documentedNo runbooks directory or external pointers to incident response procedures in README/AGENTS.md/docs
✓structured_loggingUses tracing crate for structured logging with configurable levels and output
Security1/5 (20%)
—automated_security_reviewSkipped: 403 from code-scanning API (requires admin:repo_hook scope not available for OSS evaluation)
—branch_protectionSkipped: 404 from rulesets and branch protection API (requires admin access not available for OSS evaluation)
✗codeownersNo CODEOWNERS file in root or .github/ directory
—dast_scanningSkipped: Primarily Unix socket service, not public-facing web service requiring DAST
✗dependency_update_automationNo Renovate or Dependabot configuration found
✓gitignore_comprehensive.gitignore properly excludes .env files, target/ build artifacts, .DS_Store, and temp files
✗log_scrubbingNo log sanitization/redaction configured in tracing setup (no redact patterns or scrubbing middleware)
—pii_handlingSkipped: Infrastructure tool without personal/user data processing
—privacy_complianceSkipped: Infrastructure tool without end-user data collection (internal build service)
—secret_scanningSkipped: 404 from secret scanning API (requires admin access not available for OSS evaluation)
✗secrets_managementNo secrets manager integration (AWS Secrets Manager, Vault, SOPS, etc.) or encrypted secrets pattern found
Task Discovery0/3 (0%)
—backlog_healthSkipped: No open issues to assess backlog health
✗issue_labeling_systemNo open issues to verify consistent labeling system (priority, type, area labels)
✗issue_templatesNo .github/ISSUE_TEMPLATE/ directory with issue templates
✗pr_templatesNo .github/pull_request_template.md for PR description structure
Product & Analytics0/2 (0%)
✗error_to_insight_pipelineNo error tracking to issue creation integration (Sentry-GitHub, automated issue creation from errors)
✗product_analytics_instrumentationNo product analytics (Mixpanel, Amplitude, PostHog, etc.) instrumented

start building

Ready to build the software of the future?

Start building

Arrow Right Icon