Factory.ai

Strong Dev Environment

Keystone reaches Level 3 with 100% Dev Environment pass rate. Currently becoming autonomous-capable with 28/58 criteria passing (48%). Key areas for improvement include the opportunities listed below.

Strengths

01
Dev Environment (100%)
Includes Database Schema, Devcontainer, Env Template.
02
Formatter
Prettier configured with .prettierrc.json defining code formatting rules
03
Lint Config
ESLint configured with typescript-eslint strict config in eslint.config.mjs

Opportunities

01
Cyclomatic Complexity
No complexity rules configured in ESLint. No evidence of complexity analysis tooling.
02
Dead Code Detection
ESLint no-unused-vars rule is explicitly disabled. No knip, ts-prune, or similar tools configured.
03
Duplicate Code Detection
No jscpd, PMD CPD, or similar duplicate code detection tools found in CI or pre-commit

All Criteria

Style & Validation4/11 (36%)
—code_modularizationSkipped - monorepo with clear package boundaries via pnpm workspaces, boundaries are organizational not enforced
✗cyclomatic_complexityNo complexity rules configured in ESLint. No evidence of complexity analysis tooling.
✗dead_code_detectionESLint no-unused-vars rule is explicitly disabled. No knip, ts-prune, or similar tools configured.
✗duplicate_code_detectionNo jscpd, PMD CPD, or similar duplicate code detection tools found in CI or pre-commit
✓formatterPrettier configured with .prettierrc.json defining code formatting rules
✗large_file_detectionNo git hooks, CI jobs, or linter rules checking file size/line count found
✓lint_configESLint configured with typescript-eslint strict config in eslint.config.mjs
—n_plus_one_detectionSkipped - framework/library repository, not an application with database/ORM usage patterns to detect
✗naming_consistencyNo @typescript-eslint/naming-convention rules or documented naming conventions found in ESLint config
✗pre_commit_hooksNo Husky, lint-staged, or .pre-commit-config.yaml found. Code quality checks only run in CI.
✓strict_typingTypeScript strict mode enabled in tsconfig.json with 'strict': true
✗tech_debt_trackingNo TODO/FIXME scanner, linter rules for tech debt comments, or SonarQube configuration found
✓type_checkTypeScript configured with 'strict': true in tsconfig.json
Build System8/12 (67%)
✗agentic_developmentNo agent co-authorship in recent 100 commits. No Factory/Claude agent workflows or config directories found.
—automated_pr_reviewSkipped - gh CLI available but no evidence of automated review generation (Danger, AI bots) found
✓build_cmd_docREADME documents getting started. Package.json scripts include 'build' command clearly listed.
—build_performance_trackingSkipped - no build caching, metrics export, or deliberate build optimization tracking found in CI
—dead_feature_flag_detectionSkipped - prerequisite feature_flag_infrastructure not met, no feature flags to detect as stale
✓deployment_frequencyRegular releases via gh release list: 10 releases from Feb 2024 to May 2025, averaging 1-2 per month
✓deps_pinnedpnpm-lock.yaml committed, ensuring reproducible dependency installations
✓fast_ci_feedbackCI checks complete in ~2-3 minutes based on statusCheckRollup data from recent PRs, well under 10 min threshold
✗feature_flag_infrastructureNo LaunchDarkly, Statsig, Unleash, GrowthBook, or custom feature flag system found
—heavy_dependency_detectionSkipped - backend library/framework, not a bundled frontend application requiring bundle size analysis
✗monorepo_toolingUses pnpm workspaces and preconstruct but lacks Turborepo/Nx/Lerna for managed builds and caching
—progressive_rolloutSkipped - library/framework repo publishing to npm, not an infrastructure deployment with canary/percentage rollouts
✓release_automationChangesets + publish.yml workflow automates npm publishing on workflow_dispatch trigger
✓release_notes_automationChangesets configured (.changeset/ directory with config.json) for automated changelog and release notes generation
—rollback_automationSkipped - library/framework repo, rollback via npm package versions handled by package managers, not infra-based
✓single_command_setupREADME documents setup via pnpm install. Devcontainer has postCreateCommand: pnpm install for automated setup.
✗unused_dependencies_detectionNo depcheck, npm-check, knip, or similar unused dependency detection tools configured
✓vcs_cli_toolsgh CLI authenticated and working, confirmed via 'gh auth status' returning success
—version_drift_detectionSkipped - monorepo but no syncpack/manypkg for version drift detection, Renovate handles updates uniformly
Testing4/7 (57%)
—flaky_test_detectionSkipped - no test retry config (jest-retry, pytest-rerunfailures) or flaky test tracking tools found
✗integration_tests_existNo Cypress or Playwright configuration found. Admin UI tests exist but are Jest-based, not E2E browser tests.
✗test_coverage_thresholdsJest coverage script exists but no coverageThreshold in config. No Codecov/Coveralls PR checks enforcing minimums.
✓test_isolationJest default parallel execution (not --runInBand). Vitest globals configured supporting isolated test runs.
✓test_naming_conventionsJest/Vitest configs imply .test.ts naming pattern, consistent across codebase with __tests__/ directories
✗test_performance_trackingNo test timing output flags, test analytics platforms, or evidence of test performance monitoring
✓unit_tests_existExtensive test suite with .test.ts files across packages, api-tests, admin-ui-tests directories
✓unit_tests_runnableTest scripts 'pnpm test' (Jest) and 'pnpm test:vitest' configured and runnable in package.json
Documentation3/6 (50%)
✗agents_mdNo AGENTS.md file found at repository root to document setup for AI agents
—agents_md_validationSkipped - prerequisite agents_md not met, no AGENTS.md file to validate
—api_schema_docsSkipped - GraphQL schema files exist but are example projects, not API docs for the framework itself
✓automated_doc_generationChangesets generates CHANGELOG.md files automatically. Documentation site exists (docs/) but manual content.
✓documentation_freshnessCONTRIBUTING.md was modified within last 180 days per git log check
✓readmeREADME.md exists with project description, usage documentation, and links to extended docs website
✗service_flow_documentedNo architecture diagrams (.mermaid, .puml) or service dependency documentation found in docs/
✗skillsNo skills directories (.factory/skills/, .skills/, .claude/skills/) found in repository
Dev Environment3/3 (100%)
✓database_schemaPrisma schema.prisma files exist throughout examples and test projects defining data models
✓devcontainer.devcontainer/devcontainer.json configured with Node.js image, Prisma, GraphQL extensions, and pnpm install
—devcontainer_runnableSkipped - devcontainer CLI not available to verify container can be built and run successfully
✓env_template.env.example files exist in examples (docs, cloudinary, assets-s3) documenting required environment variables
—local_services_setupSkipped - no docker-compose.yml for local dependencies. CI uses service containers for postgres in tests.
Debugging & Observability1/7 (14%)
✗alerting_configuredNo PagerDuty, OpsGenie, or custom alerting rules found in documentation or code
—circuit_breakersSkipped - library/framework without external service dependencies requiring circuit breaker patterns
—code_quality_metricsSkipped - no code scanning, coverage bots in PR comments, or SonarQube/quality gate enforcement found
✗deployment_observabilityNo monitoring dashboard links (Datadog, Grafana) or deployment notification integrations found in docs
✓distributed_tracingOpenTelemetry configured in logging-opentelemetry example for trace ID propagation
✗error_tracking_contextualizedNo Sentry, Bugsnag, or Rollbar configuration found for error tracking with source maps and context
—health_checksSkipped - library/framework, not a deployed service requiring /health endpoints or liveness probes
✗metrics_collectionNo metrics/telemetry libraries (Datadog, Prometheus, New Relic) found in dependencies
—profiling_instrumentationSkipped - library/framework where profiling is for users' applications, not the framework itself
✗runbooks_documentedNo runbooks/ directory or links to incident response procedures in README, AGENTS.md, or docs/
✗structured_loggingNo structured logging library (winston, pino, bunyan) in root package.json. Logging example uses OpenTelemetry.
Security4/6 (67%)
—automated_security_reviewSkipped - Socket Security check in CI is dependency scanning, not security review generation. No SAST reports found.
—branch_protectionRequires GitHub admin API access to verify branch protection rules (skipped).
✓codeownersCODEOWNERS file exists at .github/workflows/CODEOWNERS with team assignments (dcousens, mitchellhamilton)
—dast_scanningSkipped - library/framework not deployed as web service requiring OWASP ZAP or dynamic security testing
✓dependency_update_automationRenovate configured (renovate.json) with scheduled dependency updates and PR creation
✓gitignore_comprehensive.gitignore properly excludes .env, node_modules, dist, .DS_Store, and .keystone build artifacts
✗log_scrubbingNo log sanitization/scrubbing mechanisms configured or documented in logging utilities
—pii_handlingSkipped - framework/library doesn't process user data directly, PII handling is responsibility of applications built with it
—privacy_complianceSkipped - framework without end-user data collection, privacy compliance applies to applications using Keystone
—secret_scanningSkipped - gh api secret-scanning/alerts returns 404, feature not enabled. No gitleaks/trufflehog in CI.
✓secrets_management.env properly gitignored, .env.example templates provided, GitHub Actions uses secrets.* references in workflows
Task Discovery1/4 (25%)
✗backlog_healthMost open issues lack labels. Many old issues (>1 year) with no activity. Only ~6% have labels vs 70% threshold.
✗issue_labeling_systemPoor labeling: only 3 of 50 open issues have labels. No consistent priority/type/area label taxonomy.
✓issue_templates.github/ISSUE_TEMPLATE/ directory exists with bug report template (1.Bug_report.md)
✗pr_templatesNo .github/pull_request_template.md or GitLab merge request templates found
Product & Analytics0/2 (0%)
✗error_to_insight_pipelineNo Sentry-GitHub integration or error-to-issue automation found in workflows or configurations
✗product_analytics_instrumentationNo Mixpanel, Amplitude, PostHog, Heap, or GA4 instrumentation found in dependencies

start building

Ready to build the software of the future?

Start building

Arrow Right Icon