Factory.ai

Strong Dev Environment

Keystone reaches Level 3 with 100% Dev Environment pass rate. Currently becoming autonomous-capable with 28/58 criteria passing (48%). Key areas for improvement include the opportunities listed below.

Strengths

01
Dev Environment (100%)
Includes Database Schema, Devcontainer, Env Template.
02
Formatter
Prettier configured with .prettierrc.json defining code formatting rules
03
Lint Config
ESLint configured with typescript-eslint strict config in eslint.config.mjs

Opportunities

01
Cyclomatic Complexity
Add complexity analysis to identify and refactor overly complex functions.
02
Feature Flag Infrastructure
Add feature flags to enable safer deployments and gradual rollouts.
03
Monorepo Tooling
Consider Turborepo or Nx for better caching, incremental builds, and dependency graph awareness.

All Criteria

Style & Validation4/11 (36%)
—code_modularizationSkipped - monorepo with clear package boundaries via pnpm workspaces, boundaries are organizational not enforced
✗cyclomatic_complexityNo complexity rules configured in ESLint. No evidence of complexity analysis tooling.
✗dead_code_detectionESLint no-unused-vars rule is explicitly disabled. No knip, ts-prune, or similar tools configured.
✗duplicate_code_detectionNo jscpd, PMD CPD, or similar duplicate code detection tools found in CI or pre-commit
✓formatterPrettier configured with .prettierrc.json defining code formatting rules
✗large_file_detectionNo git hooks, CI jobs, or linter rules checking file size/line count found
✓lint_configESLint configured with typescript-eslint strict config in eslint.config.mjs
—n_plus_one_detectionSkipped - framework/library repository, not an application with database/ORM usage patterns to detect
✗naming_consistencyNo @typescript-eslint/naming-convention rules or documented naming conventions found in ESLint config
✗pre_commit_hooksNo Husky, lint-staged, or .pre-commit-config.yaml found. Code quality checks only run in CI.
✓strict_typingTypeScript strict mode enabled in tsconfig.json with 'strict': true
✗tech_debt_trackingNo TODO/FIXME scanner, linter rules for tech debt comments, or SonarQube configuration found
✓type_checkTypeScript configured with 'strict': true in tsconfig.json
Build System8/12 (67%)
✗agentic_developmentNo agent co-authorship in recent 100 commits. No Factory/Claude agent workflows or config directories found.
—automated_pr_reviewSkipped - gh CLI available but no evidence of automated review generation (Danger, AI bots) found
✓build_cmd_docREADME documents getting started. Package.json scripts include 'build' command clearly listed.
—build_performance_trackingSkipped - no build caching, metrics export, or deliberate build optimization tracking found in CI
—dead_feature_flag_detectionSkipped - prerequisite feature_flag_infrastructure not met, no feature flags to detect as stale
✓deployment_frequencyRegular releases via gh release list: 10 releases from Feb 2024 to May 2025, averaging 1-2 per month
✓deps_pinnedpnpm-lock.yaml committed, ensuring reproducible dependency installations
✓fast_ci_feedbackCI checks complete in ~2-3 minutes based on statusCheckRollup data from recent PRs, well under 10 min threshold
✗feature_flag_infrastructureNo LaunchDarkly, Statsig, Unleash, GrowthBook, or custom feature flag system found
—heavy_dependency_detectionSkipped - backend library/framework, not a bundled frontend application requiring bundle size analysis
✗monorepo_toolingUses pnpm workspaces and preconstruct but lacks Turborepo/Nx/Lerna for managed builds and caching
—progressive_rolloutSkipped - library/framework repo publishing to npm, not an infrastructure deployment with canary/percentage rollouts
✓release_automationChangesets + publish.yml workflow automates npm publishing on workflow_dispatch trigger
✓release_notes_automationChangesets configured (.changeset/ directory with config.json) for automated changelog and release notes generation
—rollback_automationSkipped - library/framework repo, rollback via npm package versions handled by package managers, not infra-based
✓single_command_setupREADME documents setup via pnpm install. Devcontainer has postCreateCommand: pnpm install for automated setup.
✗unused_dependencies_detectionNo depcheck, npm-check, knip, or similar unused dependency detection tools configured
✓vcs_cli_toolsgh CLI authenticated and working, confirmed via 'gh auth status' returning success
—version_drift_detectionSkipped - monorepo but no syncpack/manypkg for version drift detection, Renovate handles updates uniformly
Testing4/7 (57%)
—flaky_test_detectionSkipped - no test retry config (jest-retry, pytest-rerunfailures) or flaky test tracking tools found
✗integration_tests_existNo Cypress or Playwright configuration found. Admin UI tests exist but are Jest-based, not E2E browser tests.
✗test_coverage_thresholdsJest coverage script exists but no coverageThreshold in config. No Codecov/Coveralls PR checks enforcing minimums.
✓test_isolationJest default parallel execution (not --runInBand). Vitest globals configured supporting isolated test runs.
✓test_naming_conventionsJest/Vitest configs imply .test.ts naming pattern, consistent across codebase with __tests__/ directories
✗test_performance_trackingNo test timing output flags, test analytics platforms, or evidence of test performance monitoring
✓unit_tests_existExtensive test suite with .test.ts files across packages, api-tests, admin-ui-tests directories
✓unit_tests_runnableTest scripts 'pnpm test' (Jest) and 'pnpm test:vitest' configured and runnable in package.json
Documentation3/6 (50%)
✗agents_mdNo AGENTS.md file found at repository root to document setup for AI agents
—agents_md_validationSkipped - prerequisite agents_md not met, no AGENTS.md file to validate
—api_schema_docsSkipped - GraphQL schema files exist but are example projects, not API docs for the framework itself
✓automated_doc_generationChangesets generates CHANGELOG.md files automatically. Documentation site exists (docs/) but manual content.
✓documentation_freshnessCONTRIBUTING.md was modified within last 180 days per git log check
✓readmeREADME.md exists with project description, usage documentation, and links to extended docs website
✗service_flow_documentedNo architecture diagrams (.mermaid, .puml) or service dependency documentation found in docs/
✗skillsNo skills directories (.factory/skills/, .skills/, .claude/skills/) found in repository
Dev Environment3/3 (100%)
✓database_schemaPrisma schema.prisma files exist throughout examples and test projects defining data models
✓devcontainer.devcontainer/devcontainer.json configured with Node.js image, Prisma, GraphQL extensions, and pnpm install
—devcontainer_runnableSkipped - devcontainer CLI not available to verify container can be built and run successfully
✓env_template.env.example files exist in examples (docs, cloudinary, assets-s3) documenting required environment variables
—local_services_setupSkipped - no docker-compose.yml for local dependencies. CI uses service containers for postgres in tests.
Debugging & Observability1/7 (14%)
✗alerting_configuredNo PagerDuty, OpsGenie, or custom alerting rules found in documentation or code
—circuit_breakersSkipped - library/framework without external service dependencies requiring circuit breaker patterns
—code_quality_metricsSkipped - no code scanning, coverage bots in PR comments, or SonarQube/quality gate enforcement found
✗deployment_observabilityNo monitoring dashboard links (Datadog, Grafana) or deployment notification integrations found in docs
✓distributed_tracingOpenTelemetry configured in logging-opentelemetry example for trace ID propagation
✗error_tracking_contextualizedNo Sentry, Bugsnag, or Rollbar configuration found for error tracking with source maps and context
—health_checksSkipped - library/framework, not a deployed service requiring /health endpoints or liveness probes
✗metrics_collectionNo metrics/telemetry libraries (Datadog, Prometheus, New Relic) found in dependencies
—profiling_instrumentationSkipped - library/framework where profiling is for users' applications, not the framework itself
✗runbooks_documentedNo runbooks/ directory or links to incident response procedures in README, AGENTS.md, or docs/
✗structured_loggingNo structured logging library (winston, pino, bunyan) in root package.json. Logging example uses OpenTelemetry.
Security4/6 (67%)
—automated_security_reviewSkipped - Socket Security check in CI is dependency scanning, not security review generation. No SAST reports found.
—branch_protectionRequires GitHub admin API access to verify branch protection rules (skipped).
✓codeownersCODEOWNERS file exists at .github/workflows/CODEOWNERS with team assignments (dcousens, mitchellhamilton)
—dast_scanningSkipped - library/framework not deployed as web service requiring OWASP ZAP or dynamic security testing
✓dependency_update_automationRenovate configured (renovate.json) with scheduled dependency updates and PR creation
✓gitignore_comprehensive.gitignore properly excludes .env, node_modules, dist, .DS_Store, and .keystone build artifacts
✗log_scrubbingNo log sanitization/scrubbing mechanisms configured or documented in logging utilities
—pii_handlingSkipped - framework/library doesn't process user data directly, PII handling is responsibility of applications built with it
—privacy_complianceSkipped - framework without end-user data collection, privacy compliance applies to applications using Keystone
—secret_scanningSkipped - gh api secret-scanning/alerts returns 404, feature not enabled. No gitleaks/trufflehog in CI.
✓secrets_management.env properly gitignored, .env.example templates provided, GitHub Actions uses secrets.* references in workflows
Task Discovery1/4 (25%)
✗backlog_healthMost open issues lack labels. Many old issues (>1 year) with no activity. Only ~6% have labels vs 70% threshold.
✗issue_labeling_systemPoor labeling: only 3 of 50 open issues have labels. No consistent priority/type/area label taxonomy.
✓issue_templates.github/ISSUE_TEMPLATE/ directory exists with bug report template (1.Bug_report.md)
✗pr_templatesNo .github/pull_request_template.md or GitLab merge request templates found
Product & Analytics0/2 (0%)
✗error_to_insight_pipelineNo Sentry-GitHub integration or error-to-issue automation found in workflows or configurations
✗product_analytics_instrumentationNo Mixpanel, Amplitude, PostHog, Heap, or GA4 instrumentation found in dependencies

start building

Ready to build the software of the future?

Start building

Arrow Right Icon