Factory.ai

Strong Documentation

Clawdbot reaches Level 3 with 71% Documentation pass rate. Currently becoming autonomous-capable with 22/54 criteria passing (41%). Key areas for improvement include the opportunities listed below.

Strengths

01
Formatter
All 4 applications have formatters: Main uses Oxfmt (.oxfmtrc.jsonc), iOS/macOS use SwiftFormat, Android uses Kotlin formatting (lintOptions in gradle).
02
Lint Config
All 4 applications have linting: Main/UI use Oxlint (.oxlintrc.json), iOS/macOS use SwiftLint (.swiftlint.yml), Android uses Kotlin lint (build.gradle.kts).
03
Strict Typing
Main has tsconfig strict:true, iOS has SWIFT_STRICT_CONCURRENCY:complete in project.yml, macOS uses strict Swift, Android uses Kotlin (strict by default).

Opportunities

01
Cyclomatic Complexity
Add complexity analysis to identify and refactor overly complex functions.
02
Feature Flag Infrastructure
Add feature flags to enable safer deployments and gradual rollouts.
03
Monorepo Tooling
Consider Turborepo or Nx for better caching, incremental builds, and dependency graph awareness.

All Criteria

Style & Validation4/11 (36%)
—code_modularizationSkipped: no module boundary enforcement tools (eslint-plugin-boundaries, dependency-cruiser, ArchUnit) detected.
✗cyclomatic_complexityNo cyclomatic complexity analysis tools (ESLint complexity, lizard, gocyclo, SonarQube) detected for any application.
✗dead_code_detectionNo dead code detection tools (knip, ts-prune, vulture, SonarQube) found in configurations or CI.
✗duplicate_code_detectionNo duplicate code detection tools (jscpd, PMD CPD, SonarQube) found in CI or configuration files.
✓formatterAll 4 applications have formatters: Main uses Oxfmt (.oxfmtrc.jsonc), iOS/macOS use SwiftFormat, Android uses Kotlin formatting (lintOptions in gradle).
✗large_file_detectionNo git-lfs, CI file size checks, or linter rules for max-lines detected in the repository.
✓lint_configAll 4 applications have linting: Main/UI use Oxlint (.oxlintrc.json), iOS/macOS use SwiftLint (.swiftlint.yml), Android uses Kotlin lint (build.gradle.kts).
—n_plus_one_detectionSkipped: no database/ORM usage evident (no Prisma, TypeORM, SQLAlchemy) requiring N+1 query detection.
✗naming_consistencyMain app has Oxlint rules. iOS/macOS have SwiftLint but naming rules unclear. Android has lint warningsAsErrors but no explicit naming rules visible.
✗pre_commit_hooksOnly main application has pre-commit hook (git-hooks/pre-commit runs format-staged.js). iOS/macOS/Android lack pre-commit hooks.
✓strict_typingMain has tsconfig strict:true, iOS has SWIFT_STRICT_CONCURRENCY:complete in project.yml, macOS uses strict Swift, Android uses Kotlin (strict by default).
✗tech_debt_trackingNo TODO/FIXME scanner, SonarQube, or other tech debt tracking tools found in CI or configuration.
✓type_checkAll 4 applications have type checking: Main has tsconfig.json with strict:true, iOS/macOS use Swift (inherently typed), Android uses Kotlin (typed by default).
Build System4/10 (40%)
✗agentic_developmentGit log shows no co-authorship with AI agents (e.g., factory-droid[bot]) in recent 20 commits.
—automated_pr_reviewSkipped: requires admin API access not available for OSS evaluation without authentication.
✓build_cmd_docREADME.md documents build commands: 'pnpm install', 'pnpm build', 'pnpm ui:build'.
—build_performance_trackingSkipped: no build metrics, caching evidence, or authenticated gh CLI access to analyze build timing.
—dead_feature_flag_detectionSkipped: prerequisite feature_flag_infrastructure not met (no feature flags in use).
—deployment_frequencySkipped: requires authenticated gh CLI to analyze release frequency and deployment patterns.
✓deps_pinnedpnpm-lock.yaml is committed to the repository, ensuring reproducible builds.
—fast_ci_feedbackSkipped: requires authenticated gh CLI to analyze PR status checks and measure CI duration.
✗feature_flag_infrastructureNo feature flag infrastructure (LaunchDarkly, Statsig, Unleash, GrowthBook, or custom) detected.
—heavy_dependency_detectionSkipped: bundle analysis tools (webpack-bundle-analyzer, size-limit) only applicable to main app; none detected. iOS/macOS/Android don't use bundlers.
✗monorepo_toolingMonorepo uses pnpm workspaces but lacks dedicated tooling (Turborepo, Nx, Lerna) for package boundaries.
—progressive_rolloutSkipped: no evidence of canary deployments or percentage-based rollouts for this CLI/desktop application.
✗release_automationCI builds code but no automated npm publish, CD pipeline, or release automation detected in workflows.
✗release_notes_automationCHANGELOG.md appears manually maintained with no semantic-release, changesets, or similar automation detected.
—rollback_automationSkipped: no evidence of automated rollback capability for this CLI/desktop application.
✓single_command_setupREADME documents single command setup: 'npm install -g clawdbot@latest && clawdbot onboard --install-daemon'.
✗unused_dependencies_detectionNo unused dependency detection tools (depcheck, deptry, cargo-udeps) found in package.json scripts or CI.
✓vcs_cli_toolsGitHub CLI (gh) is installed and working (version 2.52.0 verified).
—version_drift_detectionSkipped: no syncpack, manypkg, or similar version consistency tooling found for this monorepo.
Testing2/6 (33%)
—flaky_test_detectionSkipped: no authenticated gh CLI access to detect retries; no test retry configs (jest-retry, pytest-rerunfailures) found.
✗integration_tests_existMain app has integration tests (*.e2e.test.ts, vitest.e2e.config.ts). iOS/macOS/Android lack visible integration test suites.
✗test_coverage_thresholdsMain app has vitest coverage thresholds (70% lines/functions, 55% branches, 70% statements). iOS/macOS/Android lack coverage thresholds.
✗test_isolationMain app has parallel test execution (vitest maxWorkers configured). iOS/macOS/Android lack visible parallel test configuration.
✓test_naming_conventionsAll apps enforce naming: Main uses vitest include patterns (*.test.ts), iOS/macOS use Swift test conventions (*Tests.swift), Android uses Kotlin test patterns.
✗test_performance_trackingMain app CI shows test timing in output. iOS/macOS/Android lack visible test performance tracking configuration.
✓unit_tests_existAll apps have tests: Main has *.test.ts files (vitest), iOS has Tests/ directory, macOS has ClawdbotIPCTests, Android has app/src/test/.
—unit_tests_runnableSkipped: requires fully configured dev environment not available for OSS evaluation. Main app has 'pnpm test' script.
Documentation5/7 (71%)
✓agents_mdAGENTS.md exists at repo root with comprehensive documentation (>100 characters) covering project structure, commands, testing, and conventions.
✗agents_md_validationNo CI job, pre-commit hook, or automation that validates AGENTS.md commands or keeps it consistent with code.
—api_schema_docsSkipped: no OpenAPI/Swagger specs or GraphQL schemas found. Main app provides protocol schema but not API docs for HTTP endpoints.
✗automated_doc_generationNo API doc generators (Swagger/OpenAPI), JSDoc, Sphinx, or automated doc creation tools detected.
✓documentation_freshnessREADME.md, AGENTS.md, and CONTRIBUTING.md all modified within last 180 days per git log.
✓readmeREADME.md exists at repo root with extensive setup, usage, and documentation (505 lines).
✓service_flow_documentedREADME contains architecture diagrams showing system flow (Gateway control plane, channels, nodes, CLI interactions).
✓skillsSkills directory exists at ./skills/ with 52+ valid skills in {skill-name}/SKILL.md format with YAML frontmatter (name, description).
Dev Environment2/3 (67%)
—database_schemaSkipped: no database schema files (Prisma, TypeORM, SQLAlchemy, SQL schemas) detected in the repository.
✗devcontainerNo .devcontainer/devcontainer.json found in the repository.
—devcontainer_runnableSkipped: prerequisite devcontainer not met (no devcontainer configuration exists).
✓env_template.env.example exists at repo root documenting required Twilio environment variables.
✓local_services_setupdocker-compose.yml exists documenting gateway and CLI services with volume mounts and environment variables.
Debugging & Observability1/7 (14%)
✗alerting_configuredNo alerting infrastructure (PagerDuty, OpsGenie) or alerting rules detected in configuration or documentation.
—circuit_breakersSkipped: no circuit breaker libraries (opossum, resilience4j, polly) found; not applicable to all apps.
—code_quality_metricsSkipped: requires admin API access for code scanning analyses; main app has coverage tracking but other metrics unclear.
✗deployment_observabilityNo deployment observability (monitoring dashboard links, deploy notification integrations) found in docs or code comments.
✗distributed_tracingNo trace ID or request ID propagation (OpenTelemetry, X-Request-ID headers) detected in source code.
✗error_tracking_contextualizedNo error tracking services (Sentry, Bugsnag, Rollbar) found in package.json dependencies.
—health_checksSkipped: health checks only applicable to deployed services. Main gateway may have health endpoints but iOS/macOS/Android are client apps.
✗metrics_collectionNo metrics/telemetry instrumentation (Datadog, Prometheus, New Relic, CloudWatch, Axiom) found in dependencies.
—profiling_instrumentationSkipped: no profiling infrastructure (APM tools, continuous profiling) detected for any application.
✓runbooks_documentedDocumentation includes extensive runbooks in docs/ directory (gateway, troubleshooting, health checks, background process management).
✗structured_loggingMain app has custom logger module (src/logger.ts, src/logging/). iOS/macOS/Android use native logging but not verified as structured.
Security2/5 (40%)
—automated_security_reviewSkipped: requires admin API access not available for OSS evaluation to check code-scanning alerts; detect-secrets runs in CI.
—branch_protectionSkipped: requires admin API access not available for OSS evaluation to check rulesets or legacy branch protection.
✗codeownersNo CODEOWNERS file found in root or .github/ directory.
—dast_scanningSkipped: DAST tools (OWASP ZAP, Burp Suite) not applicable to CLI/desktop applications; main gateway could benefit but none detected.
✗dependency_update_automationNo Dependabot or Renovate configuration detected in .github/ or repository root.
✓gitignore_comprehensive.gitignore properly excludes .env, node_modules, build artifacts (dist), IDE configs (.vscode), and OS files (.DS_Store).
✗log_scrubbingMain app has logging redaction (src/logging/redact.ts). iOS/macOS/Android lack visible log scrubbing mechanisms.
—pii_handlingSkipped: no PII detection/handling tooling (Microsoft Presidio, AWS Macie, data masking libraries) detected.
—privacy_complianceSkipped: no evidence of privacy compliance infrastructure (consent management, GDPR/CCPA handling) for this CLI/desktop application.
—secret_scanningSkipped: requires admin API access not available for OSS evaluation; CI includes detect-secrets baseline check (.secrets.baseline exists).
✓secrets_managementSecrets management via .env.example template, .env in gitignore, and GitHub Actions secrets (secrets.* references in CI).
Task Discovery2/3 (67%)
—backlog_healthSkipped: requires authenticated gh CLI to analyze open issues for titles, labels, and age statistics.
✓issue_labeling_systemIssue templates include consistent labels (bug, feature request as shown in templates).
✓issue_templates.github/ISSUE_TEMPLATE/ exists with bug_report.md, feature_request.md, and config.yml templates.
✗pr_templatesNo .github/pull_request_template.md or PULL_REQUEST_TEMPLATE.md found in the repository.
Product & Analytics0/2 (0%)
✗error_to_insight_pipelineNo error-to-issue automation (Sentry-GitHub integration, error tracking with issue creation) detected in configuration.
✗product_analytics_instrumentationNo product analytics tools (Mixpanel, Amplitude, PostHog, Heap, GA4) found in dependencies.

start building

Ready to build the software of the future?

Start building

Arrow Right Icon