All Reports
Open Xtract
Python
mellow-artificial-intelligence/open-xtract
Pass Rate
40%
Open source extraction tool
L1100%
L2100%
L30%
L40%
L50%
Strong Security
Open Xtract reaches Level 3 with 60% Security pass rate. Currently becoming autonomous-capable with 23/57 criteria passing (40%). Key areas for improvement include the opportunities listed below.
Strengths
01
Formatter
Ruff format is configured in pyproject.toml and used in CI (ruff format --check)
02
Lint Config
Ruff is configured in pyproject.toml with linting rules (E, F, I, UP, B, SIM)
03
Build Cmd Doc
Build commands documented - README has 'uv add open-xtract', CONTRIBUTING.md has 'uv sync --dev'
Opportunities
01
Cyclomatic Complexity
Add complexity analysis to identify and refactor overly complex functions.
02
Feature Flag Infrastructure
Add feature flags to enable safer deployments and gradual rollouts.
03
Integration Tests Exist
Add integration tests to verify component interactions and catch issues unit tests miss.
All Criteria
Style & Validation2/11 (18%)
—code_modularizationSkipped: small library with only 454 lines of code where module boundaries are not meaningful
✗cyclomatic_complexityNo complexity analysis tools configured (no lizard, radon, or ruff complexity rules)
✗dead_code_detectionNo dead code detection tools found (no vulture, dead, or similar tools)
✗duplicate_code_detectionNo duplicate code detection tools configured (no jscpd, PMD CPD, or similar)
✓formatterRuff format is configured in pyproject.toml and used in CI (ruff format --check)
✗large_file_detectionNo file size detection tools (no git hooks, CI jobs, or linter rules checking file size)
✓lint_configRuff is configured in pyproject.toml with linting rules (E, F, I, UP, B, SIM)
—n_plus_one_detectionSkipped: no database or ORM usage in this library
✗naming_consistencyNo naming convention rules found in ruff config or documented in AGENTS.md/CONTRIBUTING.md
✗pre_commit_hooksNo .pre-commit-config.yaml or husky/lint-staged configuration found
✗strict_typingNo mypy strict mode configuration found (no mypy.ini or [tool.mypy] in pyproject.toml)
✗tech_debt_trackingNo tech debt tracking tools (no TODO scanner in CI, no SonarQube, no tech debt markers)
✗type_checkNo mypy or type checker configuration found in pyproject.toml or separate config files
Build System7/12 (58%)
✗agentic_developmentNo agent co-authorship found in git log (no droid or agent identifiers in author/co-author fields)
✗automated_pr_reviewNo automated PR review generation found (gh pr list shows empty reviews and comments arrays)
✓build_cmd_docBuild commands documented - README has 'uv add open-xtract', CONTRIBUTING.md has 'uv sync --dev'
—build_performance_trackingSkipped: no build caching or performance tracking configured
—dead_feature_flag_detectionSkipped: prerequisite fails (no feature flag infrastructure)
✗deployment_frequencyReleases are monthly/bi-weekly (v0.2.0 Jan, v0.1.4 Dec, v0.1.3 Nov), not multiple per week
✓deps_pinnedDependencies pinned - uv.lock lockfile is committed
✓fast_ci_feedbackCI feedback is fast - average CI duration ~9 seconds (well under 10 minute threshold)
✗feature_flag_infrastructureNo feature flag infrastructure (no LaunchDarkly, Statsig, or custom feature flags)
—heavy_dependency_detectionSkipped: Python library, not a bundled JavaScript application
—monorepo_toolingSkipped: single application repository, not a monorepo
—progressive_rolloutSkipped: not an infrastructure repository, it's a library
✓release_automationRelease automation via GitHub release workflow that publishes to PyPI on version bumps
✓release_notes_automationAutomated release notes via GitHub release workflow (generate_release_notes: true in release.yml)
—rollback_automationSkipped: not an infrastructure repository, it's a library
✓single_command_setupSingle command setup documented in CONTRIBUTING.md: 'uv sync --dev'
✗unused_dependencies_detectionNo unused dependency detection tools found (no depcheck, deptry, or pip-extra-reqs)
✓vcs_cli_toolsGitHub CLI (gh) is installed and authenticated (verified with gh auth status)
—version_drift_detectionSkipped: single application repository, not a monorepo
Testing3/7 (43%)
—flaky_test_detectionSkipped: gh CLI available but no retry config or flaky test tracking tools found
✗integration_tests_existNo integration tests directory or files found (no tests/integration/ or e2e tests)
✗test_coverage_thresholdsNo coverage threshold enforcement (no coverageThreshold in pytest config or CI gates)
✗test_isolationNo parallel test execution configured (no pytest-xdist or parallelization)
✓test_naming_conventionsPytest naming conventions configured in pyproject.toml with testpaths=['tests']
✗test_performance_trackingNo test timing output or test analytics platforms configured
✓unit_tests_existUnit tests exist in tests/test_extract.py and tests/test_temporal.py
✓unit_tests_runnableTests are runnable - pytest --collect-only successfully collected 43 test items
Documentation2/6 (33%)
✗agents_mdNo AGENTS.md file exists at repository root
—agents_md_validationSkipped: prerequisite fails (no AGENTS.md file)
—api_schema_docsSkipped: this is a library, not an API service with endpoints
✗automated_doc_generationNo automated documentation generation tools or workflows found
✓documentation_freshnessCONTRIBUTING.md was modified in the last 180 days (git log shows recent updates)
✓readmeREADME.md exists with comprehensive setup, usage instructions, and examples
✗service_flow_documentedNo architecture diagrams or service flow documentation found
✗skillsNo skills directory found (.factory/skills/, .skills/, .claude/skills/)
Dev Environment1/3 (33%)
—database_schemaSkipped: no database usage (uses external Temporal for durable execution)
✗devcontainerNo .devcontainer directory or devcontainer.json configuration found
—devcontainer_runnableSkipped: no devcontainer configuration exists
✗env_templateNo .env.example file and no environment variables documented in README or AGENTS.md
✓local_services_setupdocker-compose.temporal.yml provides local Temporal, PostgreSQL, and UI services
Debugging & Observability3/7 (43%)
✗alerting_configuredNo PagerDuty, OpsGenie, or alerting rules found
—circuit_breakersSkipped: unclear if needed for this library's LLM API calls
—code_quality_metricsSkipped: requires admin API access not available for OSS evaluation (403 on code scanning)
✗deployment_observabilityNo monitoring dashboard references found in docs (no Datadog, Grafana, New Relic links)
✓distributed_tracingLogfire provides distributed tracing via instrument_pydantic_ai() and instrument_httpx()
✗error_tracking_contextualizedNo Sentry, Bugsnag, or Rollbar configuration found
—health_checksSkipped: library, not a deployed service with endpoints
✓metrics_collectionLogfire provides instrumentation for metrics collection via logfire.configure()
—profiling_instrumentationSkipped: no profiling tools configured
✗runbooks_documentedNo runbooks or references to runbooks found in documentation
✓structured_loggingLogfire logging library is used (imported in __init__.py, configured via configure_logging())
Security3/5 (60%)
—automated_security_reviewSkipped: requires admin API access not available for OSS evaluation (403 on code scanning API)
—branch_protectionSkipped: requires admin API access not available for OSS evaluation (empty rulesets, 404 on branch protection)
✓codeowners.github/CODEOWNERS file exists with valid team assignment (* @colesmcintosh)
—dast_scanningSkipped: library, not a deployed web service
✗dependency_update_automationNo Dependabot or Renovate configuration found
✓gitignore_comprehensive.gitignore properly excludes .env, __pycache__, .venv, build, dist, and Python artifacts
✗log_scrubbingNo log sanitization mechanisms found (no redaction, masking, or scrubbing patterns)
—pii_handlingSkipped: library that doesn't directly process user data
—privacy_complianceSkipped: library, not an application that collects end-user data
—secret_scanningSkipped: requires admin API access not available for OSS evaluation (404 on secret scanning API)
✓secrets_management.env files properly gitignored, GitHub Actions secrets used, no hardcoded secrets found
Task Discovery2/4 (50%)
✗backlog_healthOnly 1 open issue with descriptive title but no labels (0% meet both requirements)
✗issue_labeling_systemNo consistent labeling system (1 open issue has no labels, no label schema documented)
✓issue_templates.github/ISSUE_TEMPLATE/ exists with bug_report.yml and feature_request.yml templates
✓pr_templates.github/PULL_REQUEST_TEMPLATE.md exists with sections for summary, changes, testing, and checklist
Product & Analytics0/2 (0%)
✗error_to_insight_pipelineNo error-to-issue automation (no Sentry-GitHub integration or error tracking with issue creation)
✗product_analytics_instrumentationNo product analytics platforms found (no Mixpanel, Amplitude, PostHog, etc.)