Add template output validation to catch malformed generated documents #41

Closed
opened 2026-04-08 11:36:13 -04:00 by pook · 26 comments
Owner

Add a post-generation validation step that checks AI-generated compliance documents for completeness and correctness before returning to the user.

  • Validate generated document contains all required sections for its type (e.g., privacy policy must have data collection, user rights, contact info sections)
  • Check for placeholder text or incomplete generation markers (e.g., '[COMPANY NAME]', '...', 'TODO')
  • Validate no HTML injection or malformed markup in output
  • If validation fails, retry generation once before returning error
  • Log validation failures with document type and failure reason for monitoring

Acceptance criteria:

  • Each document type has a defined list of required sections
  • Placeholder detection catches common AI generation artifacts
  • Failed validation triggers one retry automatically
  • Validation failures are logged with structured metadata
  • Add tests with sample good and bad generated documents

Generated by CEO Planner (priority: 3)

Add a post-generation validation step that checks AI-generated compliance documents for completeness and correctness before returning to the user. - Validate generated document contains all required sections for its type (e.g., privacy policy must have data collection, user rights, contact info sections) - Check for placeholder text or incomplete generation markers (e.g., '[COMPANY NAME]', '...', 'TODO') - Validate no HTML injection or malformed markup in output - If validation fails, retry generation once before returning error - Log validation failures with document type and failure reason for monitoring Acceptance criteria: - Each document type has a defined list of required sections - Placeholder detection catches common AI generation artifacts - Failed validation triggers one retry automatically - Validation failures are logged with structured metadata - Add tests with sample good and bad generated documents --- *Generated by CEO Planner (priority: 3)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 6h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 6h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 6h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 6h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 6h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 6h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 7h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 7h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 7h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 7h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 7h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 7h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 7h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 7h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 7h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 7h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 8h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 8h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 8h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 8h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 8h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 8h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 8h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 8h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 8h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 8h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 8h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 8h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 9h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 9h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 9h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 9h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 9h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 9h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 9h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 9h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 9h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 9h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 10h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 10h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 19h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 19h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 19h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 19h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 19h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 19h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 19h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 19h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

⚠️ Stale Task Alert — This task has been open for 19h with no associated PR.

Possible causes:

  • Worker failed to execute (check logs)
  • Claude CLI produced no changes
  • Task may be too complex for single-session execution

The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped.

— CEO Planner (automated)

⚠️ **Stale Task Alert** — This task has been open for 19h with no associated PR. Possible causes: - Worker failed to execute (check logs) - Claude CLI produced no changes - Task may be too complex for single-session execution The CEO planner will re-evaluate this task. If it remains stale for 24h+, it will be closed and re-scoped. *— CEO Planner (automated)*
Author
Owner

Bulk-closed 2026-04-10 during pipeline triage.

Context: CEO agent had created 100 open agent-task issues against compliancebot, largely duplicates of each other and of the 50 currently-open PRs. Root cause traced to a git-push race in agent-worker executor (dispatch jobs collided on branch agent/dispatch/* because jobId prefix truncated to literal "dispatch"). Fix deployed: runId is now threaded from Paperclip shim through /dispatch → TaskJob → executor, and branches are keyed on a 12-char unique run key.

What to do next:

  1. Triage the 50 open PRs at https://192.168.183.110:3000/pook/compliancebot/pulls — many are ready to merge
  2. CEO should halt new task creation until open PRs drop below 10
  3. Surviving kept issues: #313, #314, #315, #341, #342, #350, #351, #352 (PR review/merge tasks)

This issue was superseded, not abandoned. Reopen if still relevant after PR triage.

Bulk-closed 2026-04-10 during pipeline triage. **Context:** CEO agent had created 100 open agent-task issues against compliancebot, largely duplicates of each other and of the 50 currently-open PRs. Root cause traced to a git-push race in agent-worker executor (dispatch jobs collided on branch `agent/dispatch/*` because jobId prefix truncated to literal "dispatch"). Fix deployed: runId is now threaded from Paperclip shim through /dispatch → TaskJob → executor, and branches are keyed on a 12-char unique run key. **What to do next:** 1. Triage the 50 open PRs at https://192.168.183.110:3000/pook/compliancebot/pulls — many are ready to merge 2. CEO should halt new task creation until open PRs drop below 10 3. Surviving kept issues: #313, #314, #315, #341, #342, #350, #351, #352 (PR review/merge tasks) This issue was superseded, not abandoned. Reopen if still relevant after PR triage.
pook closed this issue 2026-04-10 14:48:30 -04:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
pook/compliancebot#41
No description provided.