When we started building Welcomeloom, the workflow design seemed obvious: new hire record triggers four tasks, tasks run in parallel, system tracks completion. The concept fits on a napkin. The implementation surface is considerably larger.
This is a writeup of the engineering decisions behind Welcomeloom's workflow engine -- what we built, what we changed, and where the hard problems actually lived.
Why parallel tracks are harder than sequential ones
A sequential workflow is straightforward to reason about: step A completes, step B starts. State is simple -- you're always in exactly one step.
Onboarding automation is fundamentally parallel. The I-9, equipment request, IT provisioning, and calendar scheduling all need to start at the same time and complete before Day 1. They're owned by different systems, respond to different APIs, and fail in different ways. The workflow engine needs to track four concurrent state machines, each with its own success conditions and failure modes, and surface a coherent status view for HR.
Our first version treated this as four independent workflows started simultaneously. That worked at the happy-path level. It broke on failure recovery: if IT provisioning failed and needed to retry, the other three tracks didn't know about it. HR had no unified view of "onboarding #37 is blocked on one track." They had four separate status records and had to reconcile them manually.
Moving to a parent-child workflow model
The design change that fixed this was introducing a parent onboarding record with child track records. The parent is the single source of truth for the overall onboarding state. Each track is a child workflow that reports status back to the parent.
The parent record:
- Holds the hire's identity, start date, and workflow configuration.
- Tracks the aggregate status across all four tracks.
- Owns the Day-1 readiness check -- a scheduled job that runs at a configurable checkpoint before the hire's start date and flags any track that isn't complete.
- Emits the summary notification to HR when all tracks close.
Each child track:
- Holds the state machine for that specific integration (I-9, equipment, IT, calendar).
- Handles its own retry logic for API failures.
- Reports terminal states (complete, failed, awaiting-input) back to the parent.
This model meant that the HR dashboard could show a single card per onboarding -- "3/4 tracks complete, 1 awaiting IT confirmation" -- instead of four separate records. Exception visibility improved substantially.
The failure modes that required the most design work
Each integration track has a different failure profile. Designing the retry and escalation logic for each was the most time-consuming part of the engine build.
I-9 track: Failures here are mostly user-side -- the hire doesn't complete Section 1, the hire provides invalid documents, the employer can't verify in person. The failure mode is delay, not system error. Our approach: automated reminders at 48 and 72 hours, with escalation to HR coordinator at 72 hours rather than retry. An automated system can't resolve an identity verification dispute.
Equipment track: Failures are vendor API errors and out-of-stock conditions. Standard retry with exponential backoff handles transient API failures. Out-of-stock is a business exception that requires HR notification, not retry. We had to model these as distinct failure reasons, not a single "failed" state.
IT provisioning track: Failures split between API errors (service down) and configuration errors (account type not defined for the hire's role). Configuration errors surface immediately and require a human to resolve the account template -- they can't be retried. API errors retry. Distinguishing between them required parsing error responses from multiple IT systems, which was fiddly but necessary.
Calendar track: Calendar API failures are almost always transient. The retry pattern is simple. The harder problem was handling calendar permissions: if the hiring manager's calendar doesn't allow external write access, the invite fails silently in some systems. We moved to a pattern of sending the invite from a Welcomeloom service account and including the manager as an optional attendee, which worked around the permissions issue without requiring IT involvement.
The Day-1 readiness check
The pre-Day-1 check is a scheduled job that runs 48 hours before each hire's start date. It reads the parent record state, identifies any tracks not in terminal-success state, and either auto-retries (for tracks in retry-eligible states) or fires an HR alert (for tracks in human-required states).
This check is the backstop. Even if a retry loop failed silently somewhere in the workflow, the Day-1 check catches it with enough lead time for a coordinator to manually intervene. Building the system to be catchable -- to surface its own failures explicitly -- was as important as building it to succeed.
What we'd design differently
Two decisions we'd revisit:
First, we built the retry logic into each track's workflow code rather than centralizing it. That made the code harder to reason about and led to inconsistencies between tracks. A centralized retry coordinator with per-track configuration would have been cleaner.
Second, our initial event schema was too opaque -- internal state machine transitions that were meaningful to the engine but not surfaceable to HR without translation. We rebuilt the event log to emit human-readable status messages alongside internal states, which made debugging and the HR dashboard significantly easier. We should have designed for the human-readable log from the start.