How one score event triggers 20+ reward evaluations across sync, async, and scheduled paths
1. The Problem
A player steps into one of our Core Soccer Tech pods, plays a 90-second game, and scores 847 points. The kiosk needs to respond instantly: show the score, trigger a celebration animation if they hit a personal high, and display any XP (experience points) earned.
But behind that single score, the system has to answer over 20 questions. Did they beat their high score? Did they unlock a new game mode? Did that score push them past a reward threshold? How many of today's seven daily missions did that game advance? What about the eight weekly missions running in parallel? And the eight seasonal missions that span the entire quarter?
All of this from one POST request.
The naive approach is to evaluate everything synchronously inside the score submission endpoint. The kiosk waits, the database holds a long transaction, and every player in the facility feels the lag. At 75+ touches per 90-second game and multiple pods running simultaneously, that is not a theoretical concern. It is a production one.
We needed to split the work. Some rewards have to be synchronous because the kiosk needs the data immediately for animations. Some can be deferred by a few seconds. Some only make sense at the end of a week or quarter.
The result is a three-path reward processing engine: synchronous rewards evaluated inside the score transaction, asynchronous processing via a BullMQ worker for missions and achievements, and cron-driven distribution for leaderboard rewards.
2. Architecture Overview
The backend is a NestJS 11 application on Node.js 24. It talks to PostgreSQL through the Knex query builder (via nestjs-knex); there is no ORM (Object-Relational Mapper), so every query in this article is either a Knex builder call or raw SQL. Schema changes, unique constraints, and database triggers all live in Knex migrations. Background jobs use BullMQ 5 on Redis, and scheduled jobs use @nestjs/schedule.
The reward engine sits between score ingestion and four output tables. Every reward, regardless of its source, writes to one or more of these: player_transaction (the universal ledger of XP and CC, the platform's credit currency), player_metadata (running totals maintained by database triggers), player_achievement (one-time milestones), and player_notification (kiosk alerts).
Three ingestion paths, including classic and weekly pattern games, OnePod batch sessions, and battle rooms, all converge into a single addScore() method. From there, processing splits into three paths based on urgency:
Sync Path (green): Runs inside the database transaction. Returns rewards in the HTTP response so the kiosk can show celebration animations immediately. Handles single game rewards, high score detection, and game mode unlocks.
Async Path (purple): Score events are enqueued to a BullMQ queue after the transaction commits. A background worker evaluates achievements, badges, daily missions, weekly missions, seasonal missions, and level-ups with a concurrency cap of 5 parallel jobs.
Scheduled Path (red): Cron jobs run on fixed schedules in the America/Denver timezone. Every Monday: the weekly pattern rotates at 12:00 AM, weekly leaderboard rewards are distributed at 12:05 AM, and the "top 10 in your age group" seasonal mission is updated at 12:10 AM. Battle season rewards run at 12:05 AM on the first day of each quarter. These jobs rank players and distribute CC, XP, and badges based on leaderboard position.
3. The Unified Entry Point
All three ingestion paths converge into addScore(). This is a deliberate architectural choice: a single method handles score creation and sync reward evaluation inside one database transaction, then dispatches the async event after that transaction commits.
Why a Single Entry Point?
Classic games, OnePod batch sessions (2–4 players ranked together), and battle rooms (deferred scoring until all players finish) have very different ingestion patterns. But once a score is created, the reward logic is identical. By funneling everything through addScore(), we guarantee that every score, regardless of how it was submitted, gets the same reward treatment.
Transaction Boundary
The transaction boundary is the most important design decision in this system. Inside the transaction, we run eight operations in a fixed order:
- Fetch the previous high score for this player and game mode (weekly pattern scores excluded).
- Increment player.game_count and read the new value back.
- Insert the score into soccer_game_score.
- Insert the game-specific record (one_pod_score or weekly_pattern_score) if the game style needs one.
- Insert a NewGameUnlock achievement if the new game count crosses an unlock threshold.
- Evaluate single game rewards (section 4) and, if any qualify, record the XP and write one player_transaction row.
- Insert a PersonalHighScore achievement if the score beats the previous high.
- Log today's play date into player_daily_play_log.
The transaction callback returns, Knex issues COMMIT, and only then is the ScoreEventPayload enqueued to BullMQ. Enqueueing is deliberately outside the transaction. If the enqueue fails, the score is saved but async rewards are lost, and because the enqueue call is not caught, the kiosk receives a 500 for a score that was in fact persisted. We accepted this tradeoff to keep the transaction short and avoid coupling score persistence to Redis availability.
const result = await this.knex.transaction(async trx => {
const prevHighScore = await getPrevHighScore(trx, playerId, req.game_mode_id)
//step 1
const [{ game_count }] = await trx('player')
.where('id', playerId).increment('game_count', 1).returning('game_count')
//step 2
const [createdScore] = await trx('soccer_game_score')
.insert({ player_id: playerId, game_mode_id: req.game_mode_id, score: req.score, game_style: req.game_style })
.returning('*')
//step 3
// steps 4 to 8 run here, all on `trx`
return { createdScore, prevHighScore, newGameCount: game_count }
})
// COMMIT has happened. Now, and only now, hand the event to the worker.
await this.scoreEventsQueue.addScoreEventJob({
player_id: playerId,
score_id: result.createdScore.id,
score: result.createdScore.score,
game_mode_id: result.createdScore.game_mode_id,
game_style: result.createdScore.game_style,
prev_high_score: result.prevHighScore,
new_game_count: result.newGameCount,
season: result.createdScore.season,
})Battle rooms and OnePod sessions insert several scores in one outer transaction, so they pass their own trx into addScore(). Battle completion also passes skipEnqueue = true, collects one payload per player, and enqueues them all after the outer commit.
The ScoreEventPayload
The payload sent to the async worker carries everything needed to evaluate all downstream rewards without additional database lookups at the entry point:
{ player_id, score_id, score, game_mode_id, game_style, prev_high_score, new_game_count, season }prev_high_score is included because the worker needs it for score level achievement checks. Without it, the worker would need to query historical scores, adding latency and database load to every async job.
4. The Sync Path: Immediate Rewards
These rewards are evaluated inside the addScore() transaction and returned in the HTTP response. The kiosk uses them to display immediate feedback: XP earned animations, high score celebrations, and game unlock screens.
Single Game Rewards
Four reward types are evaluated per score, and all can stack:
Reward | Condition | Notes |
play_any_game | Always qualifies | Base XP for playing |
score_x | score >= floor(third_best_score * threshold_pct / 100) | Needs at least 3 scores in the mode |
score_y | Same formula, higher threshold_pct | Harder tier |
new_high_score | score > prev_high_score AND games_in_mode >= min_high_score_games | min_high_score_games comes from global config, default 10 |
Each row is configured in single_game_reward_config (threshold_pct, xp_reward, animation, and a game_style exclusion list). Weekly pattern games skip this block entirely.
The best-score formula is worth noting: instead of comparing against the player's top score, we compare against their third-best score in that game mode. This prevents a single lucky outlier from permanently raising the bar; a player has to post three strong games before the threshold moves.
// Top 3 in this mode, read after the new score was inserted (so it is included)
const topScores = await trx('soccer_game_score').select('score')
.where({ player_id, game_mode_id })
.whereNot('game_style', 'weekly_pattern')
.orderBy('score', 'desc').limit(3)
const thirdBestScore = topScores.length >= 3 ? topScores[2].score : null
// score_x / score_y
if (thirdBestScore !== null && config.threshold_pct !== null) {
const threshold = Math.floor((thirdBestScore * config.threshold_pct) / 100)
qualifies = score >= threshold
}
// new_high_score
qualifies = gamesInMode >= minHighScoreGames && score > prevHighScoreIf a player has fewer than 3 scores in a mode, thirdBestScore is null and score_x and score_y are not awarded at all. Only play_any_game fires on those first two games. There is one side effect of including the new score in the top-3 query: on exactly the third game in a mode, the new score can be the third best itself, so score_x and score_y automatically qualify for that game.
High Score Detection and Game Unlocks
If the new score beats the player's previous high score (and the game style is not weekly_pattern), a PersonalHighScore achievement is inserted. This check has no game-count gate, so a player on their third game gets the celebration animation but not the new_high_score XP. Similarly, when game_count crosses an unlock threshold, a NewGameUnlock achievement is created with metadata indicating which game mode (CoreOG, Mirror, Xpert, Clutch, or Mirror_X) was unlocked.
5. The Async Path: BullMQ Worker
After the transaction commits, a ScoreEventPayload is enqueued to a Redis-backed BullMQ queue named scoreEvents. A worker picks it up and processes the steps below sequentially.
Worker Configuration
The worker runs in the same Node.js process as the HTTP server with a concurrency cap of 5 parallel jobs. Successful jobs are removed immediately; the last 100 failed jobs are retained for debugging. This is a pragmatic choice: a separate worker process would add deployment complexity for a system that handles scoring bursts measured in dozens per minute, not thousands.
this.queue = new Queue('scoreEvents', { connection: redisConfig })
this.worker = new Worker<ScoreEventPayload>(
'scoreEvents',
async job => this.withDeadlockRetry(() => this.processScoreEvent(job.data)),
{ connection: redisConfig, concurrency: 5 },
)
// called by addScore() after COMMIT
await this.queue.add('processScore', payload, {
removeOnComplete: true,
removeOnFail: 100,
})That is the whole configuration. We do not set attempts or backoff, so BullMQ's defaults apply: a job that throws is moved straight to the failed set with no automatic retry. The only retry in this path is application-level. withDeadlockRetry() reruns the job up to three times when PostgreSQL reports a deadlock (error code 40P01), waiting 50 * attempt + random(0..50) milliseconds between tries. Everything else fails once.
Stalled jobs follow BullMQ defaults too: a job lock lasts 30 seconds and is renewed every 15; if the process dies or blocks the event loop long enough for the lock to expire, the stalled checker (every 30 seconds) puts the job back in the wait list once (maxStalledCount: 1). A second stall causes the job to fail. After a final failure the error is logged with the player_id and score_id, the job sits in the failed set until it is pushed out by the 100-job cap, and nothing replaces it. There is no dead-letter queue and no alerting on the failed set today.
Achievement Evaluation (Steps 1–2)
Step 1 checks whether the new score crossed a score-level tier. These configurable thresholds are defined per game mode and can award XP, credits, and badges. Step 2 checks career total points milestones, using a database trigger-maintained counter on player_metadata. Both use an achievementExists(score_id, type) idempotency check before inserting, preventing duplicates if the same event is ever processed twice. A points-badge evaluation runs between step 2 and the missions.
Mission Processing (Steps 3–5)
This is where the bulk of the work happens. Steps 3 through 5 evaluate daily, weekly, and seasonal missions respectively. They run sequentially within a single job, not in parallel. This was a deliberate choice: sequential execution means a database error in daily missions prevents weekly and seasonal from running for that event, but it also means we never have two mission services competing for database connections from the same job. A level-up check runs last, after the seasonal missions.
6. The Mission System: Three Time Horizons
The mission system operates across three time horizons, each with fundamentally different state management patterns. Missions unlock on a player's fifth game overall: games 1 to 4 are skipped, game 5 creates the mission rows, and processing starts from game 6.
Daily Missions
Seven mission types reset every day. On a player's first score each day, getOrCreateDailyMissions() lazily initializes all seven missions with fresh progress counters. Goals are personalized; the TotalPoints mission, for example, sets its goal to the player's third-best Core OG score multiplied by a configurable factor (minimum 1), ensuring the target scales with skill level.
The ScoreYStreak mission is the most interesting pattern: it requires the player to score above a threshold N times in a row. One miss resets progress to zero, so each consecutive successful score increases the cost of breaking the streak. The backend tracks this with a simple conditional: if score >= threshold, increment progress; else, reset to zero.
Weekly Missions
Eight mission types reset every Monday. The week anchor is computed in the database as date_trunc('week', (current_timestamp AT TIME ZONE 'America/Denver')::date). The most complex is TopNScore, where N is locked when the week's missions are created, based on the player's total game count: experienced players must score in their top 5, newer players in their top 10. This graduated difficulty prevents both sandbagging and frustration.
Weekly missions also track consecutive play days using an append-only player_daily_play_log table. Each day a player plays, a row is inserted with INSERT ... ON CONFLICT (player_id, play_date) DO NOTHING (Knex .onConflict([...]).ignore()) for concurrent access safety. The streak count is computed from this log at evaluation time.
Seasonal Missions
Eight missions spanning an entire quarter, with 16 to 200 threshold milestones each. Unlike daily and weekly missions that track per-score progress, seasonal missions use aggregate counters stored in player_season_data. Each score event increments these counters with a single UPDATE, and the mission service compares current values against threshold tables.
The idempotency mechanism here is elegant: each player's mission row stores a last_completed_threshold_id. When processing, the service iterates through thresholds in sort order, skipping anything at or below this ID, and stopping at the first uncrossed threshold. The ID update and reward insert happen in the same database transaction, so if the transaction rolls back, neither persists.
for (const t of thresholds) { // sort_order ascending
if (mission.last_completed_threshold_id !== null && t.id <= mission.last_completed_threshold_id) continue
if (t.threshold_value > mission.current_value) break
newThresholds.push(t)
}
if (newThresholds.length === 0) continue
await trx('player_seasonal_mission').where({ id: mission.id })
.update({ last_completed_threshold_id: newThresholds.at(-1).id })
await this.levelUpService.recordXp(trx, playerId, season, totalXp, XpSource.SeasonalMission)
await trx('player_transaction').insert({ player_id: playerId, transaction_type: 'add',
reward_name: 'seasonal_mission', xp_earned: totalXp, free_live_credits: totalCc })Rotating Game Modes
Both daily and weekly missions include a "play the rotating game mode" mission type. Daily rotation is deterministic: day of year (Denver local date) modulo 4 maps to Mirror, Xpert, Clutch, or Mirror_X. Weekly rotation is derived from the week's start date and stored in a weekly_mission_rotation table with ON CONFLICT (week_start_date) DO NOTHING for concurrent initialization. This keeps missions fresh without requiring manual configuration.
7. The Scheduled Path: Cron Jobs
Four cron jobs handle scheduled game-pattern updates and time-bounded reward processing. All run in the America/Denver timezone to match the facilities' operational hours. The scheduler is @nestjs/schedule (which wraps the cron npm package), and every job passes the same constant:
export const CURR_APP_TIMEZONE = 'America/Denver'
@Cron('0 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:00 AM: rotate weekly pattern
@Cron('5 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:05 AM: weekly leaderboard rewards
@Cron('10 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:10 AM: top 10 age group mission
@Cron('5 0 1 1,4,7,10 *', { timeZone: CURR_APP_TIMEZONE }) // Jan/Apr/Jul/Oct 1, 12:05 AM: battle season rewardsDaylight Saving Time (DST) is handled by the cron library, which evaluates the expression in the named zone, so "12:05 AM Denver" stays 12:05 AM local across the March and November changes. Daily mission dates, the weekly anchor, and the play log use the same zone inside PostgreSQL (current_timestamp AT TIME ZONE 'America/Denver'); the database session itself runs in UTC. Game-mode rotation and the battle season calendar resolve the same zone in Node.js.
Each job is wrapped in a small withRetry() helper that allows three attempts with a fixed five-minute wait between them. A job that fails all three attempts logs the error and waits for its next scheduled run.
Weekly Leaderboard Rewards – Monday 12:05 AM
Every Monday, the system ranks the previous week's weekly pattern scores. Players are split into age group leaderboards (grouped by age bracket and gender) plus an overall leaderboard. Ranks are matched against a leaderboard_reward_config table that maps rank ranges to CC, XP, and badge rewards. A unique index on (player_id, leaderboard_type, season, week_start_date) prevents duplicate awards.
Top 10 Age-Group Update – Monday 12:10 AM
Five minutes after weekly rewards, a second cron processes the seasonal mission that tracks how many weeks a player placed in their age group's top 10. It reads from the player_leaderboard_reward table that the 12:05 AM job just populated.
The five-minute gap is the only dependency mechanism. The 12:10 AM job does not check whether the 12:05 AM job finished; it simply queries last week's top-10 rows and processes whoever is there. If the first job is still running, or is in its 5-minute retry wait, the second job sees a partial or empty result and finishes early. Two properties make this acceptable rather than dangerous. First, the job recomputes each player's count from player_leaderboard_reward as an absolute value rather than an increment and skips the player when the stored value already matches. This allows the job to be rerun manually without double-awarding. Second, the first job's retry schedule (12:10 and 12:15) means the rows usually exist within minutes. Affected players are picked up the next time they finish in the top 10, when the recount includes the missed week. We accept that a player can see this mission update one week late.
Battle Season Rewards – Quarterly
On the first day of each quarter, battle season rewards process. Two leaderboards are computed: total battle points (sum of points from battle royale and OnePod ranked games) and battle efficiency (points-to-expected ratio, preventing pure volume from dominating). Both leaderboards use the same check-then-insert and unique index pattern for idempotency, with the season start date in week_start_date.
8. Idempotency and Known Risks
No distributed system is perfectly reliable. Here is an honest assessment of where our idempotency holds and where it has gaps.
Where It Holds
System | Mechanism | Status |
Achievements | achievementExists(score_id, type) before every insert | Fully idempotent |
Seasonal thresholds | last_completed_threshold_id bookmark, updated in the reward transaction | Safe via transaction |
Leaderboard rewards | Check-then-insert inside a transaction, unique index as backstop | Fully idempotent |
Level-up rewards | player_level_reward_history unique on (player_id, level_config_id, season) with ON CONFLICT DO NOTHING | Fully idempotent |
The achievement check is a single indexed lookup:
private async achievementExists(score_id: number, achievement_type: AchievementType) {
return !!(await this.knex('player_achievement').where({ score_id, achievement_type }).first())
}Leaderboard rewards use the unique index from the migration plus an explicit check, because the check lets us skip the CC, XP and notification writes cleanly instead of catching a constraint error:
// migration
table.unique(['player_id', 'leaderboard_type', 'season', 'week_start_date'],
{ indexName: 'uq_player_leaderboard_reward' })
await this.knex.transaction(async trx => {
const existing = await trx('player_leaderboard_reward')
.where({ player_id, leaderboard_type, season, week_start_date }).first()
if (existing) return
await trx('player_leaderboard_reward').insert({ player_id, leaderboard_type, season, week_start_date,
rank, cc_awarded: tier.cc_reward, xp_awarded: tier.xp_reward, badge_config_id: tier.badge_config_id ?? null })
if (tier.cc_reward > 0) {
await trx('player_transaction').insert({ player_id, transaction_type: 'add',
free_live_credits: tier.cc_reward, free_virtual_credits: 0, xp_earned: 0 })
}
if (tier.xp_reward > 0) {
await this.levelUpService.recordXp(trx, player_id, season, tier.xp_reward, XpSource.WeeklyPattern)
}
})Wherever the article says a row is inserted with conflict protection, the actual statement is PostgreSQL's INSERT ... ON CONFLICT (...) DO NOTHING, written in Knex as .onConflict([...]).ignore(). Reward history rows use .onConflict([...]).merge([...]), which becomes ON CONFLICT ... DO UPDATE SET.
Where It Has Gaps
Daily and weekly missions track progress with increment-based counters, and the seasonal player_season_data counters (games played, total points, days played) are incremented the same way. If a score event is processed twice, those counters get double-incremented. The completion rewards are protected by a separate reward history table and the threshold bookmark, but the progress inflation can cause missions to complete earlier than intended.
With no BullMQ retries configured, a second run of the same event can come from only two places: the in-process deadlock retry, which re-runs processScoreEvent() from the top after a partial write (the function is not one transaction), and BullMQ's stalled-job recovery, which re-queues a job once if its lock expired mid-run.
Known Risks We Accept
Pool exhaustion: The HTTP server and BullMQ workers share a single database connection pool. Under high scoring load, 5 concurrent workers multiplied by multiple queries per job plus HTTP traffic can compete for connections. This has not been a production issue yet, but it could become a scaling constraint if scoring volume increases significantly.
Sequential mission processing: Steps 3 to 5 in the worker run sequentially. A failure in daily missions prevents weekly and seasonal from running for that score event. We chose this over parallel execution to avoid connection pool pressure, but it means one service's bug can cascade.
Single-process cron: The cron jobs have no distributed lock. The API runs as one Node.js process today; running two instances would fire every cron job in both, and the second run would fail on the unique index rather than double-award.
Building Reward Systems That Can Handle Different Processing Needs
A reward system does not need to process every event in the same way. In this case, separating immediate, asynchronous, and scheduled work keeps player-facing responses fast while moving heavier evaluations and time-based rewards to paths better suited to them. The approach has been running in production across multiple Core Soccer Tech facilities, handling over 38 million scored points from 7,000+ players. The architecture also shows why transaction boundaries, idempotency, queue behavior, failure handling, and known system constraints need to be considered together when designing production systems. These are the same concerns that shape backend engineering for applications that need to support growing workloads without making every workflow part of one synchronous request.
