Delivered remotely for a business based in Amsterdam, Netherlands. Names withheld by agreement.
Project Overview
Industry: Live-event entertainment and prediction gaming.
Type of solution: An end-to-end product build — a REST API over a relational database, a set of scheduled data-ingestion pipelines against a licensed third-party provider, and a cross-platform mobile application — delivered as a first working version of a new consumer product.
Business context: Prediction and fantasy products in live-event entertainment have historically been built for committed enthusiasts: season-long formats, deep statistical interfaces, and a level of sustained attention most people will not give. The gap is short-form. A user should be able to open an app, pick a single upcoming stage, make a small number of clear head-to-head calls, and be done — then watch it resolve over the course of an afternoon. The commercial model is sponsor-funded rather than transactional: winners receive physical merchandise supplied by sponsor brands. No money moves through the application, and no payment credential of any kind enters it.
General users: Players creating and resolving prediction entries, with a role model in place for moderation and administration as the product grows.
General purpose: To turn the live data stream from competitive events into a short-form game that can be played casually, resolved the same day, and funded by sponsors rather than by the player.
The Business Challenge
The upstream data feed permits roughly one request per minute. This is the fact that shapes everything. The licensed provider supplying schedules, venues, participant reference data, rankings, pairings and live scoring enforces a hard rate limit measured in requests per minute, not per second. A product that needs to look live cannot be built by calling that provider when a user opens a screen.
Different data changes at wildly different rates. Venue and layout reference data is effectively static. Participant lists change daily in the run-up to an event. Pairings are released on a fixed cadence shortly before each stage begins. Live scoring changes every few minutes while an event is in progress. A single refresh strategy either wastes the entire rate-limit budget on data that has not changed, or leaves live data stale. Neither is acceptable.
The game is time-bounded and the clock is external. Entries must close before an event's first start time — a deadline set by the event, not by the platform. A user's entry means something different depending on whether its event has not started, is in play, or has finished, and the transition between those states is driven by the calendar rather than by anything a user does.
The data a user needs may not exist yet. Pairings are published by the provider shortly before each stage. A user browsing two days ahead is looking at an event whose match-ups have genuinely not been decided. "Not available yet" has to be a normal, well-handled state in the product rather than an error condition.
Provider payloads are deeply nested and inconsistently shaped. Fields appear conditionally, identifiers need transformation before they can be used as keys, and the structure varies between the provider's endpoints. This all has to be normalised on the way in, because nothing downstream can be built on data that is shaped differently depending on where it came from.
Ingestion has to be safely rerunnable without a job framework. These pipelines run on a schedule against a live database. A rerun must not duplicate anything, and a run that fails part-way must be safely resumable — with no orchestration layer to lean on.
It had to be demonstrable before the rules were finished. The game mechanics were still being defined while the data layer was being built. The product needed to be shown, played and reacted to early, which meant sequencing the build so that the parts that were certain could ship while the parts that were not could be represented and revisited.
Our Approach
Never call the provider from a request handler. The single decision the whole architecture rests on. Data ingestion was built entirely out-of-band, as standalone scheduled processes that write directly to the database. By the time a user request arrives, everything it needs is already local. Upstream rate limits and upstream latency became a background concern instead of a user-facing one, permanently — and they stay that way no matter how many people are using the app at once.
Split ingestion by how fast the data changes, not by what it is. Rather than one pipeline per entity, we built four pipelines along the axis of volatility: a rarely-changing tier for schedules, venues and published rankings; a daily tier for participant lists and statistics; a pre-event tier timed to pairing releases; and a live tier for in-play standings and detailed scoring. Each tier gets a cadence proportionate to how quickly its data goes stale, so the rate-limit budget is spent where it earns something.
Treat the rate limit as a design constraint, not an error to handle. Pacing between successive provider calls is built into the pipelines by construction, so the limit is respected rather than discovered through failures and retries.
Derive entry status instead of storing it. An entry's lifecycle state — not yet started, in play, finished — is computed at read time from the event's own dates rather than held in a column that something has to maintain. There is no state machine, no scheduled job advancing statuses, and no way for a stored status to disagree with reality. A whole category of bug simply does not exist in this system.
Make refreshes safely repeatable. Every pipeline checks for existing records before inserting, and volatile data is refreshed by a scoped delete-and-reinsert bounded to a single event and stage. A rerun is harmless, a partial run is resumable, and a live refresh is cheap and cannot leave a half-updated slice behind.
Match the data-access technology to the intent. Entity writes the application originates — accounts, entries, picks — go through an ORM, where its consistency and relationship handling are worth having. Multi-table analytical reads over ingested data go through parameterised raw SQL against a connection pool, where an ORM would have produced materially worse queries. Two tools, chosen deliberately, each doing what it is good at.
Cancel superseded requests everywhere, by default. Every asynchronous operation in the mobile client is a saga registered to cancel its own in-flight predecessor. In a step-by-step flow where a user can move faster than the network, this is what stops a stale response overwriting fresh state — and applying it uniformly rather than selectively means it cannot be forgotten on the one screen where it matters.
Prepare the read path before it is needed. The read patterns here are naturally nested, and the current interface satisfies them with follow-up queries. A batched-join query layer was written against the core entities in advance, ready to collapse those round trips once the data model settled — an identified path forward rather than a rewrite waiting to happen.
Cover unavoidable waiting with motion. The moments where the system is working and the user cannot see it — computing an entry, waiting on results, confirming an outcome — use purposeful animated states rather than spinners. Latency that cannot be removed can at least be made to feel intentional.
The Solution
Volatility-tiered ingestion. Four independent scheduled pipelines, each matched to the refresh rate its data actually needs, collectively keeping a complete local mirror of schedules, venues, participants, rankings, statistics, fields, pairings and live scoring current within the upstream rate limit.
A local data mirror serving every request. The API reads exclusively from local storage. Provider availability, provider latency and provider rate limits are invisible to users by design.
Account and identity. Registration and sign-in with hashed credentials and stateless bearer-token authentication, a role model with dedicated role-checking middleware, and a complete multi-step account-recovery flow with its own dedicated screens on the client.
Event discovery by date range. A user selects a date window and receives the candidate events within it, each with its venue detail and prize preview attached.
Entry creation. An entry is created against a chosen event and a chosen stage, with the flow advancing automatically through the available stages and on to the next event in the selected window.
Derived entry lifecycle. Each entry presents as not yet started, in play, or finished — computed from the event's own calendar at the moment it is read, with a different destination screen for each state.
The prediction flow. For a chosen event and stage, the platform presents a shortlist of match-ups, drills down through segment and match-up level, and records the user's call. Comparative context is shown against each option so the choice is informed rather than blind.
An entry deadline enforced against the event clock. A countdown component makes the closing time visible and unambiguous, tied to the event's own first start time rather than to a platform-side timer.
Standings and rankings. Per-event standings across entrants and an overall ranking view, both built over ingested results.
Participant directory and profiles. A searchable directory, with individual profiles presenting ingested statistics across tabbed sections including a chart view.
Sponsored prize fulfilment. Winning entrants view the sponsored item for their event and submit shipping details for physical delivery, followed by confirmation. No payment mechanism is involved at any point in this flow.
A sponsor showcase. Dedicated placement for the sponsor brands funding the prize pool, surfaced within the entry flow as well as on its own screen.
Key Features
Volatility-tiered data ingestion Four independent scheduled pipelines split by how fast the underlying data changes rather than by entity type, each running at a cadence proportionate to its data's staleness rate — the mechanism that makes a heavily rate-limited feed viable behind a live-feeling product.
Fully decoupled ingestion path No user request ever calls the upstream provider. Rate limits and upstream latency cannot affect user-facing response times regardless of concurrent load.
Derived entry lifecycle state Entry status is computed at read time from event dates rather than stored and maintained, eliminating both the state machine and the possibility of stored status disagreeing with reality.
Idempotent, resumable refreshes Existence checks before insert, and scoped delete-and-reinsert bounded to a single event and stage, so reruns are harmless and partial runs are safely resumable without an orchestration layer.
Deadline enforcement against an external clock Entry closing is tied to the event's own first start time and made visible through a countdown, so the cut-off is unambiguous to the player and driven by the event rather than the platform.
Guided prediction flow with comparative context A shortlist of match-ups is prepared server-side and presented with supporting context for each option, so a player makes an informed call in seconds rather than researching a full field.
Request cancellation applied uniformly Every asynchronous operation cancels its superseded predecessor by default, preventing out-of-order responses from corrupting state in a fast-moving step-by-step flow.
Intent-matched data access An ORM for transactional entity writes and parameterised raw SQL for multi-table analytical reads — each chosen for the workload it serves rather than one forced across both.
Sponsor-funded physical rewards A complete prize path from win notification through item presentation to shipping capture and confirmation, with no payment mechanism anywhere in the product.
Purposeful motion over unavoidable latency Animated state screens at the specific moments where the system is working invisibly, making necessary waiting feel deliberate rather than broken.
Technical Architecture
Mobile client. A cross-platform application built on a managed React Native workflow, with a drawer-based authenticated shell containing nested stacks and modal groups, and a separate stack for the unauthenticated flow — the two swapped at the root according to stored token state resolved at launch. All asynchronous work runs through a saga middleware layer with uniform cancellation of superseded requests. The interface uses virtualised swipeable lists, tabbed views, chart components, comparative progress indicators, a circular countdown, a code-entry field, date-range selection and animated state screens, with every dimension driven through a custom viewport-scaling utility for proportional consistency across device sizes.
API layer. A compact Node/Express application with resource routers mounted behind stateless bearer-token middleware, and separate composable middleware for role checks. Handlers are async-aware throughout.
Data access layer. Deliberately split by intent. An ORM owns the entities the application creates and their relationships; a connection pool with parameterised raw SQL serves multi-table analytical reads over ingested data. A batched-join query layer over the core entities is prepared for the nested read patterns as the data model settles.
Ingestion layer. Four standalone scheduled processes, tiered by data volatility, calling a licensed third-party provider with pacing built in to respect its rate limit, normalising nested payloads, and writing directly to the database with existence checks and scoped refreshes.
Data layer. A relational schema covering accounts and roles, event and venue reference data, participant reference data and statistics, published rankings, fields, pairings, standings, detailed scoring, user entries, user picks and rankings — with the reference schema maintained explicitly alongside the ORM models.
Supporting services. Transactional email over SMTP for account recovery, and managed cloud relational storage.
Flow: Licensed live-data provider → Four volatility-tiered scheduled ingestion pipelines → Local relational store → Token-authenticated REST API (ORM writes / raw SQL analytical reads) → Mobile client with saga-managed state and uniform request cancellation
Technology Stack
| Category | Technology |
|---|---|
| Backend runtime | Node.js |
| Backend framework | Express |
| Database | PostgreSQL |
| Transactional data access | Sequelize ORM |
| Analytical data access | Connection-pooled parameterised raw SQL |
| Prepared query layer | GraphQL with batched-join resolution |
| Authentication | Stateless signed bearer tokens |
| Password hashing | bcrypt |
| Outbound integration | HTTP client with rate-limit pacing |
| Transactional email | SMTP delivery |
| Mobile framework | React Native, managed workflow |
| Mobile navigation | Stack, drawer and nested modal navigators |
| Mobile state management | Redux with saga middleware |
| Mobile persistence | Device async storage |
| Mobile data display | Tabbed views, chart components, virtualised swipeable lists |
| Mobile input | Date-range selection, confirmation code entry, search |
| Mobile motion | Vector animation for transitional states |
| Mobile layout | Custom viewport-scaling utility with self-hosted type family |
| Hosting target | Managed cloud relational database with scheduled job execution |
Technical Challenges & Solutions
| Challenge | Our Approach |
|---|---|
| An upstream data feed limited to roughly one request per minute, behind a product that must feel live | Ingestion moved entirely out of the request path into scheduled processes writing to a local mirror, so no user request ever depends on the provider — making upstream rate limits and latency permanently invisible to users regardless of concurrent load. |
| Data within one feed changing at wildly different rates, from near-static to every few minutes | Four independent pipelines split by data volatility rather than by entity, each on a cadence proportionate to how fast its data goes stale, so the constrained request budget is spent only where it buys freshness. |
| An entry's meaning changing over time according to an external event calendar | Lifecycle state derived at read time from the event's own dates rather than stored in a maintained column — removing the state machine, the maintenance job, and any possibility of stored status disagreeing with reality. |
| Scheduled pipelines that must be safely rerunnable and resumable without an orchestration layer | Existence checks before every insert, and volatile data refreshed by scoped delete-and-reinsert bounded to a single event and stage, so reruns are harmless, partial runs resume cleanly, and no refresh can leave a half-updated slice. |
| Deeply nested, conditionally shaped provider payloads with identifiers unusable as keys | Normalisation applied at ingestion time, so every downstream consumer works against one consistent internal shape rather than accommodating provider variation at each point of use. |
| Nested read patterns that an ORM would serve poorly and raw SQL would serve well | Data access split by intent — ORM for transactional entity writes, parameterised pooled SQL for multi-table analytical reads — with a batched-join query layer prepared in advance as the path to collapsing remaining round trips. |
| A multi-step flow over a mobile connection where users move faster than the network | Uniform request cancellation across every asynchronous operation, applied by default rather than selectively, so a superseded response can never overwrite fresher state. |
| Users reaching data the provider has not published yet | "Not yet available" designed as a normal, explicitly handled product state with its own presentation, rather than surfacing as an error or an empty screen. |
| Building a demonstrable product while the game rules were still being defined | Sequenced so the settled data and identity layers were built properly and completely, while unsettled areas were represented and deliberately deferred — keeping the product showable and playable without committing engineering effort to rules that were still moving. |
Security & Reliability
Credential handling. Account passwords are hashed with a deliberately slow adaptive hashing function and are never returned by any authentication response.
Stateless token authentication. Access is governed by signed bearer tokens verified on every protected route, keeping the application tier free of session state and horizontally scalable.
Role-based middleware. A role model with a dedicated join table and separate role-checking middleware, composable onto individual routes as elevated capabilities are introduced.
Parameterised queries throughout. Every database statement in the API uses bound parameters rather than string construction, including the multi-table analytical reads.
Multi-step account recovery. Recovery is a sequenced flow with email verification and code entry rather than a single-action reset, with a dedicated client screen for each step.
Ingestion isolation. Because the ingestion layer is entirely separate from the request path, a provider outage or upstream failure degrades data freshness rather than taking the product down. The application continues serving from its local mirror.
Idempotent data operations. Existence checks and scoped refreshes mean a repeated or interrupted ingestion run cannot corrupt the data set.
Environment-based configuration. Application database connectivity is resolved from environment configuration rather than compiled into the application.
Scalability & Performance
A read-mostly workload served by a read-mostly architecture. The overwhelming majority of traffic is reads against locally mirrored data. The write path that matters — ingestion — is fully decoupled and scheduled, so it never competes with user requests.
Upstream constraints permanently isolated. Provider rate limits and provider latency sit behind the ingestion boundary. Adding users does not add load on the provider at all, which is the property that makes this product scalable despite its dependency.
Stateless application tier. Token-based authentication with no server-side session state means the API tier scales horizontally without shared infrastructure.
Scheduled work outside the request process. Ingestion runs as separate scheduled execution rather than inside the web process, so long-running data work cannot affect request handling.
Query strategy matched to query shape. Analytical multi-table reads run as purpose-written parameterised SQL through a connection pool rather than being assembled by an ORM, with a batched-join layer prepared to collapse remaining nested round trips as volumes grow.
Bounded refresh operations. Live data refreshes are scoped to a single event and stage rather than rebuilding tables, keeping the most frequent write operation in the system cheap.
Efficient client rendering. Virtualised list components for all data-driven lists, cancellation of superseded requests to avoid wasted render cycles, and a scaling utility that keeps layout computation trivial across device sizes.
Latency covered where it cannot be removed. Purposeful animated states at the specific points where the system is working invisibly.
Business Outcomes
- A short-form prediction product exists and is playable end to end, from account creation through entry building and prediction to result and prize fulfilment.
- A severely rate-limited data dependency was made compatible with a live-feeling product, through an ingestion design that isolates the constraint completely rather than working around it request by request.
- Data freshness is proportionate to need across four distinct volatility tiers, so the constrained request budget is spent where it changes what a user sees.
- Entry state cannot drift out of sync with reality, because it is derived from the event calendar rather than stored and maintained.
- Scheduled data operations are safely repeatable, so an interrupted or repeated run is a non-event rather than an incident.
- The product was demonstrable early, with the settled data and identity layers built properly while unsettled game rules were deliberately deferred rather than guessed at.
- A sponsor-funded reward model is supported end to end with no payment mechanism and no payment credential entering the application at any point.
- The path to scale is identified rather than deferred, with a batched-join query layer written against the core entities ahead of the volumes that will require it.
Why it worked
Most products that depend on someone else's data feed are built as though the feed were free. Then the rate limit arrives, and the architecture has to be unpicked from the request handlers outwards. It is one of the most common and most expensive corrections in integration-heavy product work, and it is entirely avoidable — but only if the constraint is treated as an architectural input on day one rather than an operational problem discovered later.
Our team builds for the constraint first. Here that meant moving every provider call out of the request path before a single screen was designed, then splitting ingestion along the axis that actually mattered — how fast each kind of data goes stale — so a tight request budget could be spent where it changed what a user saw. It meant deriving entry state from the event calendar instead of storing it, removing an entire class of synchronisation bug rather than building the machinery to manage one. It meant choosing an ORM and raw SQL for different jobs in the same codebase because the two workloads genuinely differ, and cancelling superseded requests uniformly rather than on the screens where someone happened to remember.
It also meant being straightforward about sequencing. This was a new product with rules still being written, and the honest approach was to build the parts that were settled properly and completely, represent the parts that were not, and keep the thing playable while the questions got answered. That is how early-stage products actually get built well — not by pretending everything is known, but by knowing which decisions are load-bearing and getting those right first.
Our teams work across Node and Express API design, relational data modelling, scheduled ingestion against rate-limited and awkwardly shaped third-party feeds, and React Native product delivery — with the judgement to identify which single constraint a system must be designed around, and the discipline to design around it before anything else is built.
Final Summary
Live competitive events generate a continuous stream of structured data, and almost every product built on it asks for a season of commitment. This one asks for a minute. A player picks a date window, chooses a single stage of a single event, makes a handful of head-to-head calls on prepared match-ups, and watches it resolve the same afternoon — with a sponsor-supplied physical prize at the end instead of a wager, and no payment mechanism anywhere in the product.
The engineering story is the interesting one. The licensed feed powering the entire experience permits roughly one request per minute, and the app has to feel live. Our team resolved that by ensuring no user request ever touches the provider: ingestion was moved entirely out of the request path into scheduled processes maintaining a local mirror, then split into four pipelines along the axis of data volatility — near-static reference data, daily participant changes, pre-event pairing releases, and in-play scoring — so a tight request budget buys freshness only where freshness is visible. Refreshes are idempotent and scoped, entry lifecycle state is derived from the event calendar rather than stored, and the read and write paths use different data-access technologies because they are genuinely different workloads.
On the client, a saga-driven state layer cancels superseded requests uniformly so a fast-moving multi-step flow cannot be corrupted by an out-of-order response, and unavoidable waiting is covered with purposeful motion rather than spinners. The result is a first working version of a new consumer product — built with its hardest constraint designed around from the beginning rather than discovered in production, and with the path to scale written before the volumes that will need it.