From Overnight Batch To Intraday: What An Investment Bank's Dev Team Must Learn To Build A Real-Time Risk Engine On Top Of Legacy

~70% - Of most financial-services IT budgets spent maintaining legacy systems - an estimated $350bn a year industry-wide  ·  Overnight → intraday - Volatility events in 2026 compress the most consequential risk decisions to intraday or real time  ·  Batch is a blind spot - A VaR number computed last night cannot inform a decision the market is forcing this afternoon  ·  Concurrency - Legacy analytical databases were not built for the concurrent query patterns live intraday risk demands

Every investment bank has one, and every developer who has worked on it knows its personality: the overnight risk batch. It is the sprawling, fragile, hours-long run that starts after the close, pulls positions, prices, curves and reference data from a dozen systems that were never designed to talk to each other, reconciles them as best it can, and grinds out the firm's Value-at-Risk, sensitivities and exposures for the risk managers to read over breakfast. It is a triumph of engineering endurance, and in 2026 it is increasingly the wrong answer to the wrong question. The moments that generate the most consequential risk decisions - a central-bank surprise, an options-expiry cascade, a credit event, a geopolitical shock - compress the decision window to intraday or real time, and a risk number computed last night, on last night's positions and last night's prices, is not a slightly stale answer but a blind spot exactly when the firm most needs to see. So banks, asset managers and prime brokers are re-architecting risk analytics from overnight batch to intraday and real-time engines - and they are doing it on top of legacy platforms that consume an estimated 70% of financial-services IT budgets and some $350 billion a year across the industry, and whose fundamental query patterns simply do not map onto what live risk needs. As a team that builds trading and risk technology, we think this is one of the most instructive pain points in enterprise engineering, and this playbook is a practical, code-side account of what a bank's dev team must learn to make the transition.

Lesson One: Why The Batch Cannot Simply Be Made Faster

The first thing a dev team learns is that you cannot get to real-time risk by optimising the overnight batch, because the batch's slowness is a symptom of its architecture, not of inefficient code. The batch exists because risk data is fragmented: positions live in one set of systems, market data in another, curves and volatilities in a third, reference data in a fourth, and each fragment has its own schema, timing and quality, so the batch's first and largest job is reconciling them into something coherent enough to compute on. Making that reconciliation faster does not remove it. The second reason is the platforms it runs on: the legacy analytical databases and warehouses underneath most risk stacks were built for scheduled, sequential, large-scan workloads - exactly what a nightly batch is - and they do not scale efficiently under the concurrency that intraday risk demands, where many desks, traders and risk managers need many live calculations at once and the query patterns of live risk simply do not map cleanly onto what those platforms were designed to do. The third reason is that batch computes everything from scratch each time, which is tolerable once a night and impossible every few seconds. So the transition is not a performance project; it is an architectural one, and recognising that early saves a team from months spent shaving minutes off a run that needs to become continuous.

Lesson Two: Unify The Data, Then Stream It

The foundation of a real-time risk engine is a unified, cross-asset risk data model - a single coherent representation of positions, trades, market data, curves and reference data across every asset class, into which the fragmented silos are continuously fed - because you cannot compute live risk on data you are still reconciling. Building it is the largest and least glamorous part of the work: defining a canonical model that can represent equities, rates, credit, FX and derivatives consistently; writing the adapters that translate each legacy system's output into it; and establishing the data quality rules that decide what to do when sources disagree. Once the model exists, the second shift follows: replace the once-a-night pull with continuous streaming, so that a new trade, a price tick, a curve update or a reference-data change flows into the unified model as an event the moment it happens rather than being collected at the close. This turns the risk data layer from a snapshot into a living state, which is the precondition for everything intraday. It also changes the platform underneath: the live state needs a store built for high-concurrency, low-latency reads and writes - the kind of real-time analytical database now used for pre-trade risk and intraday margin - rather than a warehouse built for nightly scans. A team that gets the unified model and the streaming right has done most of the hard work; a team that tries to compute real-time risk directly on top of fragmented legacy sources will never get there.

• Define a canonical cross-asset model - one coherent representation of positions, market data, curves and reference data that every source is translated into.
• Adapt each legacy source into it - the reconciliation the batch did nightly becomes continuous translation at the boundary of each silo.
• Stream events, don't snapshot - trades, ticks, curve and reference updates flow into the live model as they happen.
• Use a store built for concurrency - live risk needs high-concurrency low-latency access, not a warehouse tuned for scheduled scans.
• Encode data-quality rules - decide explicitly what happens when sources disagree, because live risk cannot wait for a human to reconcile.

Lesson Three: Compute Incrementally, Accelerate The Heavy Parts

With live data in place, the engine itself must change from recomputing everything to computing incrementally: when a single trade arrives or a price moves, the engine updates the affected exposures and sensitivities rather than re-running the firm's entire risk from scratch, which is the only way a calculation can keep pace with the market. Architecturally that means decomposing risk into components that can be updated independently and aggregated - position-level sensitivities that roll up to desk and firm level, so a change touches a small part of the tree and the aggregates are recomputed cheaply - and caching intermediate results (revalued curves, calibrated surfaces) so they are reused until their inputs change. Some risk analytics, though, are irreducibly heavy: full revaluation of large derivatives books across many scenarios, Monte Carlo simulation for VaR and expected shortfall, stress testing across thousands of paths. This is where 2026's stack pulls in GPUs, because at intraday scale risk succeeds or fails on the ability to deliver predictable, low-latency compute for exactly these workloads, and the massively parallel arithmetic of scenario revaluation maps naturally onto GPU hardware in a way that turns an overnight job into minutes or seconds. The dev-team skills this demands are real and specific: designing computations for incremental update and caching, structuring scenario workloads for parallel execution, and operating GPU compute reliably - a long way from the sequential batch programming most risk teams grew up with, and a genuine learning curve to respect.

The Bottom Line

The overnight risk batch is a monument to engineering endurance and, in 2026, the wrong answer to the wrong question: the moments that force the most consequential risk decisions - central-bank surprises, expiry cascades, credit events, geopolitical shocks - compress the window to intraday or real time, and a number computed last night is a blind spot exactly when the firm most needs to see. Moving to a real-time risk engine on top of legacy platforms that consume 70% of IT budgets is one of the defining pain points of investment-bank technology, and what a dev team must learn is specific. The batch cannot be optimised into real time, because its slowness is architectural - fragmented data, platforms built for scheduled scans not concurrency, and from-scratch recomputation. The foundation is a unified cross-asset risk data model fed by continuous streaming of trades, prices, curves and reference data into a store built for concurrent low-latency access. The engine must compute incrementally - updating affected components and aggregating, caching intermediates - and pull in GPUs for the irreducibly heavy scenario, Monte Carlo and stress workloads. And above all the transition must be incremental, running the new engine in parallel with the batch, reconciling until they agree, migrating desk by desk, and retiring the old path only once the new one has earned it - because the big-bang rewrite is the classic way this programme dies. Respecting that curve, and building the unified, streaming, incremental, accelerated engine correctly without ever breaking the firm, is exactly the correctness-critical financial engineering we specialise in.

References & Further Reading

• Databricks - modern risk demands a real-time foundation: the CRO's mandate: https://www.databricks.com/blog/modern-risk-demands-real-time-foundation-cros-mandate
• SingleStore - real-time pre-trade risk and intraday margin analytics (legacy platform concurrency limits): https://www.singlestore.com/blog/real-time-pre-trade-risk-intraday-margin-analytics/
• Arc Compute - from overnight risk to intraday decisions: how AI and GPUs are reshaping financial risk management: https://www.arccompute.io/resources/arc-blog/from-overnight-risk-to-intraday-decisions-how-ai-and-gpus-are-reshaping-financial-risk-management
• Thoughtworks - the stranger core: modernising bank infrastructure for 2026 (legacy cost and incremental modernisation): https://www.thoughtworks.com/insights/articles/the-silent-run-on-the-bank
• Deloitte Insights - 2026 banking and capital markets outlook: https://www.deloitte.com/us/en/insights/industry/financial-services/financial-services-industry-outlooks/banking-industry-outlook.html