|
|
|
|
KNOWLEDGE, REFINED • Builders 🔧 • October 16, 2026
|
|
|
📊 Scanned 7,896 articles • Selected top 14 role-critical insights for today.
|
|
|
🤔 CONTRARIAN TAKEAWAY
The other side of ai market outlook
601 articles say ai boom, but 178 argue ai bubble.
|
|
🚀 AI & Innovation
|
|
🚀 AI & Innovation
via Vercel News • 9 min read
|
AI SDK 7 is a major release for building production agents in TypeScript. The SDK has grown from model calls and chat primitives into a broader agent platform for developing, running, integrating, and observing agents across text, audio, realtime, image, and video. Every major provider is supported out of the box.At a GlanceDevelop agents with reasoning control, tool and runtime context, provider files and skills support, MCP Apps, and a terminal UI.Run agents with tool approvals, durable WorkflowAgent execution, first-class timeouts, and sandbox support.Integrate any agent harness such as Codex, Claude Code, Deep Agents, OpenCode, or Pi.Observe agent behavior with redesigned telemetry, @ai-sdk/otel, Node.js tracing channel support, lifecycle callbacks, and step performance statistics.Build beyond text with stable speech/transcription APIs, richer file parts, image generation and editing, multimodal embeddings, reranking, realtime voice (experimental), and video generation (experimental).Upgrade with clearer APIs through the new migration skill, codemods, and migration paths for ESM, Node.js 22, instructions, telemetry, stream, finalStep, runtimeContext, and toolsContext.Before You UpgradeAI SDK 7 introduces two breaking requirements:Node.js 22 minimum: Node 22 is required because the SDK depends on APIs (including the native fetch implementation and improved AsyncLocalStorage semantics) that are not backported to earlier LTS lines.ESM imports required: AI SDK 7 requires ESM imports (import syntax or .mjs files). CommonJS require() is not supported. Update your package.json to include "type": "module" or migrate individual files to .mjs.Run the v7 codemods to automate most import and rename changes before manually reviewing semantic migration items. See the full v7 migration guide.Develop AgentsProvider-agnostic reasoning control: generateText and streamText now support a top-level reasoning option that maps to provider-native settings across OpenAI, Anthropic, Google, Groq, xAI, Bedrock, Fireworks, DeepSeek, Open Responses, and OpenAI-compatible providers. Note that exact behavior and available parameters vary by provider.Typed runtime context: Shared orchestration state lives in runtimeContext and flows through prepareStep, approval functions, lifecycle callbacks, telemetry, ToolLoopAgent, and WorkflowAgent.Scoped tool context: Tools can declare a contextSchema, and callers provide per-tool values through toolsContext, so third-party tools only receive the secrets or config they need.Provider file uploads: uploadFile uploads large inputs once and reuses provider references in later calls, reducing redundant uploads in repeated PDF, dataset, image, and multi-step workflows.Provider skill uploads: uploadSkill brings the same pattern to provider-managed skill environments.MCP Apps: MCP support now includes model-visible versus app-only tools, app metadata, sandboxed iframe rendering, and JSON-RPC communication for tools, resources, logs, and display updates. Add an MCP app to your agentic app.Terminal agent development: @ai-sdk/tui runs AI SDK agents in an interactive terminal UI, letting you test reasoning, tools, and markdown output before building a full app. Learn how to build your own terminal agent.Run Agents in ProductionAI SDK 7 adds the primitives agents need once they leave local demos.Tool approvals: generateText, streamText, and ToolLoopAgent can define approval policies at the call or agent level. Policies can require user approval, auto-approve, auto-deny, or delegate to typed approval functions.Hardened approval replay: Higher-risk approval flows can revalidate tool inputs and policies before continuation, use WorkflowAgent approval validation, and opt into HMAC signing. HMAC signing cryptographically binds the original tool inputs to the approval token, preventing tampering with tool arguments between the approval request and resumption.Durable execution: @ai-sdk/workflow introduces WorkflowAgent for long-running agents. Execution state is persisted to durable storage between steps, so agents survive deploys, process restarts, interruptions, and delayed approvals. To learn more, see the WorkflowAgent migration guide.Workflow-aware agent features: WorkflowAgent supports streaming, tools, approvals, typed runtime and tool context, lifecycle callbacks, stable telemetry, provider-executed approval resumption, and tool result conversion.First-class timeouts: Text generation and agent APIs can define total, per-step, per-chunk, default tool, and per-tool timeout budgets. Timeout aborts use TimeoutError, and abort reasons flow through stream and UI protocols.Sandboxed execution: The sandbox abstractions support command execution, streaming output, working directories, environment variables, abort signals, and step-level sandbox overrides.Integrate Agent HarnessesAI SDK 7 introduces a harness layer for bringing established agents into the AI SDK ecosystem. Wrap harnesses such as Claude Code, Codex, Deep Agents, OpenCode, and Pi behind the same agent interface used by the rest of the SDK.HarnessAgent: Run external agent harnesses through the AI SDK Agent interface, with standard generate and stream results.Adapters for agents: Claude Code, Codex, Deep Agents, OpenCode, and Pi harness adapters let teams plug existing agent runtimes into AI SDK apps.Configurable harness runs: Harness agents can receive sandboxes, instructions, custom skills, and tools, so the same runtime can be shaped for different products and workflows.Durable, resumable sessions: Workflow utilities, session bridging, and APIs for interrupted-turn continuation make harness runs suitable for longer tasks.Gateway-ready authentication: Harness adapters support Vercel OIDC for AI Gateway, simplifying hosted and sandboxed agent execution.Observe AgentsGlobal telemetry integrations: Register telemetry once and receive structured events across model calls, steps, tools, embeddings, reranking, and agent execution.Dedicated OpenTelemetry package: OpenTelemetry support now lives in @ai-sdk/otel, with GenAI-semantic convention spans and metrics, supplemental AI SDK attributes, and span-enrichment hooks. To learn more, see the @ai-sdk/otel docs.Node.js tracing channel: AI SDK 7 emits structured telemetry through the Node.js tracing channel, allowing observability providers to subscribe once while preserving async context across streaming and tool execution.Sensitive context controls: Runtime and tool context can be deliberately included in telemetry, with controls to prevent secrets from being exposed by default.Lifecycle callbacks: Callbacks are more consistent across core functions, agents, tools, embeddings, reranking, structured output, and UI streams. Callback payloads carry richer data for step, usage, content, file, source, warning, tool, model-call, and error events.Performance statistics: Step results expose timing and throughput metrics, including response time, total step time, tool execution time, time to first output, and output tokens per second.Build Beyond TextAI SDK 7 expands the SDK across realtime, video, speech, transcription, images, files, embeddings, and structured output.Realtime (experimental): Browser-to-provider WebSocket sessions for OpenAI, Google, and xAI, with audio/text conversations, client-driven tool calls, and normalized routing through AI Gateway.Video generation (experimental): Video generation works across AI Gateway, Google AI Studio, Google Vertex, fal, Replicate, ByteDance Seedance, Kling AI, Prodia, and xAI, with support for long-running SSE responses and safer bounded downloads.Stable speech and transcription: generateSpeech, transcribe, SpeechResult, and TranscriptionResult are stable exports.Images as files: Image parts move toward the same canonical file model as other media. Tool outputs can use a single file shape for inline data, URLs, provider references, and text-backed content.Richer media and model support: Providers add image generation, image editing, multimodal embeddings, speech, transcription, reranking, file references, reasoning files, and provider-specific media metadata.Structured output reliability: JSON Schema post-processing is stricter for Zod and Standard Schema inputs, malformed JSON extraction and repair is available for structured outputs and tool calls, and array output mode preserves transforms, coercions, defaults, and pipes.UI, Streams, and Message HandlingThe UI and streaming work in v7 focuses on making agent streams correct, composable, and reliable.Direct agent transport: DirectChatTransport can call an Agent directly from UI code.Tool approval in UI flows: UI messages support automatic approval responses and improved approval replay behavior.Framework improvements: React useChat callbacks update with current props/state; sendAutomaticallyWhen supports async conditions; Vue gains an idiomatic useChat composable; and Angular APIs are aligned with current AI SDK patterns.More reliable streams: Provider streams that end before a finish chunk are treated as errors, tool execution errors are emitted and cleaned up predictably, and streaming reasoning edge cases are handled more consistently.Provider metadata preservation: Provider metadata is retained across text generation, UI streams, tool invocations, and multi-turn provider ID mapping.Multi-step results: Top-level content, tool calls/results, files, sources, warnings, and usage now represent the full run. Final-step-only details are available through finalStep.Configure MCPThe MCP package grows from a tool transport layer into a richer integration surface for agent tools and app UIs.Protocol and metadata: MCP clients support protocol version 2025-11-25, server metadata, server instructions, ping responses, negotiated protocol headers, and public listTools().Typed tool outputs: MCP tools can expose outputSchema and structuredContent, and tool definitions can be separated from executable tools.Resource content: Tool results and prompt messages can include MCP resource_link content.App rendering: MCP Apps use tool metadata to render app-specific UI inside sandboxed iframes while keeping model-visible and app-only tools separate.Transport reliability: HTTP, SSE, and OAuth transports support custom fetch, redirect configuration, OAuth refresh deduplication, state validation, asynchronous client authentication, richer errors, and better SSE frame handling.Configure Runtime and PackagingNode.js 22 minimum: AI SDK packages require Node.js 22 or later.ESM imports required: AI SDK 7 requires ESM imports (import syntax or .mjs files). Update your package.json to include "type": "module" or migrate individual files to .mjs.Migration skill available: A dedicated migration skill is available that developers can install and ask their agent to use for AI SDK v6-to-v7 upgrades. Codemods available: The v7 codemods cover the majority of renames and cleanup migrations.Send this prompt to your AI coding agent to get started:Coming Out of ExperimentalThe following highlights cover the most impactful promotions:experimental_customProvider becomes customProviderexperimental_generateImage becomes generateImageexperimental_output becomes outputexperimental_prepareStep becomes prepareStepexperimental_telemetry becomes telemetryRenamed APIsThe following highlights cover the most impactful renames:system option becomes instructionsSystem messages inside prompt or messages require allowSystemInMessages: trueonFinish becomes onEndStreamTextResult.fullStream becomes streamCallSettings is split into model generation options and request/transport optionsDeprecated APIsThe following highlights cover the most impactful deprecations:Tool approval: needsApproval on tool() and dynamicTool() is deprecated. Move approval logic to toolApproval on generateText, streamText, or ToolLoopAgent.Stream response helpers: Result methods like result.toUIMessageStreamResponse() and result.toTextStreamResponse() are deprecated. Use top-level helpers like createUIMessageStreamResponse and createTextStreamResponse.Vue chat: The Vue Chat class is deprecated. Use the useChat composable instead.Other Migration ThemesReasoning configuration is centralized: The top-level reasoning option replaces overlapping provider-specific reasoning settings unless a provider-specific override is intentional.OpenTelemetry moved to @ai-sdk/otel: OpenTelemetry span collection is no longer built into the ai package. Telemetry is registered globally, and custom tracers move into the OpenTelemetry constructor.Request and response body retention is opt-in: Text generation results exclude request and response bodies by default.Multi-step results now represent the full run: Top-level usage, content, tool calls/results, files, sources, and warnings accumulate across all steps; final-step-only data lives under finalStep.Message parts are more canonical: Legacy media and image-specific parts move toward file parts with media types.Package-specific behavior changed: MCP HTTP/SSE redirects are treated as errors, OpenAI Responses reasoning summaries default to detailed, and Anthropic cache creation token metadata moves into standard usage fields.Upgrade PathFollow these steps to migrate an existing project to AI SDK 7:Update Node.js to 22+: Verify your runtime and CI environments meet the minimum requirement before upgrading packages.Update packages: Bump ai and all @ai-sdk/* packages to their v7 releases in your package.json.Run the v7 codemods: The codemods automate the majority of renames, import changes, and API moves. Review the diff before committing.Migrate OpenTelemetry: If you use tracing, move your setup to @ai-sdk/otel and register telemetry globally. Refer to the @ai-sdk/otel docs and the OpenTelemetry span schema for attribute details.Review semantic changes manually: Codemods cannot fully decide runtime requirements, ESM imports, instruction/message behavior, runtime/tool context separation, approval policy placement, stream helper usage, and multi-step result shapes. See the full v7 migration guide.For a guided migration, install the AI SDK v7 migration skill with npx skills add vercel/ai --skill migrate-ai-sdk-v6-to-v7. Read more
|
💡 WHY THIS MATTERS FOR YOUR ROLE
It gives engineering teams a production-ready foundation for building AI agents without stitching together multiple libraries.
|
|
|
|
🚀 AI & Innovation
via AWS News Blog • 1 min read
|
Learn how Amazon CloudWatch Omni delivers AI-powered observability purpose-built for generative AI and agentic workloads. Trace, evaluate, and experiment with AI agents across any framework—directly from your IDE or a standalone web experience—using open standards and built-in evaluators for quality, correctness, and coherence.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Observability for AI agents is becoming as critical as APM for traditional apps, and this sets the standard.
|
|
|
|
🚀 AI & Innovation
via Vercel News • 4 min read
|
Zo Computer on Vercel20x reduction in retry rate (7.5% → 0.34%)99.93% chat success rate (up from 98%)P99 latency cut 38% (131s → 81s)New models added in less than 1 minuteEvery company has servers that store data, run services, and do work around the clock. Consumers just have apps. Rob Cheung, co-founder of Zo Computer, is closing that gap. Zo is a personal AI cloud: your own servers and data that power an always-on agent."Cloud is one of the best computing models of all time, and consumers have zero direct access because it's so complicated," explained Rob Cheung, co-founder and CEO of Zo. "Now, with AI, it's finally possible for all of us to have cloud computers." Zo is a full computing environment, not just a chatbot. Rob laughs about his mom running servers and databases without knowing it. People use Zo to manage small businesses, do research, organize finances, and track health data.The 8-person company is two and a half years old and they have an ambitious goal: to onboard one million new users to personal cloud computing in 2026. That means millions of AI model calls every day, and when Zo users text their agent like a friend, they expect the same responsiveness.Death by a thousand adaptersZo gives users access to any model they want, and supports bring-your-own-key. That means their backend has to talk to every major provider: OpenAI, Anthropic, MiniMax, GLM, Fireworks, and more.Before they moved to Vercel, that meant custom adapter code for each model. Every provider required different handling for images, different key management, and different edge cases. On top of the code complexity, Zo's team was managing retries, provider routing, and fallback logic themselves.Every time a provider shipped a new model, an engineer had to write a new adapter, test the edge cases, and run the deployment pipeline. With new models released weekly, it was a constant drag on a small team building a consumer product, and their users felt it.Zo's baseline for AI model calls was a 98% success rate with a 7.5% retry rate. That means 1 in 50 messages failed or retried, adding up to tens of thousands of model fallbacks every day.AI SDK + AI Gateway: two layers, one integrationZo moved to Vercel's AI SDK and AI Gateway, which solved two distinct problems.AI SDK replaced the custom adapter code. Instead of per-provider implementations with bespoke edge case handling, Zo's engineers got a unified interface for every model, from image support to response format normalization.AI Gateway replaced the infrastructure-level complexity. Retries, fallback routing, provider health monitoring, and uptime were all handled at the routing layer in Vercel instead of in Zo's codebase. Rob's co-founder built APIs at Stripe, where developer experience was the product. He describes the combined effect of AI SDK and AI Gateway the same way: everything just works, and the pieces you don't see matter most. New model support went from an hour-long, multi-file code change to adding a config string in 30 seconds. The day MiniMax shipped M2.7, Zo had it live for users immediately. No adapter code, no edge case testing, no deploy cycle.For an 8-person team focusing on onboarding their first million users to personal cloud computing, cutting out interruptions for model support has been a huge relief.20x improvement in reliabilityDuring the rollout, Zo ran Vercel and non-Vercel routes simultaneously, creating a live A/B comparison under identical production conditions.The results:PeriodRoutePOST errorChat successRetry rateAvg attemptsBefore switchNon-Vercel4.59%99.73%7.52%1.12After switchNon-Vercel10.38%97.86%17.07%1.29After switchVercel0.45%99.93%0.34%1.00The non-Vercel route actually degraded during the same period that Vercel held steady. Retry rate dropped from 7.5% to 0.34%, a 20x improvement. Average attempts per chat hit 1.00, meaning virtually every request succeeded on the first try.On MiniMax M2.5, Zo's most-used model, the latency improvement was significant. In an apples-to-apples comparison over the same window, Vercel handled 18,139 chats versus 21,105 on non-Vercel and still performed better across the board:Average latency improved 25.7%P95: 46s → 34s (25% improvement)P99: 131s → 81s (38% improvement)For Zo's users, the P99 number matters most because they text their agents constantly throughout the day. A 131-second worst-case wait breaks that experience completely, but now 99% of requests complete in under 81 seconds.By the end of the test, 91.88% of Zo's traffic routed through Vercel, handling 3.3x larger context windows (42,500 average input tokens vs. 12,700) at a lower error rate than the non-Vercel path.Scaling to a million personal cloud ownersVercel handles Zo's AI layer through AI SDK and AI Gateway and hosts their public-facing marketing site. With reliable AI infrastructure and no adapter code to maintain, the team can focus on the product instead of the plumbing.With the pace of model developments in AI, Rob used to worry about the work required to keep up. “Now I don’t worry about it,” he said, “because with Vercel, the infrastructure just works.”Zo Computer is a personal AI cloud platform that gives every user their own cloud computer, housing data, services, and a personal agent. Users interact through a conversational interfaces like iMessage, or log in and use the environment directly. Founded two and a half years ago, Zo is an 8-person team based in New York City. Learn more at zo.computer. Read more
|
💡 WHY THIS MATTERS FOR YOUR ROLE
It provides a proven playbook for making AI systems production-grade, directly impacting uptime and user experience.
|
|
|
☁️ Enterprise Tech
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
Amazon Aurora PostgreSQL now lets you directly query Apache Iceberg and Parquet data stored in your data lake alongside live operational data—no ETL pipelines required. Powered by DuckDB embedded within Aurora, this capability enables single queries that join transactional and historical data using familiar PostgreSQL syntax. It supports AWS Glue Data Catalog, S3, and S3 Tables, with optimizations like predicate pushdown and caching for efficient performance.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
It lets engineering leaders collapse their data stack and query lake data directly from Aurora, cutting ETL costs and latency.
|
|
|
|
☁️ Enterprise Tech
via Netflix TechBlog - Medium • 15 min read
|
Netflix Conductor : The Next Chapterby Aravindan Ramkumar on behalf of the Conductor teamConductor is the workflow orchestration engine Netflix uses to stitch micro-services into reliable, observable business processes.If you’ve followed Conductor from the outside, the last thing you heard was probably the note on its GitHub repository: Netflix discontinued maintenance of Conductor OSS to refocus its resources on the internal fork.That announcement marked the end of the open-source chapter, and from the outside it may have looked like the end of the story. It wasn’t. Internally, Conductor never slowed down. If anything, it grew faster, and the demands of that growth pushed us to rebuild it from the ground up. This post is the answer to the natural question that announcement leaves open: what happened to Conductor after that?A lot, it turns out. The clearest way to put it is in terms of growth. Not long ago, Conductor crossing a billion workflow executions in a single year felt like a milestone. Today it clears that bar roughly every quarter, and the monthly volume is still climbing. Sustaining that kind of growth meant rethinking nearly every layer of the system. The data plane it runs on was modernized piece by piece. The evaluation engine was rewritten. The way it allocates work, controls concurrency, and runs compute all changes. The through-line is simple: scale forced a rethink of the evaluation engine and the layers beneath it, and everything else followed.The rest of this post walks through that evolution: how the architecture changed, the performance and scale work at the heart of it, the leap forward in developer experience, and the newer capabilities for controlling how workflows and tasks execute.How Netflix uses ConductorWhen we last wrote about Conductor, it ran a few hundred workflow definitions for a few dozen teams. That footprint has grown by orders of magnitude. Conductor now backs roughly 200,000 workflow definitions owned by around 150 applications, spanning Content and Studio Engineering, Ads, Games, and more. The runtime side has grown to match. In a typical month it executes on the order of 420 million workflows, and on a given day it runs anywhere from 5 to 20 million, with teams adding hundreds of new workflow types every month.Conductor and the Media Production SuiteOne of the clearest ways to see what Conductor does is Netflix’s Media Production Suite (MPS). It’s a set of applications that move production data into the cloud. An average Netflix title generates around 200 terabytes of original camera files, and MPS is what gets that media ingested, validated, transcoded, and delivered to the right people around the world.The multi-step pipelines behind MPS’s workflows (Footage Ingest, Dailies, VFX Pulls, and Conform Pulls) run as Conductor workflows. A dedicated orchestration layer called Studio Orchestrator stitches together the platform services these pipelines depend on, such as asset management, media encoding, and storage, using Conductor under the hood to ensure reliable execution and the durability to recover from the inevitable failures of long-running, distributed work.Beyond the studio: Ads and GamesContent engineering is where Conductor started, but it’s no longer the whole story. Two of Netflix’s newer businesses lean on it just as heavily.Netflix Ads has made Conductor a backbone of its programmatic pipelines. Creative ingestion, advertiser-certificate management, and privacy-preserving Data Clean Rooms all run as Conductor workflows.Netflix Games uses Conductor to orchestrate the path a game takes from a partner’s submission to the app stores. When a studio uploads a build, a Conductor pipeline ingests the binary, runs the required validations (for example, iOS privacy-manifest checks), and drives App Store publishing, coordinating steps that span multiple platform services and external stores with the same reliability and visibility the studio pipelines rely on.How the architecture evolvedOur earlier post described a Conductor built on a particular set of building blocks: DynoQueues for queueing, Dynomite for metadata, Cassandra (partially) for execution data, and Elasticsearch for indexing. That stack carried Conductor a long way, but as adoption climbed, almost every layer of it was eventually rethought. The data plane Conductor runs on today looks very little like the earlier one.1.0: the original stack, and where it strainedIn its first generation, Conductor leaned heavily on Dynomite (with DynoQueues as the queueing recipe layered on top) backed by Elasticsearch for indexing. It worked, but two classes of problems showed up as scale grew. The first was the datastore. Dynomite is an in-memory store: a clustering layer that adds sharding and replication on top of Redis. You scale it by adding memory and nodes, with each replica rack holding a full copy of the data, which suits a cache but not the durable record for millions of workflows a day. Dyno Queues, built on the same store, inherited those limits, and its zone-sharded design made task scheduling uneven. The second was the engine’s design: evaluation ran synchronously and mutated workflow state in several places, so competing updates could leave a workflow inconsistent with no automated way to repair it.The 2.x line was a sustained campaign to address these, and two replacements did most of the heavy lifting.2.0: Cassandra as the backboneThe most consequential change was making Cassandra the primary datastore for execution data, displacing Dynomite. Cassandra gave Conductor the elastic, horizontal scaling it needed, and removed the scaling ceiling and zone-affinity constraints that the old datastore imposed. Around the same time, large task inputs and outputs were offloaded to S3, keeping bulky data off the primary datastore and further easing the pressure on it.Timestone: a purpose-built queueing layerThe second replacement was the queue itself. Timestone, Netflix’s high-throughput, low-latency priority queueing system, took over from DynoQueues. Beyond shedding the Dynomite dependency, it brought two capabilities that mattered for reliability: a delayed-queuing mechanism, and a background repair process that could detect and recover workflows. Timestone also turned out to be foundational for work that came later. The async evaluation queue and the queue-depth-driven dynamic thread allocation described later in this post both build on it.Alongside these, the 2.x series also decoupled indexing from the critical path, moving to a Kafka-buffered, independently scalable indexer writing to Elasticsearch.3.0: consolidating the modern data planeBy Conductor 3.0, these threads came together into a coherent platform. Dynomite was fully sunset. Timestone was the queue and Cassandra the datastore. A Kafka-fed indexer pushed to Elasticsearch, with Iceberg added for long-term storage. The release also shipped automated workflow repair and multi-cluster deployment, the last of which lets workflows run in fully isolated contexts (for example, a dedicated context for the Media Production Suite, kept separate from the shared default and scaled on its own).The net effect is that by 3.0 the foundational reliability and scaling problems of the early era were largely solved, and the data plane reached roughly the shape it has today.Rebuilding the engine: Conductor 4.0By 3.0, everything around the evaluation engine had been modernized, but the engine and its data model were largely the original, and at this scale that became the bottleneck. The engine evaluated a workflow by loading its entire state into memory to decide what to do next, which meant heavy memory pressure and slow evaluations as workflows grew toward tens of thousands of tasks. It also stored a workflow’s metadata and its task payloads in the same Cassandra partition, producing oversized “wide rows” and leaving room for race conditions when several updates to the same task competed. Evaluation overhead, vertical-scaling pressure, and occasional inconsistent state needed to be addressed.A partitioned task modelThe old data model co-located everything about a workflow in one place. The rewrite separates workflow metadata from task and user data and gives each task its own record. The engine no longer loads the whole workflow graph to make a decision. It evaluates against a lightweight blueprint of the workflow and loads only the specific task data a decision needs. The practical result is a dramatically smaller memory footprint per evaluation and far less work to advance a workflow by one step.The 3.0 engine topped out around 2,500 tasks per workflow. The rewritten engine scales to 30,000 tasks per workflow, more than a 10x increase. Evaluation got faster too: in production, the rewrite delivered roughly a 40% reduction in p99 workflow-evaluation latency.Making competing updates safe using partitionsSeparating the data is only half the story. The engine also has to make competing updates to the same task safe, because a task can be written from more than one place at once. Picture a long-running task whose worker sends periodic updates to keep it alive, each one marking the task in-progress, before finally reporting it COMPLETED. If a stray in-progress update arrives just after that completion, reordered by the network, it could quietly revert a COMPLETED task back to IN_PROGRESS.The first version leaned on Cassandra lightweight transactions (LWT) to serialize these writes, but mixing LWT with ordinary writes is a known Cassandra anti-pattern. The two use different timestamping mechanisms, and combining them breaks the strict consistency Conductor needs.So the data model evolved away from locking entirely. Task state is split across two partitions: a pending partition for tasks still in flight, and a terminal partition that a task is written to exactly once, when it reaches a terminal state (COMPLETED, FAILED, etc). When the engine evaluates a workflow, it reads both and reconciles them in the application layer with one simple rule: a terminal state always wins over a non-terminal one, and between two writes of the same kind (two in-progress, or two terminal) ordinary last-write-wins applies. So in the race above, when the task is marked COMPLETED it lands in the terminal partition, and the late in-progress update loses on read.The net of the rewrite is reliable task lifecycle management, a lower memory footprint, faster evaluation, and far higher concurrent throughput.Removing the last synchronous bottleneck: async workflow evaluationA faster engine still left one structural problem. Workflow evaluation ran synchronously. When an update arrived, a new workflow or a task completing, the request itself evaluated the workflow, and to do that safely it had to take a lock on that workflow. Under load, concurrent updates to the same workflow raced for that lock. Only one won, and the rest were deferred to a background repair process that, in extreme cases, could take several minutes to catch up. The result was exactly the kind of unpredictable tail latency that hurts most under peak load.The fix was to decouple the trigger from the evaluation. Updates no longer evaluate inline. Instead they enqueue the workflow onto a Timestone exclusive queue, and a central asynchronous evaluator processes each workflow’s pending work in order. Because the queue is exclusive, only one worker ever evaluates a given workflow at a time, so the lock contention simply goes away.The results are most dramatic at the tail (p99 and p99.9), which is the point, since the tail is what users feel under load:Unsuccessful lock-acquire attempts, which had spiked to around 2,700 per interval under contention, drop to essentially zero with async evaluation.Controlling how workflows executeRun Strategies: native control over workflow concurrencyFor most of its life, Conductor would happily run as many instances of a workflow as you started, all at once. That’s the right default, but it left a gap. When several workflow instances act on the same entity, the same movie id, the same creative, the same script, there is no built-in way to serialize them or guarantee only one runs at a time. Teams that needed this had to build it themselves: custom locks, custom queues, custom coordination logic bolted onto their workflows. That’s extra complexity in every workflow that needs it, and a real source of operational risk.Run Strategies are a way to declare, on a workflow definition, how concurrent executions should be handled. There are two strategies today. Parallel, the default, applies no restrictions and behaves exactly as workflows always have. Sequential runs at most one workflow at a time per sequence key, with subsequent requests enqueued and started automatically, in order, as each running workflow finishes. A run strategy is declared on the workflow definition, and can be set directly in the Workflow SDK as a @RunStrategy annotation argument on the @WorkflowMethod.The key idea is the sequence key: two executions that share a sequence key run one after another, and executions with different keys still run in parallel. So a content-ingestion workflow keyed on movie id processes each title strictly serially while happily running many different titles concurrently, with no custom locking required.Beyond polled workers: managed, serverless tasksA task meant a polled worker: your application implements the worker, polls the Conductor server for work, runs it, and reports the result back. That model is simple and language-agnostic, and it’s still the backbone for most workloads. But it has a cost. You operate the worker fleet. You stand it up, keep it running, and scale it, even for work that’s bursty or runs rarely.So Conductor added managed tasks: built-in task types where Conductor drives the compute directly, with no long-lived worker for you to host or scale. Two integrations carry most of this today: Titus for containers and Stratum for serverless media-processing functions.A TITUS task launches a Titus job to run a script, whether shell, Python, or whatever your image packages, using Titus, Netflix’s container management platform, as the compute plane. You point the task at a Docker image and an entry point, and specify the shape of the run, such as cpu and memory. The lifecycle is fully managed. The task starts IN_PROGRESS, Conductor monitors the Titus job, and it transitions on the container’s exit behavior. An exit code of 0 marks the task COMPLETED, and a non-zero code, a timeout, or a missing job marks it FAILED. The upshot is that you can spin up containerized compute as a step in a workflow without running a worker that sits around polling.A STRATUM task invokes a Stratum function and collects its output. Stratum is the serverless function layer of Netflix’s Cosmos platform, purpose-built for media-processing workloads, and a Conductor STRATUM task is the natural way to call one from a workflow. You give it a function name and an ordered list of arguments. The task stays IN_PROGRESS until the function finishes, then resolves to COMPLETED, FAILED, or TIMED_OUT. Because Stratum exposes functions to Conductor directly, teams can fold resource-intensive media operations into a workflow without owning the compute that runs them.Polled workers, Titus jobs, and Stratum functions are three points on a spectrum, and a single workflow can mix all three: a long-lived worker for steady high-volume work, a Titus container for an occasional heavy batch step, and a Stratum function for media processing. Managed tasks shift the operational burden from the team to the platform.Scaling task polling: dynamic thread allocationThe engine work in the previous section made Conductor evaluate workflows fast. But there’s a second half to throughput: how quickly workers pick up the tasks that evaluation schedules. That’s a polling problem, and at scale it has its own failure mode, one that showed up most sharply for Studio Orchestrator, the heavy Conductor user behind the Media Production Suite pipelines from earlier in this post.Conductor clients poll for work using worker threads, and historically the number of threads was fixed per task type. That’s predictable, and it prevents any one task from starving the others, but it’s static. During burst traffic, when thousands of instances of a single task suddenly pile into a queue, a fixed thread count can’t ramp up to drain them. The symptom was distinctive: CPU on the instances stayed flat while queues backed up, and idle threads sat stranded on quiet queues unable to help the overloaded one.Allocating threads by real-time queue depthThe fix is a dynamic allocation strategy that rebalances threads across task queues continuously, based on how much work each queue actually has, while still guaranteeing every queue a floor so nothing starves.Each instance periodically polls Timestone for the current depth of every task queue it manages. It then splits its thread pool into two parts. A minimum allocation gives every queue a baseline number of threads, so no queue is ever fully starved. A floating pool holds the remaining threads, distributed across queues in proportion to each queue’s share of the total load. A queue with a deep backlog draws a large slice of the floating pool. An idle queue falls back toward its minimum, freeing threads for whoever needs them.One guardrail keeps this from becoming its own problem: a multiplier ceiling. A queue’s thread count can grow by at most a configurable factor per cycle, so even a dramatic load spike ramps up in controlled steps rather than all at once, giving downstream and dependent services time to absorb the increase instead of being bombarded. Allocations are recomputed every polling interval, so the system tracks shifting load in near real time while always honoring the per-queue minimums.Two things come out of this. First, because threads are allocated where they’re actually needed, each instance can safely run a much larger pool. Second, CPU utilization becomes an accurate signal of how busy an instance is, which means autoscaling on CPU makes the cluster scale out in response to real computational demand.Developer experience: Workflow SDKFor most of Conductor’s life, authoring a workflow meant writing JSON. You declared each task, then wired one task’s output into the next with ${taskRef.output.field} string templates, and on the worker side you pulled inputs back out of a Map and cast them to the types you expected. It worked, but it pushed an entire class of errors to runtime, and made large workflows genuinely hard to read.The Workflow SDK is our answer: a type-safe Java DSL for authoring Conductor workflows. You write Java code. The compiler and IDE catch mistakes before anything deploys, and a Gradle plugin translates your Java code into the JSON workflow definition the Conductor server understands.Crucially, the SDK gives you the best of both worlds. One of Conductor’s defining strengths has always been visibility. Because a workflow is a declared graph rather than imperative code, during execution the exact control flow can be inspected, every branch, loop, and parallel fork, and light up live as an execution moves through it. Writing workflows in imperative code gives you none of that. A timeline is not a workflow diagram. The SDK keeps both halves. You write in idiomatic Java and at execution time you still get the full visual graph: the same task-by-task flow, inputs and outputs, and at-a-glance “where is this stuck and why” that Conductor has always offered.The authoring pains it removesNo type safety. In the JSON model, a task input like “movieId”: “${awesomeTask2.output.movieId}” is just a string template. The worker reads it back with inputData.get(“movieId”) and casts. If the upstream value is actually an Integer and you cast to String, you get a ClassCastException at runtime. Fix that, and a subtle typo bites next: the JSON declares an input variable movieId but the worker reads inputData.get(movieid), which returns null and you get a NullPointerException downstream. None of this is visible until the workflow runs.public class GetMovieWorker implements Worker { @Override public TaskResult execute(Task task) { Map inputData = task.getInputData(); long movieId = (Integer) inputData.get("movieId"); // cast, fingers crossed // more logic here }}Verbosity. Defining a couple of tasks is already wordy. A real workflow’s JSON balloons into hundreds of lines that are difficult to review or refactor. A complex branching workflow becomes a single large JSON blob that no one wants to touch.{ "name": "wrap_movie_workflow", "version": 1, "semver": "0.0.1", "schemaVersion": 2, "restartable": true, "workflowStatusListenerEnabled": false, "ownerEmail": "team@netflix.com", "timeoutPolicy": "ALERT_ONLY", "timeoutSeconds": 0, "tasks": [ { "name": "get_movie", "taskReferenceName": "movie", "type": "SIMPLE", "inputParameters": { "movieId": "${workflow.input.movieId}" } }, { "name": "wrap_movie", "taskReferenceName": "wrapMovie", "type": "SIMPLE", "inputParameters": { "movie": "${movie.output}" } } ]}How the Java DSL works@TaskMethodpublic Movie getMovie(long movieId) { /* real worker logic */ }@TaskMethodpublic WrappedMovie wrapMovie(Movie movie) { /* real worker logic */ }@WorkflowStub(taskNames = {"movieapp.getMovie", "movieapp.wrapMovie"})public interface MovieWorkflows { @WorkflowGenerated(name = "movieapp.getMovie") Movie getMovie(int movieId); @WorkflowGenerated(name = "movieapp.wrapMovie") WrappedMovie wrapMovie(Movie movie); @WorkflowMethod(name = "wrap_movie_workflow", ownerEmail = "team@netflix.com") default WrappedMovie wrapMovieWorkflow(MovieInput input) { Movie movie = getMovie(input.getMovieId()); // long in, Movie out (checked) return wrapMovie(movie); // Movie in, WrappedMovie out (checked) }}The DSL has two core building blocks. The first, @TaskMethod, is a method that is executed at runtime and can contain arbitrary code. Crucially, it takes and returns real types: a getMovie task can take a long movieId and return a Movie object directly, with no maps and no casts. The second, @WorkflowMethod, is a method that is not executed and cannot contain arbitrary code. It expresses the workflow as ordinary method calls (call getMovie(…), then pass the result to wrapMovie(…)), and a Gradle build plugin parses it into a Conductor JSON definition at build time. The server does not see the Java code. It gets the same JSON it always has.Code generation is central to how this works. In the @WorkflowStub annotation you declare the task and workflow names you want to use, and at build time the Gradle plugin generates their input and output types along with typed stub methods that you call inside a @WorkflowMethod. Those generated stubs are what let the method body compile against real signatures with full type safety. Because the stubs are generated from names rather than shared code, teams can reuse each other’s tasks and workflows without sharing libraries.Real control flow, expressed in JavaThe examples above are linear, but most production workflows branch, loop, and fan out. The DSL expresses these as the Java constructs you’d reach for anyway, and the SDK parses them into the corresponding Conductor operators. A useful mental model: you’re describing a workflow graph in Java, not running Java, so the parser only supports the constructs that map cleanly to Conductor (if/else, switch, do-while, parallel forks, sub-workflow calls) and asks you to express a few things its way. Iteration, for instance, is always a do-while, since there are no for or while tasks in Conductor.A common shape is to kick off work, poll until it settles, then branch on the outcome, which is a do-while loop wrapped around an if/else:@WorkflowMethod(name = "process_asset_workflow", ownerEmail = "team@netflix.com")default void processAssetWorkflow(AssetInput input) { JobStatus status = submitJob(input); do { // loops are do-while status = checkStatus(input.getJobId()); } while (status.getState() == "PENDING"); if (status.getState() == "SUCCEEDED") { // branch on the result publish(input); } else { flagForReview(input); }}Independent work runs concurrently with ParallelExecution.fork(…) plus allOf(…) (a FORK and JOIN), and any other @WorkflowMethod can be called as a sub-workflow just by listing it in workflowNames:@WorkflowStub( taskNames = {"movieapp.getMovie", "movieapp.enrichMovie", "movieapp.getArtwork", "movieapp.refreshCache", "movieapp.publishPage"}, workflowNames = {"movieapp.localize_page_workflow"})public interface MoviePageWorkflows { // stubs not shown for brevity @WorkflowMethod(name = "prepare_movie_page", ownerEmail = "team@netflix.com") default void prepareMoviePage(MoviePageInput input) { // three independent branches, run in parallel Fork details = ParallelExecution.fork(() -> { Movie movie = getMovie(input.getMovieId()); return enrichMovie(movie); }); Fork art = ParallelExecution.fork(() -> getArtwork(input.getMovieId())); Fork cache = ParallelExecution.fork(this::refreshCache); ParallelExecution.allOf(details, art, cache); // FORK_JOIN and JOIN WrappedMovie enriched = details.get(); Artwork artwork = art.get(); LocalizedPage localized = localizePageWorkflow(input); // sub-workflow call publishPage(enriched, artwork, localized); }}Sharing tasks and workflows across applicationsTeams shouldn’t re-implement common tasks. Listing another team’s task or workflow in your @WorkflowStub is enough to call it like a local method, or invoke it as a sub-workflow, and the shared POJO types (a Movie, say) come along directly, with no copy-pasting between codebases.Feature parity and rapid adoptionThe SDK is gaining rapid adoption and has become the primary choice for authoring new workflows at Netflix. We continue to invest in the developer experience, with intelligent IDE plugins, better debugging, and an integrated testing framework so that testing a workflow in code feels like testing any other code.In production: Netflix AdsNetflix Ads runs complex, multi-step processes on Conductor with the Workflow SDK, and three examples show the range. Creative ingestion is a heavily branching pipeline where the SDK turned a sprawling JSON blob into type-safe, testable code, and a sequential run strategy eliminated a duplicate-processing race by serializing work per creative. Advertiser certificate management automates a previously manual, hard-to-scale process across several system integrations, with downstream automation when certificates expire. And Data Clean Rooms are privacy-preserving environments where Netflix and a partner run joint analysis on shared data, built on the SDK from day one. Provisioning a clean room is a multi-day sequence (create at a vendor, wait for the partner to join, activate, and eventually archive) that previously relied on hand-rolled async polling. Re-authored as Conductor workflows, each phase became a durable, observable state machine with automatic retries, state that survives restarts, and long partner waits that yield rather than hold a thread.The common thread is that the SDK made multi-process, multi-integration workflows easy to author and maintain, and let teams lean on Conductor’s reliability and visibility instead of hand-rolling retries and distributed locking.What’s nextLike the last time we wrote about Conductor, the road ahead is less a fixed roadmap than a wish-list, a sense of where we’re headed, shaped by what teams keep asking for.On scale, the demands keep growing. As Netflix expands into new content types, like live, games, and podcasts, we anticipate our workflow load could grow 5x in the coming year, and the engine work described above is what gives us confidence we can absorb it without re-architecting again.On developer experience, the Workflow SDK now covers all task types, so the focus shifts to the experience around writing workflows: a richer integrated testing story so a workflow can be validated against real worker behavior as easily as any other code, deeper agent tooling such as skills, plugins, and MCPs, and safe schema evolution: a schema registry for shared tasks and workflows, with guardrails that block non-backward-compatible changes so they can evolve without breaking the teams that depend on it.On execution control, sequential is only the first run strategy. A family of richer policies is on the roadmap, including running N at a time per key, idempotent de-duplication, keep-first and keep-latest, and combinations of these, to cover the coordination patterns teams currently hand-build. And managed tasks will keep expanding the ways Conductor can run compute directly, beyond today’s container and serverless function integrations.None of this is exhaustive, and priorities shift with the workloads we take on. But the direction is consistent: keep the overhead low, keep authoring safe and simple, and keep widening the range of work a single workflow can reliably coordinate.AcknowledgmentsNetflix Conductor is built and operated by the Enterprise Workflow Platform team, with contributions from many partner teams across the company.Andrew Seier, Anoop Panicker, Christian Johnson, Christine Yu, Jasmine Lee, Jiaofen Xu, Michael Torku, Peter Lau, Prashant Rastogi, Ryan Kastilani Surafel Korse and Victor Lee have contributed to Conductor. We thank Charles Zhao, Sumukh Shivaprakash and the teams in the Content and Business Products and Ads organizations for their guidance and constructive feedback while developing and operating Conductor.Netflix Conductor: The Next Chapter was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Conductor is a key tool for orchestrating microservices, and its next chapter affects how teams design resilient workflows.
|
|
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
Amazon EventBridge announces an enhanced custom event bus for organizations scaling event-driven applications across teams and accounts. Deploy a single event bus shared across all AWS accounts in your organization, with ordering guarantees, a simplified Subscriber resource, and improved economics at scale.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
It simplifies multi-account event architectures, a common pain point for enterprises scaling microservices.
|
|
|
|
☁️ Enterprise Tech
via Vercel News • 1 min read
|
Core web app decisions tend to center the backend, due to its complexity and impact over huge swaths of the business.However, frontends have grown far more important and complex in their own right. When not prioritized, the intricate infrastructure around them can quickly spin out of control, dragging teams into untold amounts of tech debt.As decoupled architecture becomes more common, developers are turning to the Frontend Cloud to automate away the behind-the-scenes hassles of creating and growing dynamic websites.Instead of managing infrastructure as a separate step of the development process, the Frontend Cloud provisions global infrastructure for you, based on your existing application code. This approach to web development massively increases developer velocity, allowing your team to experiment safely and meet shifting market demands. Teams of all sizes can effortlessly scale global apps while maintaining the highest possible bars for performance, personalization, and security.You can think of the backend cloud as your cost center and the Frontend Cloud as your profit center. Read more
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Frontend complexity is exploding, and this gives CTOs a blueprint for building resilient, scalable frontend infrastructure.
|
|
|
🔐 Security
|
|
🔐 Security
via Vercel News • 3 min read
|
Notion Workers let you write and deploy code to give Custom Agents new powers: sync external data, trigger automations, call any API. With Workers, developers can build agents that sync CRM data on a schedule, open issues when error rates spike, and turn Slack threads into formatted content. Under the hood, every Worker runs on Vercel Sandbox.The problem: safely running code from any developer or agentNotion wanted to let anyone extend their platform with custom code. That's a hard infrastructure problem, but an even bigger security problem. Every Notion Worker runs arbitrary code generated by a third-party developer or agent, on behalf of a Notion user, potentially inside an enterprise workspace.Without proper isolation, a Worker would run in the same environment as the Custom Agent, with access to its secrets, permissions, and everything else in that execution context. A single prompt injection could exfiltrate credentials or access another user's data.The requirements were clear:Hard isolation: One Notion Worker can never access another's data or stateCredential security: Notion Workers need API keys to talk to external services, but those secrets can never be exposed to the code itselfNetwork controls: Enterprise customers need guarantees about the external services a Worker is allowed to reachScale: Workers need to support millions of users running concurrent executions without performance degradationState preservation: Workers need fast cold starts, which require the ability to snapshot and restore filesystem stateEconomics: A billing model that is built for agents with low CPU utilization ratesWhy Vercel SandboxVercel Sandbox runs each Notion Worker in an ephemeral Firecracker microVM. Every VM boots its own kernel, providing stronger isolation than containers. Each execution gets its own filesystem, its own network stack, and its own security boundary. When the Notion Worker finishes, the microVM is either destroyed or snapshotted for later retrieval.To support workloads like Notion Workers at scale, Vercel Sandbox provides several critical capabilities:Credential injection. Sandbox's firewall proxy can intercept and inject API keys into outbound requests at the network level, so credentials never enter the execution environment. For agent-driven workloads, this eliminates the most dangerous prompt injection vector: an agent being tricked into exfiltrating secrets. (We wrote about this architecture in depth in security boundaries in agentic architectures).Network policies. Sandbox supports dynamic network policies that can be updated during runtime without restarting the process: start with internet access to install dependencies, then lock down egress before running untrusted code. Platform builders can pass these controls through to their own customers.Snapshots. Install dependencies once, snapshot the filesystem state, and resume from that snapshot on subsequent invocations. Combined with active-CPU billing, where CPU costs only accrue when your code is actually executing, not waiting on I/O, this keeps costs predictable as usage scales.The bigger picture: Notion as a developer platformNotion Workers isn't a one-off feature. It's the beginning of Notion becoming a developer platform.This shift requires infrastructure that Notion shouldn't have to build. Secure code execution, credential management, network isolation, file-sytem based snapshotting: these are hard problems that compound as the platform scales. Vercel Sandbox handles the infrastructure complexity so Notion can focus on the developer experience. What developers are building with Notion WorkersNotion Workers support three main patterns: third-party data syncing, custom automations, and AI agent tools. Developers use them to sync external data, such as CRM records, analytics, and support tickets, into Notion on a schedule. A Worker can also be attached to a button, triggering arbitrary code with a single click. And when Notion's custom agents invoke Workers as tool calls, they become far more capable than agents limited to pre-built integrations.Extend your platform with Vercel SandboxNotion Workers requires the same capabilities as other agent platforms. Any platform that wants to let users or agents run custom code faces the same set of problems: isolation, credential security, network controls, and scale.Vercel Sandbox provides these as capabilities out of the box. If you're building a platform that needs to run untrusted code, whether for AI agents, developer plugins, or workflow automation, then this is how you do it. Read more
|
💡 WHY THIS MATTERS FOR YOUR ROLE
As agents and user code become common, secure execution environments are a must-have for any platform team.
|
|
|
|
How was today's dispatch?
Your 1-click feedback tunes our story selection algorithms for tomorrow.
|
|
🎁 VIP REFERRAL PROGRAM
Invite Colleagues, Unlock Executive Intelligence Pro
You have referred 0 colleagues.
Share your personal link to climb the reader leaderboard:
https://brief.delights.pro/join?ref=VIP
|
|
Share Your VIP Invite →
|
|
|
|
|
|
|
|
