|
|
|
|
KNOWLEDGE, REFINED • Builders 🔧 • September 27, 2026
|
|
|
📊 Scanned 6,462 articles • Selected top 10 role-critical insights for today.
|
|
|
🤔 CONTRARIAN TAKEAWAY
The other side of ai market outlook
496 articles say ai boom, but 108 argue ai bubble.
|
|
🚀 AI & Innovation
|
|
🚀 AI & Innovation
via Vercel News • 9 min read
|
AI SDK 7 is a major release for building production agents in TypeScript. The SDK has grown from model calls and chat primitives into a broader agent platform for developing, running, integrating, and observing agents across text, audio, realtime, image, and video. Every major provider is supported out of the box.At a GlanceDevelop agents with reasoning control, tool and runtime context, provider files and skills support, MCP Apps, and a terminal UI.Run agents with tool approvals, durable WorkflowAgent execution, first-class timeouts, and sandbox support.Integrate any agent harness such as Codex, Claude Code, Deep Agents, OpenCode, or Pi.Observe agent behavior with redesigned telemetry, @ai-sdk/otel, Node.js tracing channel support, lifecycle callbacks, and step performance statistics.Build beyond text with stable speech/transcription APIs, richer file parts, image generation and editing, multimodal embeddings, reranking, realtime voice (experimental), and video generation (experimental).Upgrade with clearer APIs through the new migration skill, codemods, and migration paths for ESM, Node.js 22, instructions, telemetry, stream, finalStep, runtimeContext, and toolsContext.Before You UpgradeAI SDK 7 introduces two breaking requirements:Node.js 22 minimum: Node 22 is required because the SDK depends on APIs (including the native fetch implementation and improved AsyncLocalStorage semantics) that are not backported to earlier LTS lines.ESM imports required: AI SDK 7 requires ESM imports (import syntax or .mjs files). CommonJS require() is not supported. Update your package.json to include "type": "module" or migrate individual files to .mjs.Run the v7 codemods to automate most import and rename changes before manually reviewing semantic migration items. See the full v7 migration guide.Develop AgentsProvider-agnostic reasoning control: generateText and streamText now support a top-level reasoning option that maps to provider-native settings across OpenAI, Anthropic, Google, Groq, xAI, Bedrock, Fireworks, DeepSeek, Open Responses, and OpenAI-compatible providers. Note that exact behavior and available parameters vary by provider.Typed runtime context: Shared orchestration state lives in runtimeContext and flows through prepareStep, approval functions, lifecycle callbacks, telemetry, ToolLoopAgent, and WorkflowAgent.Scoped tool context: Tools can declare a contextSchema, and callers provide per-tool values through toolsContext, so third-party tools only receive the secrets or config they need.Provider file uploads: uploadFile uploads large inputs once and reuses provider references in later calls, reducing redundant uploads in repeated PDF, dataset, image, and multi-step workflows.Provider skill uploads: uploadSkill brings the same pattern to provider-managed skill environments.MCP Apps: MCP support now includes model-visible versus app-only tools, app metadata, sandboxed iframe rendering, and JSON-RPC communication for tools, resources, logs, and display updates. Add an MCP app to your agentic app.Terminal agent development: @ai-sdk/tui runs AI SDK agents in an interactive terminal UI, letting you test reasoning, tools, and markdown output before building a full app. Learn how to build your own terminal agent.Run Agents in ProductionAI SDK 7 adds the primitives agents need once they leave local demos.Tool approvals: generateText, streamText, and ToolLoopAgent can define approval policies at the call or agent level. Policies can require user approval, auto-approve, auto-deny, or delegate to typed approval functions.Hardened approval replay: Higher-risk approval flows can revalidate tool inputs and policies before continuation, use WorkflowAgent approval validation, and opt into HMAC signing. HMAC signing cryptographically binds the original tool inputs to the approval token, preventing tampering with tool arguments between the approval request and resumption.Durable execution: @ai-sdk/workflow introduces WorkflowAgent for long-running agents. Execution state is persisted to durable storage between steps, so agents survive deploys, process restarts, interruptions, and delayed approvals. To learn more, see the WorkflowAgent migration guide.Workflow-aware agent features: WorkflowAgent supports streaming, tools, approvals, typed runtime and tool context, lifecycle callbacks, stable telemetry, provider-executed approval resumption, and tool result conversion.First-class timeouts: Text generation and agent APIs can define total, per-step, per-chunk, default tool, and per-tool timeout budgets. Timeout aborts use TimeoutError, and abort reasons flow through stream and UI protocols.Sandboxed execution: The sandbox abstractions support command execution, streaming output, working directories, environment variables, abort signals, and step-level sandbox overrides.Integrate Agent HarnessesAI SDK 7 introduces a harness layer for bringing established agents into the AI SDK ecosystem. Wrap harnesses such as Claude Code, Codex, Deep Agents, OpenCode, and Pi behind the same agent interface used by the rest of the SDK.HarnessAgent: Run external agent harnesses through the AI SDK Agent interface, with standard generate and stream results.Adapters for agents: Claude Code, Codex, Deep Agents, OpenCode, and Pi harness adapters let teams plug existing agent runtimes into AI SDK apps.Configurable harness runs: Harness agents can receive sandboxes, instructions, custom skills, and tools, so the same runtime can be shaped for different products and workflows.Durable, resumable sessions: Workflow utilities, session bridging, and APIs for interrupted-turn continuation make harness runs suitable for longer tasks.Gateway-ready authentication: Harness adapters support Vercel OIDC for AI Gateway, simplifying hosted and sandboxed agent execution.Observe AgentsGlobal telemetry integrations: Register telemetry once and receive structured events across model calls, steps, tools, embeddings, reranking, and agent execution.Dedicated OpenTelemetry package: OpenTelemetry support now lives in @ai-sdk/otel, with GenAI-semantic convention spans and metrics, supplemental AI SDK attributes, and span-enrichment hooks. To learn more, see the @ai-sdk/otel docs.Node.js tracing channel: AI SDK 7 emits structured telemetry through the Node.js tracing channel, allowing observability providers to subscribe once while preserving async context across streaming and tool execution.Sensitive context controls: Runtime and tool context can be deliberately included in telemetry, with controls to prevent secrets from being exposed by default.Lifecycle callbacks: Callbacks are more consistent across core functions, agents, tools, embeddings, reranking, structured output, and UI streams. Callback payloads carry richer data for step, usage, content, file, source, warning, tool, model-call, and error events.Performance statistics: Step results expose timing and throughput metrics, including response time, total step time, tool execution time, time to first output, and output tokens per second.Build Beyond TextAI SDK 7 expands the SDK across realtime, video, speech, transcription, images, files, embeddings, and structured output.Realtime (experimental): Browser-to-provider WebSocket sessions for OpenAI, Google, and xAI, with audio/text conversations, client-driven tool calls, and normalized routing through AI Gateway.Video generation (experimental): Video generation works across AI Gateway, Google AI Studio, Google Vertex, fal, Replicate, ByteDance Seedance, Kling AI, Prodia, and xAI, with support for long-running SSE responses and safer bounded downloads.Stable speech and transcription: generateSpeech, transcribe, SpeechResult, and TranscriptionResult are stable exports.Images as files: Image parts move toward the same canonical file model as other media. Tool outputs can use a single file shape for inline data, URLs, provider references, and text-backed content.Richer media and model support: Providers add image generation, image editing, multimodal embeddings, speech, transcription, reranking, file references, reasoning files, and provider-specific media metadata.Structured output reliability: JSON Schema post-processing is stricter for Zod and Standard Schema inputs, malformed JSON extraction and repair is available for structured outputs and tool calls, and array output mode preserves transforms, coercions, defaults, and pipes.UI, Streams, and Message HandlingThe UI and streaming work in v7 focuses on making agent streams correct, composable, and reliable.Direct agent transport: DirectChatTransport can call an Agent directly from UI code.Tool approval in UI flows: UI messages support automatic approval responses and improved approval replay behavior.Framework improvements: React useChat callbacks update with current props/state; sendAutomaticallyWhen supports async conditions; Vue gains an idiomatic useChat composable; and Angular APIs are aligned with current AI SDK patterns.More reliable streams: Provider streams that end before a finish chunk are treated as errors, tool execution errors are emitted and cleaned up predictably, and streaming reasoning edge cases are handled more consistently.Provider metadata preservation: Provider metadata is retained across text generation, UI streams, tool invocations, and multi-turn provider ID mapping.Multi-step results: Top-level content, tool calls/results, files, sources, warnings, and usage now represent the full run. Final-step-only details are available through finalStep.Configure MCPThe MCP package grows from a tool transport layer into a richer integration surface for agent tools and app UIs.Protocol and metadata: MCP clients support protocol version 2025-11-25, server metadata, server instructions, ping responses, negotiated protocol headers, and public listTools().Typed tool outputs: MCP tools can expose outputSchema and structuredContent, and tool definitions can be separated from executable tools.Resource content: Tool results and prompt messages can include MCP resource_link content.App rendering: MCP Apps use tool metadata to render app-specific UI inside sandboxed iframes while keeping model-visible and app-only tools separate.Transport reliability: HTTP, SSE, and OAuth transports support custom fetch, redirect configuration, OAuth refresh deduplication, state validation, asynchronous client authentication, richer errors, and better SSE frame handling.Configure Runtime and PackagingNode.js 22 minimum: AI SDK packages require Node.js 22 or later.ESM imports required: AI SDK 7 requires ESM imports (import syntax or .mjs files). Update your package.json to include "type": "module" or migrate individual files to .mjs.Migration skill available: A dedicated migration skill is available that developers can install and ask their agent to use for AI SDK v6-to-v7 upgrades. Codemods available: The v7 codemods cover the majority of renames and cleanup migrations.Send this prompt to your AI coding agent to get started:Coming Out of ExperimentalThe following highlights cover the most impactful promotions:experimental_customProvider becomes customProviderexperimental_generateImage becomes generateImageexperimental_output becomes outputexperimental_prepareStep becomes prepareStepexperimental_telemetry becomes telemetryRenamed APIsThe following highlights cover the most impactful renames:system option becomes instructionsSystem messages inside prompt or messages require allowSystemInMessages: trueonFinish becomes onEndStreamTextResult.fullStream becomes streamCallSettings is split into model generation options and request/transport optionsDeprecated APIsThe following highlights cover the most impactful deprecations:Tool approval: needsApproval on tool() and dynamicTool() is deprecated. Move approval logic to toolApproval on generateText, streamText, or ToolLoopAgent.Stream response helpers: Result methods like result.toUIMessageStreamResponse() and result.toTextStreamResponse() are deprecated. Use top-level helpers like createUIMessageStreamResponse and createTextStreamResponse.Vue chat: The Vue Chat class is deprecated. Use the useChat composable instead.Other Migration ThemesReasoning configuration is centralized: The top-level reasoning option replaces overlapping provider-specific reasoning settings unless a provider-specific override is intentional.OpenTelemetry moved to @ai-sdk/otel: OpenTelemetry span collection is no longer built into the ai package. Telemetry is registered globally, and custom tracers move into the OpenTelemetry constructor.Request and response body retention is opt-in: Text generation results exclude request and response bodies by default.Multi-step results now represent the full run: Top-level usage, content, tool calls/results, files, sources, and warnings accumulate across all steps; final-step-only data lives under finalStep.Message parts are more canonical: Legacy media and image-specific parts move toward file parts with media types.Package-specific behavior changed: MCP HTTP/SSE redirects are treated as errors, OpenAI Responses reasoning summaries default to detailed, and Anthropic cache creation token metadata moves into standard usage fields.Upgrade PathFollow these steps to migrate an existing project to AI SDK 7:Update Node.js to 22+: Verify your runtime and CI environments meet the minimum requirement before upgrading packages.Update packages: Bump ai and all @ai-sdk/* packages to their v7 releases in your package.json.Run the v7 codemods: The codemods automate the majority of renames, import changes, and API moves. Review the diff before committing.Migrate OpenTelemetry: If you use tracing, move your setup to @ai-sdk/otel and register telemetry globally. Refer to the @ai-sdk/otel docs and the OpenTelemetry span schema for attribute details.Review semantic changes manually: Codemods cannot fully decide runtime requirements, ESM imports, instruction/message behavior, runtime/tool context separation, approval policy placement, stream helper usage, and multi-step result shapes. See the full v7 migration guide.For a guided migration, install the AI SDK v7 migration skill with npx skills add vercel/ai --skill migrate-ai-sdk-v6-to-v7. Read more
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders building agent products should evaluate whether a unified SDK can accelerate delivery without creating new provider lock-in.
|
|
|
|
🚀 AI & Innovation
via The GitHub Blog • 1 min read
|
A rewrite this size wasn't affordable before agents. Here's what porting the Copilot agent runtime to 800,000 lines of production Rust actually took. The post Migrating the GitHub Copilot runtime to Rust, using Copilot appeared first on The GitHub Blog.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders evaluating agent-assisted modernization can use GitHub's migration as a concrete benchmark for scope, risk, and productivity.
|
|
|
|
🚀 AI & Innovation
via AWS News Blog • 1 min read
|
Learn how Amazon CloudWatch Omni delivers AI-powered observability purpose-built for generative AI and agentic workloads. Trace, evaluate, and experiment with AI agents across any framework—directly from your IDE or a standalone web experience—using open standards and built-in evaluators for quality, correctness, and coherence.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders shipping agents need reliable evaluation and tracing primitives before autonomous behavior becomes an operational blind spot.
|
|
|
☁️ Enterprise Tech
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
DynamoDB now supports native vector search with single-digit millisecond latency at 99%+ recall. It is designed for any scale, even trillions of vectors and requires zero infrastructure management.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders can reassess their vector database architecture and reduce operational overhead for AI-powered applications.
|
|
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
Amazon EventBridge announces an enhanced custom event bus for organizations scaling event-driven applications across teams and accounts. Deploy a single event bus shared across all AWS accounts in your organization, with ordering guarantees, a simplified Subscriber resource, and improved economics at scale.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders scaling event-driven architectures can simplify cross-account integration without multiplying buses and ownership boundaries.
|
|
|
|
☁️ Enterprise Tech
via Netflix TechBlog - Medium • 10 min read
|
Samuel Yeboah, Francesco Di Chiara and Mingliang LiuToday, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands.Why autoscaling is not optional at our scaleNetflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers.Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything?The first autoscaler: watching from outsideOur first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on Mantis, consuming a live feed of cluster-level metrics from Atlas, our telemetry platform, including CPU, network, Kafka lag, input-rate, and consume-rate signals for every job. The scaler combined lag-derived catch-up time, CPU/network utilization thresholds, observed performance history, and regression over recent input rate to decide when to scale up or whether a smaller cluster could handle the lookahead window. Because the autoscaler operates independently of the Flink platform, it remains unaffected by issues within Flink itself. Building it as a streaming job also made it easy to scale. Each autoscaler node handled the metrics for a subset of Flink jobs, and we never had to write custom sharding or coordination logic to keep up with a growing Flink fleet. It reliably cut resource usage by 25–45% across thousands of managed pipelines. Check our previous talk at Flink Forward 2020.But watching from outside has a ceiling. The system reasoned about a whole cluster through coarse container metrics, and it scaled a single knob, the total TaskManager count, so every operator in a job moved together. That fit the simple, single-operator pipelines it was built for, but not the multi-operator, stateful DAGs that teams were increasingly bringing to us for Ads, recommendations, and games. Those were exactly the jobs it could not reason about, and supporting each new case meant more custom logic rather than any general capability.The autoscaler is only as good as the metrics served by external systems beneath it. Those metrics could miss real trouble: a job could be completely busy without any of it showing up as CPU utilization, leaving the job stuck in a degraded state the scaler had no way to see. Recently a networking migration quietly changed how some traffic was reported, and a subset of the Atlas metrics the scaler relied on stopped capturing everything accurately. The gap stayed invisible until it surfaced in production much later.It was time to reconsider build versus buy.The second autoscaler: reasoning from insideWhen we started, the Flink community had no mature autoscaler to offer. By the time we re-evaluated, it did: the Apache Flink Autoscaler. Instead of watching containers from outside, it reasons from inside the job.Its key idea is to estimate each operator’s true processing rate (TPR): the throughput it could sustain if it were fully busy. Flink reports, per subtask, the fraction of each second spent doing actual work, separate from time spent backpressured or idle. Dividing observed throughput by that busy fraction extrapolates capacity to full utilization: an operator handling 700 records/sec while busy 70% of the time has a TPR of 700 / 0.7 = 1,000 records/sec. Starting from the sources, the autoscaler walks the job graph and uses each operator’s TPR, its input/output ratios, and a target utilization to compute the parallelism every vertex needs so that no operator becomes the bottleneck, rather than resizing the whole cluster as a unit.The two approaches make a different contract, summarized below.The decisive difference for us is the last two rows: the OSS autoscaler can scale exactly the stateful, multi-operator jobs our homegrown system could not, and it lets each job carry its own configuration — stabilization periods, thresholds, and other scaling behavior tuned to the workload. That made it the natural fit for the custom jobs teams had been scaling by hand.Making it work at Netflix scaleAdopting the algorithm was straightforward; the community had done the hard part. The work for us was running it reliably across our own jobs, and this is where our system differs most from the stock open-source deployment.Firstly, the OSS autoscaler was originally architected to reside within the Kubernetes Operator for Flink, but our Flink platform runs on its own control plane, not that operator (see our previous talk at Current Conference 2024). The community later made a fantastic decision to keep the core logic as a standalone library. They refactored four generic interfaces that made it easy to plug directly into our internal ecosystem: a context carrying job metadata and REST API info, a state store, an event handler, and a realizer that applies scaling decisions.That service is a Spring Boot application whose orchestration runs on Temporal, the durable workflow engine. An orchestrator workflow polls our Flink control plane about once a minute for the jobs with autoscaling enabled, and starts one long-running workflow per job. Each per-job workflow pulls that job’s per-vertex metrics from its Flink JobManager, runs the OSS evaluation algorithm, and, when a scaling decision results, hands it to a realizer that actuates the change through our Flink control plane.The workflow-per-job design was a direct response to pain. We first ran evaluations in a single batch loop over the whole set of jobs, and it was fragile: one slow or misbehaving job could stall metric collection and scaling for every job behind it. Giving each job its own durable workflow isolated that blast radius, so a single problematic job now fails and retries on its own, and the runtime scales out as we onboard more jobs.Secondly, three engineering gaps stood between “works in community” and “works at Netflix scale”:Metric collection at high parallelism. On big jobs, pulling metrics from the JobManager became a bottleneck, and part of the cause was in Flink’s runtime. To address that, we changed the JobManager to cache transient metric names and clean them up once instead of rescanning on every fetch, and we added server-side filtering so the autoscaler asks only for the metrics it needs. This let the autoscaler work on jobs up to 3,000 Flink subtasks, where it had previously struggled above roughly 1,000. Those are in our internal fork of Flink release, while some are contributed upstream such as FLINK-36172.Preserving forward chaining. Two separate vertices joined by a forward connection must run at the same parallelism, because records are handed over in memory on a fixed local channel. Scale one of them alone and Flink does not fail; it silently converts that edge into a network shuffle. Our fork detects forward-connected subgraphs and scales each as a unit.Respecting sink limits. Some sinks have finite write capacity, so we added detection for async-sink backpressure (also a fork change) to keep the autoscaler from scaling a job up into a sink that cannot absorb more.Before it actuates anything, the realizer runs a set of safety checks. For example, it refuses to scale a job down in a region being evacuated during a company-wide region failover. It also verifies there is enough disk for the new cluster to hold the job’s checkpoint state, and it adds a small standby buffer for larger clusters.The road to one autoscalerLast year, the OSS-based autoscaler achieved general availability for custom jobs at Netflix, yielding promising initial outcomes. For instance, our client telemetry and logging team achieved a 58% reduction in its annualized Flink compute expenditures, saving approximately $1.1 million annually. This efficiency is driven by three key factors. First, whereas static provisioning must always account for peak loads, autoscaling dynamically adapts to daily cycles, capturing the drop in traffic during nights and weekends compared to weekday peaks. Second, rather than relying on teams to manually optimize resources following performance improvements or post-holiday slowdowns, the autoscaler continually adjusts capacity. Finally, adopting uniform container dimensions enables superior bin-packing and more granular scaling increments.Additionally, scaling down too eagerly is its own trap. Cut too deep and CPU saturates, lag spikes, and the system cannot react instantly because its metric window and stabilization period have to rebuild after each restart. We now run a target utilization of 0.45, below the community default of 0.7, deliberately trading a little efficiency for stability. Fewer and calmer rescales are worth the marginal cost for large stateful jobs.While our scaler provides fine-grained signals and vertex-level decision units for stateful DAGs, fast rescaling still heavily depends on Flink Core’s state restoration performance. Today, the biggest remaining cost in scaling a stateful job isn’t the scaler’s logic — it’s the restart and state recovery process itself. Flink 2 addresses this through its disaggregated state architecture, keeping state in external storage rather than on local disk, which can sharply reduce how much a rescale or recovery depends on total state size. Having started supporting Flink 2.2 at Netflix, we plan on experimenting with this new state backend to see if it can help eliminate state recovery bottlenecks when scaling large stateful jobs.Looking ahead, we aim to migrate all internal scaler use cases onto the new one based on OSS autoscaler to simplify our operational surface area.Key TakeawaysAlong the way, three lessons that generalize beyond Flink:Metric choice matters more than algorithm sophistication. Our most useful debugging was rarely about the scaling math; it was about which signal to trust most. Understand your metrics before you tune your algorithm.Set sensible defaults, but leave room to tune. Our managed jobs are similar enough that one good default covers most of them untouched, which is the point of a platform. But forcing a single configuration on every job punishes the ones that do not fit, so we pair defaults with per-job overrides and deliberately hide the knobs that need deep expertise. Most teams should never have to think about the autoscaler.Adopt, then extend. We built in-house because in 2019 nothing mature fit our platform. When a strong community project appeared, the right move was neither to defend our investment forever nor to rip it out overnight, but to adopt it for new workloads, contribute fixes back, and plan a deliberate migration.Thanks to the Flink and Data Mesh teams for the control-plane changes this work depended on, to the Temporal team and our early pilot teams, and to the Apache Flink autoscaler maintainers whose foundation we built on. Special thanks to Andy Zhang, Calvin Cheung, Daniel Trager, Guil Pires, Mark Cho, Matthew Kornitsky, Nikhil Sulegaon, Sujay Jain, and Tom Lee.A Tale of Two Flink Autoscalers was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders running streaming platforms can apply Netflix's lessons to reduce autoscaling complexity and improve resource efficiency.
|
|
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
Amazon EC2 R9g and R9gd instances powered by AWS Graviton5 are now generally available, delivering up to 25% better compute performance than R8g, ideal for databases, in-memory caches, and real-time analytics.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders making cloud capacity decisions should benchmark Graviton5 before renewing or expanding compute fleets.
|
|
|
🔐 Security
|
|
🔐 Security
via Netflix TechBlog - Medium • 9 min read
|
By Dhruv PratapIntroductionOrganizations that have been around for a while usually run two identity systems side by side. One belongs to the cloud provider: IAM roles, instance profiles, execution roles. The other is your own, and it is the one your internal services actually check when they decide whether to answer a request.On infrastructure you build yourself, you can bootstrap your own identity however you like. On managed compute you cannot. The provider hands your process a cloud identity and nothing else.This post describes how we close that gap for Apache Spark workloads running on Amazon EMR. A workload that starts with only an AWS identity has to end up holding a first-class internal identity. Doing the exchange is straightforward. Making it trustworthy is the part that took the design work. Very little of what follows is specific to Spark or to EMR.Two identity systemsInternal service-to-service authentication at Netflix runs on a private PKI called Metatron. Every workload gets a short-lived X.509 certificate, and services authenticate each other with mutual TLS. A workload cannot simply ask for a certificate. It is issued only after the identity service has satisfied itself that the requester really is the workload it claims to be. That verification step is called attestation. In each environment we support (VMs, containers, functions) attestation rests on some environment-specific proof that the platform can check on its own.The second concept is the Data Project, which is our unit of ownership for data. A Data Project owns tables, has a set of authorized users, and has an identity of its own. Jobs run as the Data Project rather than as the person who launched them. That is what keeps table-level access control and audit consistent across scheduled and interactive use.So the problem is this. A Spark job on managed compute starts with an AWS execution role and no internal identity. Everything it needs from inside the company (reading an encrypted column, evaluating table ACLs, holding an interactive session open longer than a short-lived token lasts) requires the internal one.One identity, one roleThe design rests on a decision made outside the Spark stack entirely. Each Data Project identity maps 1:1 to a dedicated IAM role, and the service that owns Data Project metadata records that mapping.The mapping is what makes translation possible. It lets a statement in the provider’s vocabulary (“this process is running as role R”) become a statement in ours (“this process is workload W”). Without it there is nothing to translate into, and no amount of cryptography helps. The hard part of bridging two identity systems is rarely the protocol. It is committing to a mapping and then keeping it authoritative.One practical objection arrives immediately. A single AWS account cannot hold tens of thousands of IAM roles, and we expect data projects in the order of ten thousands. We handled this by sharding data project roles deterministically across a small pool of dedicated accounts, which scales well past the projected number of projects and has the useful side effect of keeping workload roles on the far side of an account boundary from the control planes that launch them.ComponentsFive components take part.The control plane is the only service allowed to launch Spark jobs. It resolves the Data Project’s IAM role, builds and signs a workload metadata payload, and submits the job with that role as the execution role. It holds a signing key issued to it for this purpose alone.The Data Project service is the authority for the identity-to-role mapping.The Identity service receives attestation requests, verifies them, and issues certificates.A Spark plugin provides the hook. Spark 3.0+ added a plugin interface with driver-side and executor-side components that are initialized while those processes are bootstrapping.AWS STS acts as a notary, which is not its usual job.Step one: the control plane makes a signed claimWhen a job is submitted, the control plane validates the caller, looks up the Data Project’s IAM role, and builds a small payload describing the workload it is about to launch. The payload names the application identity, the mapped role, and the environment, stack and detail that the resulting credentials should be scoped to. The control plane signs the payload and passes it and the signature through as ordinary job configuration.Two properties of this step matter. First, only one service can produce a valid signature, so the set of things that can assert “this workload is legitimate” is small and auditable. Second, the payload is a claim rather than an authority. On its own it proves nothing, because job configuration travels through infrastructure we do not fully control and anything that can read it can replay it.Step two: the workload proves it holds the cloud identityWhen the managed service launches the driver process, it does so under an OS user that has the execution role’s credentials available. The driver-side plugin initializes and uses those credentials to sign, but not send, a request to the provider’s identity endpoint (sts:GetCallerIdentity). The result is a short-lived pre-signed URL.That URL is a transferable proof of possession. Anyone can fetch it. Only the holder of the role’s credentials could have produced it. And the response comes from AWS rather than from the workload, stating which role signed the request. The workload cannot lie about the answer because the workload does not supply the answer.The pattern is not new. It is the same idea behind AWS IAM authentication in HashiCorp Vault and in AWS’s own function attestation flows. It generalizes well: any environment that gives a process cloud credentials and nothing else can still produce a verifiable statement about its own identity.The plugin sends one attestation request carrying both artifacts, the pre-signed URL and the signed metadata.Step three: corroborationThe identity service does four things.It fetches the pre-signed URL against AWS STS and reads back the role that signed it. This is the provider’s statement.It verifies the metadata signature against the key issued to the control plane. This is the platform’s statement.It corroborates the two. The role AWS reports must be the role the control plane said it dispatched, for the identity the metadata names.It issues certificates for that identity, scoped to the environment, stack and detail from the signed metadata.Step 3 is the point of the whole design. Neither statement is sufficient by itself, and they fail in complementary ways. The provider’s statement is unforgeable but under-specified: it gives you a role, and a role is not a workload. It says nothing about why this process exists or what it should be allowed to be. The platform’s statement is well-specified but unverifiable on its own, for the reasons above. Put together, the signature says the workload was legitimately dispatched and the pre-signed URL says this really is it. Neither party has to trust the workload’s own account of itself.There is a tension here that we settled deliberately, and other teams should expect to meet it. The provider’s answer identifies a role, not an application. Deriving an application name from a role name is fragile string matching, and the session identifiers the platform assigns are not under our control. We chose to take the application identity from the signed metadata and use the provider’s answer only for corroboration. That costs a network round trip and a stricter signing requirement, and it buys a much clearer trust story.The fan-out problemA Spark application is one driver and up to thousands of executors, created and destroyed throughout the life of a job. Attestation designed around one process per host does not survive that.There are two defensible answers.In the first, each executor attests independently. This is uniform, adds no new trust boundary, and grounds every process’s identity in the same provider-verified proof. The cost is amplification. One large job can produce thousands of STS calls and thousands of attestation requests in a burst, which looks a lot like an attack and creates a hard dependency on provider-side rate limits.In the second, executors inherit credentials from the driver. The driver attests once and distributes credentials to executors over Spark’s internal RPC, which we configure with authentication and AES-GCM encryption so that only processes in the same application can take part. Load on the identity service stays constant. The cost is a second trust boundary and a driver that is now a credential distribution point.Considering our scale at Netflix here we went with the second approach. The general lesson is that identity amplification is a capacity question, and it is better to answer it on purpose than to have it answered for you during an incident.LifecycleCertificates are short-lived by design, so bootstrapping is only half the work. On self-managed hosts a system service handles renewal. Inside managed compute there is no equivalent hook, so the driver JVM runs a timer that re-attests on a fixed interval, well ahead of expiry. Credential material is written to a temporary directory readable only by the process owner and removed when the process exits.Executors do not refresh. They are usually short-lived, and an executor that somehow outlives its credentials can exit and be replaced, which is cheaper than making every executor a renewal client.The rule worth carrying into any similar system is that attestation has to be a repeatable operation rather than a bootstrap step. Anything that can only happen once at process start will eventually be the reason a long-running job dies nine hours in.Why this generalizesIf you run your own identity system and are moving workloads onto managed compute, the specific services here matter less than the shape of the solution.Anchor on one exchangeable primitive. A single authoritative 1:1 mapping between your identity and a provider identity is what makes translation possible. The rest is plumbing.Require two independent claims and corroborate them. One from the provider, unforgeable and under-specified. One from your control plane, well-specified and replayable. Trust the intersection and never either alone.Keep the signer scarce. If exactly one service can make the platform’s claim, your trust story fits in a sentence.Hook the layer you still own. On managed compute you rarely control the host or its init system, but you almost always control the runtime: a plugin, an agent, an entrypoint. Attestation belongs there.Decide the amplification policy on purpose. Distributed engines multiply every per-process operation by their parallelism.Make attestation repeatable and credentials short-lived. Renewal is a requirement, not a follow-up.What makes the result trustworthy is that no single participant issues an identity by itself. Not the workload, not the control plane, not the provider. And the workload, which is the party in the weakest position to be trusted, is never asked to vouch for itself.AcknowledgementsThanks to Amer Hesson for the Data Project abstraction, Nick Siow for the Data Project IAM role sharding, and Doug Clark for Metatron attestation.Trading a Cloud Identity for Your Own: Workload Attestation on Managed Compute was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Builders operating multi-cloud or sensitive workloads can strengthen identity portability without surrendering trust decisions to one cloud.
|
|
|
|
How was today's dispatch?
Your 1-click feedback tunes our story selection algorithms for tomorrow.
|
|
🎁 VIP REFERRAL PROGRAM
Invite Colleagues, Unlock Executive Intelligence Pro
You have referred 0 colleagues.
Share your personal link to climb the reader leaderboard:
https://brief.delights.pro/join?ref=VIP
|
|
Share Your VIP Invite →
|
|
|
|
|
|
|
|
