|
|
|
|
KNOWLEDGE, REFINED • Builders 🔧 • September 20, 2026
|
|
|
📊 Scanned 6,444 articles • Selected top 12 role-critical insights for today.
|
|
|
🤔 CONTRARIAN TAKEAWAY
The other side of ai market outlook
492 articles say ai boom, but 102 argue ai bubble.
|
|
🚀 AI & Innovation
|
|
🚀 AI & Innovation
via The GitHub Blog • 1 min read
|
A rewrite this size wasn't affordable before agents. Here's what porting the Copilot agent runtime to 800,000 lines of production Rust actually took. The post Migrating the GitHub Copilot runtime to Rust, using Copilot appeared first on The GitHub Blog.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Implementing streaming output is key to building responsive and performant AI-powered user experiences.
|
|
|
|
🚀 AI & Innovation
via Replicate's blog • 1 min read
|
Create and run your own fine-tuned Flux models programmatically using Replicate's HTTP API.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Programmatic fine-tuning of advanced image models via API enables custom AI solutions.
|
|
|
|
🚀 AI & Innovation
via Hugging Face - Blog • 1 min read
|
Analyzes key technical developments and market implications surrounding Illustrating Reinforcement Learning from Human Feedback (RLHF). Provides essential context for enterprise engineering teams.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Understanding RLHF is key to developing and deploying more reliable and aligned AI models.
|
|
|
☁️ Enterprise Tech
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
Amazon EC2 R9g and R9gd instances powered by AWS Graviton5 are now generally available, delivering up to 25% better compute performance than R8g, ideal for databases, in-memory caches, and real-time analytics.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Evaluating new Graviton instances is crucial for optimizing cloud spend and performance for core data services.
|
|
|
|
☁️ Enterprise Tech
via AWS News Blog • 1 min read
|
DynamoDB now supports native vector search with single-digit millisecond latency at 99%+ recall. It is designed for any scale, even trillions of vectors and requires zero infrastructure management.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Native vector search in DynamoDB can significantly simplify the architecture for AI-driven applications.
|
|
|
|
☁️ Enterprise Tech
via Vercel News • 5 min read
|
To a user, an app with a Next.js frontend and a FastAPI backend feels like one product, and the same should be true for the engineers who build it. Instead, the two pieces are often deployed across different clouds with different development and deployment workflows.Today we're introducing Vercel Services in public beta, which lets you run multiple frameworks in one Vercel Project. This unlocks:Atomic deployments: Your frontend, backend, and other services stay in sync and deploy or roll back togetherShared preview deployments: See how any change affects all your servicesInternal service communication: Services can talk to each other without routing through the public InternetVercel handles the rest: routing, builds, deployments, and auto-scaling in production. The developer experience you already know from Vercel now covers your entire application.Compose applications with Vercel ServicesDeclare your services under the services key in vercel.json keeping the routing configuration explicit:Services can be mounted to a shared routing table without the need for a reverse proxy or CORS.The services configuration is recognized at multiple levels of the Vercel platform:The Deployments panel shows a visualization of services graphThe Logs UI allows filtering by individual serviceThe vercel dev CLI automatically runs all services giving you a production-like environment locallyService bindingsServices can talk to other services internally, without routing through the public Internet. Here's an example how the new bindings configuration key enables that:Now the JavaScript frontend code can talk to the Python service internally via the URL stored in the BACKEND_INTERNAL_URL environment variable:Service-to-service traffic stays on the Vercel network rather than egressing to the public internet. Many independent services can become one application connected by the same wiring, rather than separate deployments that you stitch together across hosts.Framework-defined infrastructureMost frameworks run on Vercel with zero configuration. Framework-defined infrastructure means that each service's framework is auto-detected and auto-provisioned, from FastAPI and Flask for Python to Express and Hono for TypeScript, with first-class support for Go and Rust servers.Services run on Fluid compute, autoscaling with traffic, while you pay only for active CPU time. But framework-defined optimizations go even deeper. With Django, for example, we automatically detect where static assets live and serve them from the CDN.The full stack platformVercel Services gives Vercel a structured way to build and run your application.Those services run on a platform that includes everything any backend could need, whether you're building APIs, agents, or background workers: compute, data, networking, background work, and secure connections to external services.Isolated compute for agent servicesAgents can read files, run commands, and write code. But that capability needs to be secure. Vercel Sandbox gives each agent its own Linux computer: a filesystem, a shell, Docker support, and its own kernel, completely isolated from your deployments. Agents can execute code, grep files, and spin up Redis or Postgres as dependencies. Nothing inside the sandbox can touch your production environment.With automatic persistence, state carries between sessions, and on Pro, sandboxes can run for up to 24 hours.Real-time backends with WebSocket supportVercel Functions handle persistent WebSocket connections in every Vercel runtime, whether your backend is written in Node.js, Python, or Go. They work with standard WebSocket libraries like Socket.IO, and because connections run on Fluid compute with Active CPU pricing, you only pay for the time spent processing messages, not the time a connection sits idle.Reach external services with Vercel ConnectAgents and backends need to reach services outside your project (Slack, GitHub, managed databases), which usually means storing a long-lived secret in your environment. Vercel Connect replaces that with a short-lived credential your app requests at runtime, scoped to the task in front of it. There is no long-lived secret left to leak.Databases and storage for your servicesYou can provision a database from the Vercel Marketplace in a few clicks, with credentials injected for you, from providers like Neon, Supabase, and from AWS with Aurora PostgreSQL, Aurora DSQL, and DynamoDB. Amazon OpenSearch Serverless adds full-text and vector search, and Vercel Blob handles object storage. You can query and manage any of it without leaving Vercel, and both Blob and the AWS integrations authenticate with short-lived OIDC tokens rather than stored secrets.Run durable workflows and background jobsVercel Queues processes background jobs off the request path. Vercel Workflow handles durable multi-step processes that survive crashes and redeploys. And Vercel Cron runs work on a schedule. This is the part of a backend that outlives a request, and it runs on the same platform as everything else, so the long-running side of your backend needs no separate infrastructure.Connect privately and run on the right computeWhen a backend needs to reach a private database or an internal network instead of a public service, Secure Compute, static IPs, and VPC peering open that path.Functions run for up to 30 minutes on Pro and Enterprise, and Python backends deploy with up to 500mb of dependencies.Underneath all of it, Fluid compute is the default for new projects, and Active CPU pricing bills for the time your code is actually running rather than the time it waits, the right shape for idle-heavy backend and AI work.A single platform for everything you shipVercel Services brings your frontend, backend, and supporting services into one project. They build together, preview together, deploy together, and communicate internally by default.Combined with Vercel's built-in primitives for compute, data, queues, workflows, cron, secure networking, and sandboxed agent environments, you can run the full stack without stitching together separate platforms.To start composing services, read the Vercel Services documentation and the routing and communication guide. If you are new to running a backend on Vercel, the zero-config backends post is the place to begin. Read more
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Unifying full-stack development on a single platform can streamline deployment and improve team velocity.
|
|
|
|
☁️ Enterprise Tech
via Staff Engineer: Leadership beyond the management track • 11 min read
|
I kind of think writing about engineering strategy is hard because good strategy is pretty boring, and it’s kind of boring to write about. Also I think when people hear “strategy” they think “innovation” - Camille Fournier Few companies understand their engineering strategy and vision. One consequence of this uncertainty is the industry belief that these documents are difficult to write. In some conversations, it can feel like you’re talking about something mystical, but these are just mundane documents. The reality is that good engineering strategy is boring and that it’s easier to write an effective strategy than a bad one. To write an engineering strategy, write five design documents, and pull the similarities out. That’s your engineering strategy. To write an engineering vision, write five engineering strategies, and forecast their implications two years into the future. That’s your engineering vision. If you can’t resist the urge to include your most brilliant ideas in the process, then you can include them in your prework. Write all of your best ideas in a giant document, delete it, and never mention any of them again. Now that those ideas are out of your head, your head is cleared for the work ahead. Durably useful engineering strategy and vision are the output of iterative, bottom-up organizational learning. As such, all learning contributes to your organization’s strategy and vision, but your contribution doesn’t have to be so abstract. Even if you’re not directly responsible for that work, there are practical steps that you can take to advance your organization’s strategy and vision, starting right now. When and why Before diving into the recipe for creating effective strategies and visions, a good starting question is, “When and why should I actually create them?” Strategies are tools of proactive alignment that empower teams to move quickly and with confidence. Strategies allow everyone–not just the empowered few–to make quick, confident decisions that might have otherwise cost them a week of discussion. Strategies are also the bricks that narrow your many possible futures down enough that it’s possible to write a realistic vision. If you realize that you’ve rehashed the same discussion three or four times, it’s time to write a strategy. When the future’s too hazy to identify investments worth making, it’s time to write another vision. If neither of those sound like familiar problems – move on to other work for now and return later. Write five design docs Design documents describe the decisions and tradeoffs you’ve made in specific projects. Your company might call them RFCs or tech specs. Stranger names happen, too; Uber bewilderingly called them DUCKS until they later standardized on RFC. A good design document describes a specific problem, surveys possible solutions, and explains the selected approach’s details. There are many formats to pick from; a few places to start your thinking are Design Docs, Markdown, and Git, Design Docs at Google, and Technical Decision-Making and Alignment in a Remote Culture. Whether a given project requires a design document comes down to personal judgment, but I’ve found a few rules useful. You should write design documents for any project whose capabilities will be used by numerous future projects. You should also write design documents for projects that meaningfully impact your users. You should write a design document for any work taking more than a month of engineering time. A batch of five design docs is the ideal ingredient for writing an effective strategy because design documents have what bad strategies lack: detailed specifics grounded in reality. It’s easy for two well-meaning engineers on the same team to interpret an abstract strategy in different ways, but it’s much harder to stay misaligned when you’re implementing a specific solution. A few recommendations as you write: Start from the problem. The clearer the problem statement, the more obvious the solutions. If solutions aren’t obvious, spend more time clarifying the problem. If you’re stuck articulating the problem, show what you have to five people and ask them what’s missing: fresh eyes always see the truth. Keep the template simple. Most companies have a design document template, which is a great pattern to follow. However, those templates are often expanded to serve too many goals. Overloaded templates discourage folks from writing design documents in the first place. Prefer minimal design document templates that allow authors to select the most useful sections and only insist on exhaustive details for the riskiest projects. Gather and review together, write alone. It’s very unlikely that you personally have all the relevant context to write the best design document on a given topic. Before getting far into the process, collect input from folks with relevant perspectives, particularly those who will rely on the output of your design document. However, be skeptical of carrying that collaborative process into writing the design document itself. Most folks are better writers than they are editors. This means it’s usually harder to edit a group document into clear writing than to identify one author to write a clear document. Gather perspectives widely but write alone. Just be careful not to fall in love with what you’ve written until after you’ve reviewed it with others. Prefer good over perfect. It’s better to write a good document and get it in front of others than it is to delay for something marginally better. This is particularly valuable to keep in mind when giving feedback on other folks’ designs; it’s easy to fall into the trap of expecting their designs to be just as good as your best design. Particularly as you become more senior, it’s toxic to push every design to meet the bar of your own best work. Focus on pushing designs to be good, rather than fixating on your own best as the relevant quality bar. It takes a lot of practice to write great design documents. If you want to improve yours, my best advice is to reread your designs after you’ve finished implementing them and study the places where your implementation deviated from your plan–what caused those deviations? Oh, and of course, just keep writing more of them. Synthesize those five design docs into a strategy After your organization has written five design documents, sit down and read them all together. Look for controversial decisions that came up in multiple designs, particularly those that were hard to agree on. A recent example of mine was getting stuck debating whether Redis was appropriate as durable storage or only as a cache. Rather than starting from zero in each design document review, wouldn’t it be easier if we reviewed our recent decisions about using Redis, reflected on how we made those decisions and wrote them down as a strategy? Good strategies guide tradeoffs and explain the rationale behind that guidance. Bad strategies state a policy without explanation, which decouples them from the context they were made. Without context, your strategy rapidly becomes incomprehensible–why did they decide this?–and difficult to adapt as the underlying context shifts. A few interesting strategies to read while thinking about writing your own are A Framework for Responsible Innovation and How Big Technical Changes Happen at Slack. If you’re a Good Strategy, Bad Strategy convert–and that book has wholly transformed how I think about strategy–then you’ll note this definition of strategy is the “diagnosis” and “guiding policies” sections, deferring “coherent action” to the design documents. My best advice for writing a strategy document is: Start where you are. Working on strategy, it’s easy to be paralyzed by the inherently vast ambiguity we work in, but you’ve just got to dive in and start writing. Waiting for missing information doesn’t work: every missing document is missing for a good reason. Whatever you write will need to change, and if you write something particularly bad, you’ll quickly realize the need to change it. Where you are now is always the best place to start. Write the specifics. Write until you start to generalize, and then stop writing. If you can’t be specific, wait until you’ve written more design documents. Specific statements create alignment; generic statements create the illusion of alignment. Be opinionated. Good strategies are opinionated. If they aren’t opinionated, then they won’t provide any clarity on decision making. However, being opinionated on its own isn’t enough. You also need to show your work. Show your work. In math classes growing up, you had to show your work to get full credit. Here too, you must show the rationale behind your opinions. Showing your work builds confidence in the first version of a document, but even more importantly, by showing your work, you make it possible for others to modify and extend your work as the underlying context shifts. Some of the best strategies you write may at the time feel too obvious to bother writing. “When should we write design documents?” is a strategy worth writing. “Which databases do we use for which use cases?” is a strategy worth writing. “How should we stage our migration from monolith to services?” is worth writing, too. As we leave behind the idea of strategy as demonstrations of brilliance, we can start to write far more of them, and we can write them more casually. If it ends up not being used, you can always deprecate it later. Extrapolate five strategies into a vision As you collect more strategies, it’ll become increasingly challenging to reason about how the various strategies interact. Maybe one of your strategies is to Run less software and rely more on cloud solutions, but another one of your strategies is to prefer offloading complexity to the database whenever possible. How do you reconcile those strategies if you identify a database that would allow you to offload a great deal of complexity, but that isn’t offered by your cloud vendor? Take five of your recent strategies, extrapolate how their tradeoffs will play out over the next two to three years. As you edit through the contradictions and weave the threads together, you’ve written an engineering vision. The final version will give you what Tanya Reilly calls a robust belief in the future, which makes it easier to understand how your existing strategies relate to each other and simplifies writing new strategies that stand the test of time. For a useful vision, a few things to focus on are: Write two to three years out. Companies, organizations, and technology all change quickly enough that thinking too far into the future is fraught. It also doesn’t work if you write a vision that expires in six months–how many strategies would you realistically write within that six-month window? Try to focus on two to three years out; you can expand that horizon a bit if you’re a fairly established company. Ground in your business and your users. Effective visions ground themselves in serving your users and your business. That tight connection keeps the vision aligned with your leadership team’s core values–users and business. Bad visions treat technical sophistication as a self-justifying raison d’être–a view that is never shared by your company’s leadership. Be optimistic rather than audacious. Visions should be ambitious, but they shouldn’t be audacious. They should be possible, but the best possible version if possible. Do write what you could accomplish if every project is finished on time and without major setbacks. Don’t write what you think would be possible with infinite resources. Stay concrete and specific. Visions get more useful as they get more specific. Generic statements are easy to agree with but don’t help reconcile conflicting strategies. Be a bit more detailed than you’re comfortable with. Details in visions are often illustrative rather than declarative, giving a taste of the future’s flavor rather than offering a binding commitment. Keep it one to two pages long. The reality is that most people don’t read long documents. If you write something five or six pages long, readers will start dropping off without finishing it (or will skim it very rapidly without engaging with the details). Force yourself to write something compact, and reference extra context by linking to other documents for the subset of folks who want the full details. After you finish writing your vision, the first step folks usually take is sharing it widely across the engineering organization. There is so much work behind the vision–five design docs for each strategy, five strategies for one vision–it’s hard not to get excited when you’re done. So excited that it’s easy to get discouraged, then, when the response to your strategy will almost always be muted. There are a few reasons for the muted response. First, the core audience for your vision is folks writing strategies, which is a relatively small cohort. Second, a great vision is usually so obvious that it bores more than it excites. Don’t measure vision by the initial excitement it creates. Instead, measure it by reading a design document from two years ago and then one from last week; if there’s marked improvement, then your vision is good.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Insights into Vercel's build infrastructure can inform strategies for optimizing CI/CD performance and security.
|
|
|
|
☁️ Enterprise Tech
via Netflix TechBlog - Medium • 10 min read
|
Samuel Yeboah, Francesco Di Chiara and Mingliang LiuToday, Netflix runs two Flink autoscalers. That is exactly one more than we want. We built the first one in-house years ago, when there was no mature option suited to our platform. The second came from the Apache Flink community, and it can scale workloads our homegrown system was never designed for. We now run both in production and are steadily converging on the open-source one. Along the way we learned some hard lessons about metrics, cost, and the real price of maintaining infrastructure you could instead adopt, and we hope they are useful whether you run a handful of Flink jobs or tens of thousands.Why autoscaling is not optional at our scaleNetflix has run stream processing on Apache Flink since 2017. As of 2026 we operate more than 30,000 Flink jobs across multiple AWS regions. Most are not deployed by hand; they are generated by our managed platform Data Mesh, so the majority of users never touch a Flink job directly. A smaller but growing set are custom jobs, built and operated by teams across the company for use cases like personalization, Ads, and Live events. They range from single-operator jobs that shuttle records between Kafka topics to stateful pipelines with branches, joins, and terabytes of state, and their load swings with daily cycles, launches, and regional failovers.Provisioning every one of those jobs for its peak is wasteful; provisioning for the average causes lag during surges. And in our platform a scaling action is not free: by default it means taking a savepoint, stopping the job gracefully, and restarting it at the new size, which for a large stateful job can take minutes. That leaves a genuinely hard question: how do you give each job the resources it needs, when it needs them, without a human in the loop and without breaking anything?The first autoscaler: watching from outsideOur first answer, built around 2019, was an autoscaler shaped like a stream-processing job. It ran on Mantis, consuming a live feed of cluster-level metrics from Atlas, our telemetry platform, including CPU, network, Kafka lag, input-rate, and consume-rate signals for every job. The scaler combined lag-derived catch-up time, CPU/network utilization thresholds, observed performance history, and regression over recent input rate to decide when to scale up or whether a smaller cluster could handle the lookahead window. Because the autoscaler operates independently of the Flink platform, it remains unaffected by issues within Flink itself. Building it as a streaming job also made it easy to scale. Each autoscaler node handled the metrics for a subset of Flink jobs, and we never had to write custom sharding or coordination logic to keep up with a growing Flink fleet. It reliably cut resource usage by 25–45% across thousands of managed pipelines. Check our previous talk at Flink Forward 2020.But watching from outside has a ceiling. The system reasoned about a whole cluster through coarse container metrics, and it scaled a single knob, the total TaskManager count, so every operator in a job moved together. That fit the simple, single-operator pipelines it was built for, but not the multi-operator, stateful DAGs that teams were increasingly bringing to us for Ads, recommendations, and games. Those were exactly the jobs it could not reason about, and supporting each new case meant more custom logic rather than any general capability.The autoscaler is only as good as the metrics served by external systems beneath it. Those metrics could miss real trouble: a job could be completely busy without any of it showing up as CPU utilization, leaving the job stuck in a degraded state the scaler had no way to see. Recently a networking migration quietly changed how some traffic was reported, and a subset of the Atlas metrics the scaler relied on stopped capturing everything accurately. The gap stayed invisible until it surfaced in production much later.It was time to reconsider build versus buy.The second autoscaler: reasoning from insideWhen we started, the Flink community had no mature autoscaler to offer. By the time we re-evaluated, it did: the Apache Flink Autoscaler. Instead of watching containers from outside, it reasons from inside the job.Its key idea is to estimate each operator’s true processing rate (TPR): the throughput it could sustain if it were fully busy. Flink reports, per subtask, the fraction of each second spent doing actual work, separate from time spent backpressured or idle. Dividing observed throughput by that busy fraction extrapolates capacity to full utilization: an operator handling 700 records/sec while busy 70% of the time has a TPR of 700 / 0.7 = 1,000 records/sec. Starting from the sources, the autoscaler walks the job graph and uses each operator’s TPR, its input/output ratios, and a target utilization to compute the parallelism every vertex needs so that no operator becomes the bottleneck, rather than resizing the whole cluster as a unit.The two approaches make a different contract, summarized below.The decisive difference for us is the last two rows: the OSS autoscaler can scale exactly the stateful, multi-operator jobs our homegrown system could not, and it lets each job carry its own configuration — stabilization periods, thresholds, and other scaling behavior tuned to the workload. That made it the natural fit for the custom jobs teams had been scaling by hand.Making it work at Netflix scaleAdopting the algorithm was straightforward; the community had done the hard part. The work for us was running it reliably across our own jobs, and this is where our system differs most from the stock open-source deployment.Firstly, the OSS autoscaler was originally architected to reside within the Kubernetes Operator for Flink, but our Flink platform runs on its own control plane, not that operator (see our previous talk at Current Conference 2024). The community later made a fantastic decision to keep the core logic as a standalone library. They refactored four generic interfaces that made it easy to plug directly into our internal ecosystem: a context carrying job metadata and REST API info, a state store, an event handler, and a realizer that applies scaling decisions.That service is a Spring Boot application whose orchestration runs on Temporal, the durable workflow engine. An orchestrator workflow polls our Flink control plane about once a minute for the jobs with autoscaling enabled, and starts one long-running workflow per job. Each per-job workflow pulls that job’s per-vertex metrics from its Flink JobManager, runs the OSS evaluation algorithm, and, when a scaling decision results, hands it to a realizer that actuates the change through our Flink control plane.The workflow-per-job design was a direct response to pain. We first ran evaluations in a single batch loop over the whole set of jobs, and it was fragile: one slow or misbehaving job could stall metric collection and scaling for every job behind it. Giving each job its own durable workflow isolated that blast radius, so a single problematic job now fails and retries on its own, and the runtime scales out as we onboard more jobs.Secondly, three engineering gaps stood between “works in community” and “works at Netflix scale”:Metric collection at high parallelism. On big jobs, pulling metrics from the JobManager became a bottleneck, and part of the cause was in Flink’s runtime. To address that, we changed the JobManager to cache transient metric names and clean them up once instead of rescanning on every fetch, and we added server-side filtering so the autoscaler asks only for the metrics it needs. This let the autoscaler work on jobs up to 3,000 Flink subtasks, where it had previously struggled above roughly 1,000. Those are in our internal fork of Flink release, while some are contributed upstream such as FLINK-36172.Preserving forward chaining. Two separate vertices joined by a forward connection must run at the same parallelism, because records are handed over in memory on a fixed local channel. Scale one of them alone and Flink does not fail; it silently converts that edge into a network shuffle. Our fork detects forward-connected subgraphs and scales each as a unit.Respecting sink limits. Some sinks have finite write capacity, so we added detection for async-sink backpressure (also a fork change) to keep the autoscaler from scaling a job up into a sink that cannot absorb more.Before it actuates anything, the realizer runs a set of safety checks. For example, it refuses to scale a job down in a region being evacuated during a company-wide region failover. It also verifies there is enough disk for the new cluster to hold the job’s checkpoint state, and it adds a small standby buffer for larger clusters.The road to one autoscalerLast year, the OSS-based autoscaler achieved general availability for custom jobs at Netflix, yielding promising initial outcomes. For instance, our client telemetry and logging team achieved a 58% reduction in its annualized Flink compute expenditures, saving approximately $1.1 million annually. This efficiency is driven by three key factors. First, whereas static provisioning must always account for peak loads, autoscaling dynamically adapts to daily cycles, capturing the drop in traffic during nights and weekends compared to weekday peaks. Second, rather than relying on teams to manually optimize resources following performance improvements or post-holiday slowdowns, the autoscaler continually adjusts capacity. Finally, adopting uniform container dimensions enables superior bin-packing and more granular scaling increments.Additionally, scaling down too eagerly is its own trap. Cut too deep and CPU saturates, lag spikes, and the system cannot react instantly because its metric window and stabilization period have to rebuild after each restart. We now run a target utilization of 0.45, below the community default of 0.7, deliberately trading a little efficiency for stability. Fewer and calmer rescales are worth the marginal cost for large stateful jobs.While our scaler provides fine-grained signals and vertex-level decision units for stateful DAGs, fast rescaling still heavily depends on Flink Core’s state restoration performance. Today, the biggest remaining cost in scaling a stateful job isn’t the scaler’s logic — it’s the restart and state recovery process itself. Flink 2 addresses this through its disaggregated state architecture, keeping state in external storage rather than on local disk, which can sharply reduce how much a rescale or recovery depends on total state size. Having started supporting Flink 2.2 at Netflix, we plan on experimenting with this new state backend to see if it can help eliminate state recovery bottlenecks when scaling large stateful jobs.Looking ahead, we aim to migrate all internal scaler use cases onto the new one based on OSS autoscaler to simplify our operational surface area.Key TakeawaysAlong the way, three lessons that generalize beyond Flink:Metric choice matters more than algorithm sophistication. Our most useful debugging was rarely about the scaling math; it was about which signal to trust most. Understand your metrics before you tune your algorithm.Set sensible defaults, but leave room to tune. Our managed jobs are similar enough that one good default covers most of them untouched, which is the point of a platform. But forcing a single configuration on every job punishes the ones that do not fit, so we pair defaults with per-job overrides and deliberately hide the knobs that need deep expertise. Most teams should never have to think about the autoscaler.Adopt, then extend. We built in-house because in 2019 nothing mature fit our platform. When a strong community project appeared, the right move was neither to defend our investment forever nor to rip it out overnight, but to adopt it for new workloads, contribute fixes back, and plan a deliberate migration.Thanks to the Flink and Data Mesh teams for the control-plane changes this work depended on, to the Temporal team and our early pilot teams, and to the Apache Flink autoscaler maintainers whose foundation we built on. Special thanks to Andy Zhang, Calvin Cheung, Daniel Trager, Guil Pires, Mark Cho, Matthew Kornitsky, Nikhil Sulegaon, Sujay Jain, and Tom Lee.A Tale of Two Flink Autoscalers was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
|
💡 WHY THIS MATTERS FOR YOUR ROLE
Real-world Flink autoscaling strategies offer critical lessons for managing complex data infrastructure.
|
|
|
|
How was today's dispatch?
Your 1-click feedback tunes our story selection algorithms for tomorrow.
|
|
🎁 VIP REFERRAL PROGRAM
Invite Colleagues, Unlock Executive Intelligence Pro
You have referred 0 colleagues.
Share your personal link to climb the reader leaderboard:
https://brief.delights.pro/join?ref=VIP
|
|
Share Your VIP Invite →
|
|
|
|
|
|
|
|
