1_DevOps'ish

Week Ending March 9, 2025

https://lwkd.info/2025/20250312

Developer News

The next New Contributor Orientation will be March 18. Scripts for the NCO are complete, so if you’d be willing to run one someday, please contact SIG-ContribEx.

SIG/WG Leads, please finish your 2024 annual reports if you have not already.

KubeCon London Reminders: Meet & Greet and register for the Maintainer Summit

Release Schedule

Next Deadline: Code Freeze, March 20

Exceptions are closed, and we are on the countdown to Code Freeze, plus many other deadlines.

Patch releases 1.29.15, 1.30.11, 1.31.7, and 1.32.3 are available.

KEP of the Week

KEP 4960: Container Stop Signals

This KEP aims to add the container runtime level STOPSIGNAL instruction to the Kubernetes API. The KEP proposes adding a StopSignal container lifecycle which takes a valid signal which would be sent to the container runtime when the container is stopped. Currently the only way to configure custom stop signals is to bake them in the container image definition. This is difficult if you are using third party images. Container runtimes today reads the stop signal defined in container images and if there is none defined, falls back to SIGTERM. The new StopSignal lifecycle would have a higher priority than both of these. If a container has a stop signal defined in the container spec, that signal would be used over the signal defined in the image. The stop signal would get the default value of SIGTERM if no signal is defined in either the container spec or in the image definition.

This KEP is tracked for alpha in the ongoing 1.33 release cycle.

Other Merges

kubectl: expose context parameter in DefaultRemoteExecutor

Versioned validation test utilitizes and fuzz tester for declarative validation

Minimum value validation tag +k8s:minimum added

Declarative Validation enabled for ReplicationController

Fixes for validation test for ReplicationController

Enabled Semver CEL library

Unify ListOptions validation between cache and etcd

extra_pkgs needed by main API types added to validation-gen

DRA: Prioritized Alternatives in Device Requests

Optimizations to avoid writing the checkpoint file when it’s not necessary

InPlacePodVerticalScaling: Track actuated resources to trigger resizes

Reduced log verbosity for high-frequency, low-value log entries in Job, IPAM, and ReplicaSet controllers

E2E tests for CLE

API server to no longer set the resize status to Proposed when it receives a resize request

Fix a bug on InPlacePodVerticalScalingExclusiveCPUs feature gate exclusive assignment availability check

randfill used in k/k

Dynamic configuration of service account name allowed and as part of the node audience restriction

Added consistency checking of etcd and watch cache and exposed it as apiserver_storage_digest metric

Kubelet to populate status.observedGeneration to reflect the pod’s latest metadata.generation

Fix for kubelet restart unmounting volumes of running pods if the PVC is being deleted

BalancedAllocation plugin will skip all best-effort (zero-requested) pod

User namespaces support (feature gate UserNamespacesSupport) is now enabled by default

Metrics added to expose the main known reasons for resource alingment errors

Clarifications to EndpointSlice semantics

kubelet no longer logs multiple errors when running on a system with no iptables binaries installed

New streaming proto list encoder implemented

Scheduling behavior revised to correctly handle nominated node changes

/flagz endpoint for kubelet endpoint

/flagz endpoint for kube-controller-manager endpoint

apiserver to return warnings when objects with “invalid” IP created

DRA: Prioritized Alternatives in Device Requests

Unit tests for client-go/tools/cache/listers.go

Kubernetes is now stricter about what will be accepted as IP addresses and CIDR blocks

Promotions

SELinuxChangePolicy and SELinuxMount to beta

ImageVolumeSources to beta

ClusterTrustBundles to beta

Deprecated

Several GA feature gates removed

Subprojects and Dependency Updates

Kernel Module Manager v2.3.0: golang bump, tolerance support

Shoutouts

No shoutouts this week. Want to thank someone for special efforts to improve Kubernetes? Tag them #shoutouts channel.

via Last Week in Kubernetes Development https://lwkd.info/

March 12, 2025 at 06:00PM

Spotlight on SIG Apps

https://kubernetes.io/blog/2025/03/12/sig-apps-spotlight-2025/

In our ongoing SIG Spotlight series, we dive into the heart of the Kubernetes project by talking to the leaders of its various Special Interest Groups (SIGs). This time, we focus on SIG Apps, the group responsible for everything related to developing, deploying, and operating applications on Kubernetes. Sandipan Panda (DevZero) had the opportunity to interview Maciej Szulik (Defense Unicorns) and Janet Kuo (Google), the chairs and tech leads of SIG Apps. They shared their experiences, challenges, and visions for the future of application management within the Kubernetes ecosystem.

Introductions

Sandipan: Hello, could you start by telling us a bit about yourself, your role, and your journey within the Kubernetes community that led to your current roles in SIG Apps?

Maciej: Hey, my name is Maciej, and I’m one of the leads for SIG Apps. Aside from this role, you can also find me helping SIG CLI and also being one of the Steering Committee members. I’ve been contributing to Kubernetes since late 2014 in various areas, including controllers, apiserver, and kubectl.

Janet: Certainly! I'm Janet, a Staff Software Engineer at Google, and I've been deeply involved with the Kubernetes project since its early days, even before the 1.0 launch in 2015. It's been an amazing journey!

My current role within the Kubernetes community is one of the chairs and tech leads of SIG Apps. My journey with SIG Apps started organically. I started with building the Deployment API and adding rolling update functionalities. I naturally gravitated towards SIG Apps and became increasingly involved. Over time, I took on more responsibilities, culminating in my current leadership roles.

About SIG Apps

All following answers were jointly provided by Maciej and Janet.

Sandipan: For those unfamiliar, could you provide an overview of SIG Apps' mission and objectives? What key problems does it aim to solve within the Kubernetes ecosystem?

As described in our charter, we cover a broad area related to developing, deploying, and operating applications on Kubernetes. That, in short, means we’re open to each and everyone showing up at our bi-weekly meetings and discussing the ups and downs of writing and deploying various applications on Kubernetes.

Sandipan: What are some of the most significant projects or initiatives currently being undertaken by SIG Apps?

At this point in time, the main factors driving the development of our controllers are the challenges coming from running various AI-related workloads. It’s worth giving credit here to two working groups we’ve sponsored over the past years:

The Batch Working Group, which is looking at running HPC, AI/ML, and data analytics jobs on top of Kubernetes.

The Serving Working Group, which is focusing on hardware-accelerated AI/ML inference.

Best practices and challenges

Sandipan: SIG Apps plays a crucial role in developing application management best practices for Kubernetes. Can you share some of these best practices and how they help improve application lifecycle management?

Implementing health checks and readiness probes ensures that your applications are healthy and ready to serve traffic, leading to improved reliability and uptime. The above, combined with comprehensive logging, monitoring, and tracing solutions, will provide insights into your application's behavior, enabling you to identify and resolve issues quickly.

Auto-scale your application based on resource utilization or custom metrics, optimizing resource usage and ensuring your application can handle varying loads.

Use Deployment for stateless applications, StatefulSet for stateful applications, Job and CronJob for batch workloads, and DaemonSet for running a daemon on each node. Use Operators and CRDs to extend the Kubernetes API to automate the deployment, management, and lifecycle of complex applications, making them easier to operate and reducing manual intervention.

Sandipan: What are some of the common challenges SIG Apps faces, and how do you address them?

The biggest challenge we’re facing all the time is the need to reject a lot of features, ideas, and improvements. This requires a lot of discipline and patience to be able to explain the reasons behind those decisions.

Sandipan: How has the evolution of Kubernetes influenced the work of SIG Apps? Are there any recent changes or upcoming features in Kubernetes that you find particularly relevant or beneficial for SIG Apps?

The main benefit for both us and the whole community around SIG Apps is the ability to extend kubernetes with Custom Resource Definitions and the fact that users can build their own custom controllers leveraging the built-in ones to achieve whatever sophisticated use cases they might have and we, as the core maintainers, haven’t considered or weren’t able to efficiently resolve inside Kubernetes.

Contributing to SIG Apps

Sandipan: What opportunities are available for new contributors who want to get involved with SIG Apps, and what advice would you give them?

We get the question, "What good first issue might you recommend we start with?" a lot :-) But unfortunately, there’s no easy answer to it. We always tell everyone that the best option to start contributing to core controllers is to find one you are willing to spend some time with. Read through the code, then try running unit tests and integration tests focusing on that controller. Once you grasp the general idea, try breaking it and the tests again to verify your breakage. Once you start feeling confident you understand that particular controller, you may want to search through open issues affecting that controller and either provide suggestions, explaining the problem users have, or maybe attempt your first fix.

Like we said, there are no shortcuts on that road; you need to spend the time with the codebase to understand all the edge cases we’ve slowly built up to get to the point where we are. Once you’re successful with one controller, you’ll need to repeat that same process with others all over again.

Sandipan: How does SIG Apps gather feedback from the community, and how is this feedback integrated into your work?

We always encourage everyone to show up and present their problems and solutions during our bi-weekly meetings. As long as you’re solving an interesting problem on top of Kubernetes and you can provide valuable feedback about any of the core controllers, we’re always happy to hear from everyone.

Looking ahead

Sandipan: Looking ahead, what are the key focus areas or upcoming trends in application management within Kubernetes that SIG Apps is excited about? How is the SIG adapting to these trends?

Definitely the current AI hype is the major driving factor; as mentioned above, we have two working groups, each covering a different aspect of it.

Sandipan: What are some of your favorite things about this SIG?

Without a doubt, the people that participate in our meetings and on Slack, who tirelessly help triage issues, pull requests and invest a lot of their time (very frequently their private time) into making kubernetes great!

SIG Apps is an essential part of the Kubernetes community, helping to shape how applications are deployed and managed at scale. From its work on improving Kubernetes' workload APIs to driving innovation in AI/ML application management, SIG Apps is continually adapting to meet the needs of modern application developers and operators. Whether you’re a new contributor or an experienced developer, there’s always an opportunity to get involved and make an impact.

If you’re interested in learning more or contributing to SIG Apps, be sure to check out their SIG README and join their bi-weekly meetings.

SIG Apps Mailing List

SIG Apps on Slack

via Kubernetes Blog https://kubernetes.io/

March 11, 2025 at 08:00PM

Blog: Spotlight on SIG Apps

https://www.kubernetes.dev/blog/2025/03/12/sig-apps-spotlight-2025/

In our ongoing SIG Spotlight series, we dive into the heart of the Kubernetes project by talking to the leaders of its various Special Interest Groups (SIGs). This time, we focus on SIG Apps, the group responsible for everything related to developing, deploying, and operating applications on Kubernetes. Sandipan Panda (DevZero) had the opportunity to interview Maciej Szulik (Defense Unicorns) and Janet Kuo (Google), the chairs and tech leads of SIG Apps. They shared their experiences, challenges, and visions for the future of application management within the Kubernetes ecosystem.

Introductions

Sandipan: Hello, could you start by telling us a bit about yourself, your role, and your journey within the Kubernetes community that led to your current roles in SIG Apps?

Maciej: Hey, my name is Maciej, and I’m one of the leads for SIG Apps. Aside from this role, you can also find me helping SIG CLI and also being one of the Steering Committee members. I’ve been contributing to Kubernetes since late 2014 in various areas, including controllers, apiserver, and kubectl.

Janet: Certainly! I’m Janet, a Staff Software Engineer at Google, and I’ve been deeply involved with the Kubernetes project since its early days, even before the 1.0 launch in 2015. It’s been an amazing journey!

My current role within the Kubernetes community is one of the chairs and tech leads of SIG Apps. My journey with SIG Apps started organically. I started with building the Deployment API and adding rolling update functionalities. I naturally gravitated towards SIG Apps and became increasingly involved. Over time, I took on more responsibilities, culminating in my current leadership roles.

About SIG Apps

All following answers were jointly provided by Maciej and Janet.

Sandipan: For those unfamiliar, could you provide an overview of SIG Apps’ mission and objectives? What key problems does it aim to solve within the Kubernetes ecosystem?

As described in our charter, we cover a broad area related to developing, deploying, and operating applications on Kubernetes. That, in short, means we’re open to each and everyone showing up at our bi-weekly meetings and discussing the ups and downs of writing and deploying various applications on Kubernetes.

Sandipan: What are some of the most significant projects or initiatives currently being undertaken by SIG Apps?

At this point in time, the main factors driving the development of our controllers are the challenges coming from running various AI-related workloads. It’s worth giving credit here to two working groups we’ve sponsored over the past years:

The Batch Working Group, which is looking at running HPC, AI/ML, and data analytics jobs on top of Kubernetes.

The Serving Working Group, which is focusing on hardware-accelerated AI/ML inference.

Best practices and challenges

Sandipan: SIG Apps plays a crucial role in developing application management best practices for Kubernetes. Can you share some of these best practices and how they help improve application lifecycle management?

Implementing health checks and readiness probes ensures that your applications are healthy and ready to serve traffic, leading to improved reliability and uptime. The above, combined with comprehensive logging, monitoring, and tracing solutions, will provide insights into your application’s behavior, enabling you to identify and resolve issues quickly.

Auto-scale your application based on resource utilization or custom metrics, optimizing resource usage and ensuring your application can handle varying loads.

Use Deployment for stateless applications, StatefulSet for stateful applications, Job and CronJob for batch workloads, and DaemonSet for running a daemon on each node. Use Operators and CRDs to extend the Kubernetes API to automate the deployment, management, and lifecycle of complex applications, making them easier to operate and reducing manual intervention.

Sandipan: What are some of the common challenges SIG Apps faces, and how do you address them?

The biggest challenge we’re facing all the time is the need to reject a lot of features, ideas, and improvements. This requires a lot of discipline and patience to be able to explain the reasons behind those decisions.

Sandipan: How has the evolution of Kubernetes influenced the work of SIG Apps? Are there any recent changes or upcoming features in Kubernetes that you find particularly relevant or beneficial for SIG Apps?

The main benefit for both us and the whole community around SIG Apps is the ability to extend kubernetes with Custom Resource Definitions and the fact that users can build their own custom controllers leveraging the built-in ones to achieve whatever sophisticated use cases they might have and we, as the core maintainers, haven’t considered or weren’t able to efficiently resolve inside Kubernetes.

Contributing to SIG Apps

Sandipan: What opportunities are available for new contributors who want to get involved with SIG Apps, and what advice would you give them?

We get the question, “What good first issue might you recommend we start with?” a lot :-) But unfortunately, there’s no easy answer to it. We always tell everyone that the best option to start contributing to core controllers is to find one you are willing to spend some time with. Read through the code, then try running unit tests and integration tests focusing on that controller. Once you grasp the general idea, try breaking it and the tests again to verify your breakage. Once you start feeling confident you understand that particular controller, you may want to search through open issues affecting that controller and either provide suggestions, explaining the problem users have, or maybe attempt your first fix.

Like we said, there are no shortcuts on that road; you need to spend the time with the codebase to understand all the edge cases we’ve slowly built up to get to the point where we are. Once you’re successful with one controller, you’ll need to repeat that same process with others all over again.

Sandipan: How does SIG Apps gather feedback from the community, and how is this feedback integrated into your work?

We always encourage everyone to show up and present their problems and solutions during our bi-weekly meetings. As long as you’re solving an interesting problem on top of Kubernetes and you can provide valuable feedback about any of the core controllers, we’re always happy to hear from everyone.

Looking ahead

Sandipan: Looking ahead, what are the key focus areas or upcoming trends in application management within Kubernetes that SIG Apps is excited about? How is the SIG adapting to these trends?

Definitely the current AI hype is the major driving factor; as mentioned above, we have two working groups, each covering a different aspect of it.

Sandipan: What are some of your favorite things about this SIG?

Without a doubt, the people that participate in our meetings and on Slack, who tirelessly help triage issues, pull requests and invest a lot of their time (very frequently their private time) into making kubernetes great!

SIG Apps is an essential part of the Kubernetes community, helping to shape how applications are deployed and managed at scale. From its work on improving Kubernetes’ workload APIs to driving innovation in AI/ML application management, SIG Apps is continually adapting to meet the needs of modern application developers and operators. Whether you’re a new contributor or an experienced developer, there’s always an opportunity to get involved and make an impact.

If you’re interested in learning more or contributing to SIG Apps, be sure to check out their SIG README and join their bi-weekly meetings:

Mailing List

Slack Channel

via Kubernetes Contributors – Contributor Blog https://www.kubernetes.dev/blog/

March 11, 2025 at 08:00PM

Ep14 - Ask Me Anything About DevOps, Cloud, Kubernetes, Platform Engineering,...

There are no restrictions in this AMA session. You can ask anything about DevOps, Cloud, Kubernetes, Platform Engineering, containers, or anything else.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Codefresh 🔗 GitOps Argo CD Certifications: https://learning.codefresh.io (use "viktor" for a 50% discount) ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

via YouTube https://www.youtube.com/watch?v=bGF9duYPft0

Why Most Kubernetes Dashboards Are Failing You (and What's The Future)

Discover the common mistakes of Kubernetes dashboards and explore how to improve them for better navigation, search, and debugging in large-scale clusters. Learn about the limitations of Kubernetes API and how tools like Karpor can revolutionize cluster management with advanced search capabilities. Follow along as we set up Karpor, register clusters, and demonstrate powerful SQL-based search queries that can help you find resources across multiple clusters quickly and efficiently. Tune in for insights on transforming your Kubernetes experience from a file explorer to a Google-like search interface.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Twingate 🔗 https://twingate.com ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#KubernetesDashboards #KubernetesAPI #DevOpsToolkit

Consider joining the channel: https://www.youtube.com/c/devopstoolkit/join

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬ ➡ Transcript and commands: https://devopstoolkit.live/kubernetes/why-most-kubernetes-dashboards-are-failing-you-and-whats-the-future 🔗 Karpor: https://kusionstack.io/karpor

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬ If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬ 00:00 Kuberentes Dashboards 01:11 Typical Kuberentes Dashboard (feat. Headlamp) 01:27 Twingate (sponsor) 02:29 Typical Kuberentes Dashboard (feat. Headlamp) (cont.) 05:57 How Kubernetes Dashboards Work? 13:56 Multi-Cluster Kubernetes Dashboard (feat. Karpor) 25:08 How Should Dashboards Work?

via YouTube https://www.youtube.com/watch?v=1n6tBTj2sFM

Week Ending March 2, 2025

https://lwkd.info/2025/20250306

Developer News

Benjamin Elder is implementing a policy of not re-triaging some issues; comment on the PR.

We’re promoting several Kind-based test jobs to release-blocking, since they’ve shown themselves to be reliable and able to catch real issues.

Release Schedule

Next Deadline: Code Freeze and Test Freeze March 21

We’re on our final approach to Code Freeze. Topics for the Feature Blog were frozen this week. Time to wrap up your work for 1.33.

March patch release cherry-picks are due this Friday.

Featured PRs

130349: Declarative Validation: Add validation generator

This PR kicks off the implementation of Declarative Validation in Kubernetes by introducing validation-gen, a code generator that automatically produces validation logic based on structured //+ tags in types.go files. The validation system is modular, with a core set of built-in validation rules, and future PRs will expand it with additional plugins like dnsName, enum, and union. This PR also includes a robust test suite to ensure correctness across various validation scenarios.

Other Merges

Annotations added to the APIServer audit request with auth and authz latency

endpoints.kubernetes.io/managed-by label added to Endpoints

Added declarative validation to scheme

Fixes to EndpointSlice while working on new TrafficDistribution

Tests for encoding collections in Proto

OrderedNamespaceDeletion feature gate turned on by default

conntrack reconciler to check the dst port

Added DeclarativeValidation and DeclarativeValidationMismatchMetric feature gates

E2E tests for MutatingAdmissionPolicy

selinux to ignore pods with Recursive policy

CEL CIDR library’s ContainsCIDR to allow non-equal addresses

Fix for kernel version check condition in nftables proxier

New error matcher to make writing tests easier and consistent

New Origin field to the Error type added for use by validation tests

Added missing increments of queue_incoming_pods_total metric in scheduling queue

Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate

Some cleanup before pod subresource updates

InPlacePodVerticalScaling: Moved pod resource allocation management out of the status manager

kube-proxy nftables: Optimizations to kube-proxy restart time

scheduler: added filter integration tests for NodePorts plugin

Added e2e test for topology manager with restartable init containers

Fix for a bug with starting pods with postStart hooks specified

Volume affinity scheduling error message updated to be more intuitive

InPlacePodVerticalScaling to never attempt a resize of windows pods and use allocated resources for unsupported resize pods

Added a /statusz endpoint for kube-scheduler

Promotions

RecursiveReadOnlyMounts to GA

JobBackoffLimitPerIndex to GA

Deprecated

GA feature gate AppArmor removed

Version Updates

x/oauth2 to v0.27.0

x/crypto to v0.35.0

go.opentelemetry.io dependencies to v1.33.0/v0.58.0

via Last Week in Kubernetes Development https://lwkd.info/

March 06, 2025 at 07:12AM

Ep13 - Ask Me Anything About DevOps, Cloud, Kubernetes, Platform Engineering,... w/Scott Rosenberg

There are no restrictions in this AMA session. You can ask anything about DevOps, Cloud, Kubernetes, Platform Engineering, containers, or anything else. We'll have a special guest Scott Rosenberg to help us out.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Sponsor: Codefresh 🔗 GitOps Argo CD Certifications: https://learning.codefresh.io (use "viktor" for a 50% discount) ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬ ➡ BlueSky: https://vfarcic.bsky.social ➡ LinkedIn: https://www.linkedin.com/in/viktorfarcic/

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬ 🎤 Podcast: https://www.devopsparadox.com/ 💬 Live streams: https://www.youtube.com/c/DevOpsParadox

via YouTube https://www.youtube.com/watch?v=Q5ByWaOtq9U