1_DevOps'ish

Microsoft has warned that “the world should be prepared for several lines of potential Russian attack in the digital domain over the course of this winter."

In this tutorial, you'll learn how easy it is to encrypt Kubernetes traffic using Cilium Transparent Encryption with IPsec and WireGuard.

Developer News

Google has released an update for the Chrome browser to patch a new, actively exploited zero-day vulnerability (CVE-2022-4135) that resides in the GPU

Voluntary time off to reduce short term costs

A trio of security bugs allow remote attackers to unlock or start the car, operate climate controls, pop the trunk, and more — all via poorly coded mobile apps.

Washington has its eyes on the Netherlands, a small country that could hold the key to China's future in manufacturing cutting-edge semiconductors.

MOSCOW (AP) — Former U.S. intelligence contractor Edward Snowden, who fled prosecution after revealing highly classified surveillance programs, has received a Russian passport and taken the citizenship oath, Russian news agencies quoted his lawyer as saying Friday.

After a weekend of protests, the authorities in China are using the country’s all-seeing surveillance apparatus to find those bold enough to defy them.

With supply chain attacks surging, now is the time to reflect — and look forward. ReversingLabs’ new report explores the software supply chain security trends from 2022, predicts what is to come in 2023 and offers suggestions for combatting supply chain cyber risks.

The theft of state unemployment funds is the first pandemic fraud tied to foreign, state-sponsored cybercriminals that the U.S. government has acknowledged publicly.

The first obvious casualty of large language models is homework: the real training for everyone, though, and the best way to leverage AI, will be in verifying and editing information.

A new vulnerability [CVE-2022-23093] has been reported in the ping module of the FreeBSD operating system that could be exploited to remotely crash t

Hachyderm has reached 30,000 users. A 'small sized' service in regard to scale. However, in the process we have hit very familiar 'medium sized' scale problems which caused us to migrate our services out of my basement. This is the outage report, post mortem, and high level overview of the process of migrating to Hetzner in Germany. From observation to production fixes. This is the story.

Authors: Adrian Reber (Red Hat) Forensic container checkpointing is based on Checkpoint/Restore In Userspace (CRIU) and allows the creation of stateful copies of a running container without the container knowing that it is being checkpointed. The copy of the container can be analyzed and restored in a sandbox environment multiple times without the original container being aware of it. Forensic container checkpointing was introduced as an alpha feature in Kubernetes v1.25. How does it work? With the help of CRIU it is possible to checkpoint and restore containers. CRIU is integrated in runc, crun, CRI-O and containerd and forensic container checkpointing as implemented in Kubernetes uses these existing CRIU integrations. Why is it important? With the help of CRIU and the corresponding integrations it is possible to get all information and state about a running container on disk for later forensic analysis. Forensic analysis might be important to inspect a suspicious container without stopping or influencing it. If the container is really under attack, the attacker might detect attempts to inspect the container. Taking a checkpoint and analysing the container in a sandboxed environment offers the possibility to inspect the container without the original container and maybe attacker being aware of the inspection. In addition to the forensic container checkpointing use case, it is also possible to migrate a container from one node to another node without loosing the internal state. Especially for stateful containers with long initialization times restoring from a checkpoint might save time after a reboot or enable much faster startup times. How do I use container checkpointing? The feature is behind a feature gate , so make sure to enable the ContainerCheckpoint gate before you can use the new feature. The runtime must also support container checkpointing: containerd: support is currently under discussion. See containerd pull request #6965 for more details. CRI-O: v1.25 has support for forensic container checkpointing. Usage example with CRI-O To use forensic container checkpointing in combination with CRI-O, the runtime needs to be started with the command-line option --enable-criu-support=true . For Kubernetes, you need to run your cluster with the ContainerCheckpoint feature gate enabled. As the checkpointing functionality is provided by CRIU it is also necessary to install CRIU. Usually runc or crun depend on CRIU and therefore it is installed automatically. It is also important to mention that at the time of writing the checkpointing functionality is to be considered as an alpha level feature in CRI-O and Kubernetes and the security implications are still under consideration. Once containers and pods are running it is possible to create a checkpoint. Checkpointing is currently only exposed on the kubelet level. To checkpoint a container, you can run curl on the node where that container is running, and trigger a checkpoint: curl -X POST "https://localhost:10250/checkpoint/namespace/podId/container" For a container named counter in a pod named counters in a namespace named default the kubelet API endpoint is reachable at: curl -X POST "https://localhost:10250/checkpoint/default/counters/counter" For completeness the following curl command-line options are necessary to have curl accept the kubelet 's self signed certificate and authorize the use of the kubelet checkpoint API: --insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key Triggering this kubelet API will request the creation of a checkpoint from CRI-O. CRI-O requests a checkpoint from your low-level runtime (for example, runc ). Seeing that request, runc invokes the criu tool to do the actual checkpointing. Once the checkpointing has finished the checkpoint should be available at /var/lib/kubelet/checkpoints/checkpoint-pod-name_namespace-name-container-name-timestamp.tar You could then use that tar archive to restore the container somewhere else. Restore a checkpointed container outside of Kubernetes (with CRI-O) With the checkpoint tar archive it is possible to restore the container outside of Kubernetes in a sandboxed instance of CRI-O. For better user experience during restore, I recommend that you use the latest version of CRI-O from the main CRI-O GitHub branch. If you're using CRI-O v1.25, you'll need to manually create certain directories Kubernetes would create before starting the container. The first step to restore a container outside of Kubernetes is to create a pod sandbox using crictl : crictl runp pod-config.json Then you can restore the previously checkpointed container into the newly created pod sandbox: crictl create POD_ID container-config.json pod-config.json Instead of specifying a container image in a registry in container-config.json you need to specify the path to the checkpoint archive that you created earlier: { "metadata" : { "name" : "counter" }, "image" :{ "image" : "/var/lib/kubelet/checkpoints/checkpoint-archive.tar" } } Next, run crictl start CONTAINER_ID to start that container, and then a copy of the previously checkpointed container should be running. Restore a checkpointed container within of Kubernetes To restore the previously checkpointed container directly in Kubernetes it is necessary to convert the checkpoint archive into an image that can be pushed to a registry. One possible way to convert the local checkpoint archive consists of the following steps with the help of buildah : newcontainer = $( buildah from scratch) buildah add $newcontainer /var/lib/kubelet/checkpoints/checkpoint-pod-name_namespace-name-container-name-timestamp.tar / buildah config --annotation= io.kubernetes.cri-o.annotations.checkpoint.name= container-name $newcontainer buildah commit $newcontainer checkpoint-image:latest buildah rm $newcontainer The resulting image is not standardized and only works in combination with CRI-O. Please consider this image format as pre-alpha. There are ongoing discussions to standardize the format of checkpoint images like this. Important to remember is that this not yet standardized image format only works if CRI-O has been started with --enable-criu-support=true . The security implications of starting CRI-O with CRIU support are not yet clear and therefore the functionality as well as the image format should be used with care. Now, you'll need to push that image to a container image registry. For example: buildah push localhost/checkpoint-image:latest container-image-registry.example/user/checkpoint-image:latest To restore this checkpoint image (container-image-registry.example/user/checkpoint-image:latest ), the image needs to be listed in the specification for a Pod. Here's an example manifest: apiVersion : v1 kind : Pod metadata : namePrefix : example- spec : containers : - name : container-name image : container-image-registry.example/user/checkpoint-image:latest nodeName : destination-node Kubernetes schedules the new Pod onto a node. The kubelet on that node instructs the container runtime (CRI-O in this example) to create and start a container based on an image specified as registry/user/checkpoint-image:latest . CRI-O detects that registry/user/checkpoint-image:latest is a reference to checkpoint data rather than a container image. Then, instead of the usual steps to create and start a container, CRI-O fetches the checkpoint data and restores the container from that specified checkpoint. The application in that Pod would continue running as if the checkpoint had not been taken; within the container, the application looks and behaves like any other container that had been started normally and not restored from a checkpoint. With these steps, it is possible to replace a Pod running on one node with a new equivalent Pod that is running on a different node, and without losing the state of the containers in that Pod. How do I get involved? You can reach SIG Node by several means: Slack: #sig-node Mailing list

🐢 Magical shell history. Contribute to ellie/atuin development by creating an account on GitHub.

It seems like this only solves one part of the problem, though. I'm seeing that this post above did not go through to twitter, because "User is over daily status update limit.", which is not the case for the crossposter. I'm thinking they are enforcing tighter limits around automated tools, so this might be the issue. I'll get in touch with their support, but given the :hot_shit: they seem to be going through, I would not expect a swift response. -- @renatolond

Writing to the Data volume in a VM is dismally slow. Is using shared storage any quicker? What happens when you copy a VM to an external SSD, or to another Mac?

Authors: Sascha Grunert Debugging software in production is one of the biggest challenges we have to face in our containerized environments. Being able to understand the impact of the available security options, especially when it comes to configuring our deployments, is one of the key aspects to make the default security in Kubernetes stronger. We have all those logging, tracing and metrics data already at hand, but how do we assemble the information they provide into something human readable and actionable? Seccomp is one of the standard mechanisms to protect a Linux based Kubernetes application from malicious actions by interfering with its system calls . This allows us to restrict the application to a defined set of actionable items, like modifying files or responding to HTTP requests. Linking the knowledge of which set of syscalls is required to, for example, modify a local file, to the actual source code is in the same way non-trivial. Seccomp profiles for Kubernetes have to be written in JSON and can be understood as an architecture specific allow-list with superpowers, for example: { "defaultAction" : "SCMP_ACT_ERRNO" , "defaultErrnoRet" : 38 , "defaultErrno" : "ENOSYS" , "syscalls" : [ { "names" : ["chmod" , "chown" , "open" , "write" ], "action" : "SCMP_ACT_ALLOW" } ] } The above profile errors by default specifying the defaultAction of SCMP_ACT_ERRNO . This means we have to allow a set of syscalls via SCMP_ACT_ALLOW , otherwise the application would not be able to do anything at all. Okay cool, for being able to allow file operations, all we have to do is adding a bunch of file specific syscalls like open or write , and probably also being able to change the permissions via chmod and chown , right? Basically yes, but there are issues with the simplicity of that approach: Seccomp profiles need to include the minimum set of syscalls required to start the application. This also includes some syscalls from the lower level Open Container Initiative (OCI) container runtime, for example runc or crun . Beside that, we can only guarantee the required syscalls for a very specific version of the runtimes and our application, because the code parts can change between releases. The same applies to the termination of the application as well as the target architecture we're deploying on. Features like executing commands within containers also require another subset of syscalls. Not to mention that there are multiple versions for syscalls doing slightly different things and the seccomp profiles are able to modify their arguments. It's also not always clearly visible to the developers which syscalls are used by their own written code parts, because they rely on programming language abstractions or frameworks. How can we know which syscalls are even required then? Who should create and maintain those profiles during its development life-cycle? Well, recording and distributing seccomp profiles is one of the problem domains of the Security Profiles Operator , which is already solving that. The operator is able to record seccomp , SELinux and even AppArmor profiles into a Custom Resource Definition (CRD) , reconciles them to each node and makes them available for usage. The biggest challenge about creating security profiles is to catch all code paths which execute syscalls. We could achieve that by having 100% logical coverage of the application when running an end-to-end test suite. You get the problem with the previous statement: It's too idealistic to be ever fulfilled, even without taking all the moving parts during application development and deployment into account. Missing a syscall in the seccomp profiles' allow list can have tremendously negative impact on the application. It's not only that we can encounter crashes, which are trivially detectable. It can also happen that they slightly change logical paths, change the business logic, make parts of the application unusable, slow down performance or even expose security vulnerabilities. We're simply not able to see the whole impact of that, especially because blocked syscalls via SCMP_ACT_ERRNO do not provide any additional audit logging on the system. Does that mean we're lost? Is it just not realistic to dream about a Kubernetes where everyone uses the default seccomp profile ? Should we stop striving towards maximum security in Kubernetes and accept that it's not meant to be secure by default? Definitely not. Technology evolves over time and there are many folks working behind the scenes of Kubernetes to indirectly deliver features to address such problems. One of the mentioned features is the seccomp notifier , which can be used to find suspicious syscalls in Kubernetes. The seccomp notify feature consists of a set of changes introduced in Linux 5.9. It makes the kernel capable of communicating seccomp related events to the user space. That allows applications to act based on the syscalls and opens for a wide range of possible use cases. We not only need the right kernel version, but also at least runc v1.1.0 (or crun v0.19) to be able to make the notifier work at all. The Kubernetes container runtime CRI-O gets support for the seccomp notifier in v1.26.0 . The new feature allows us to identify possibly malicious syscalls in our application, and therefore makes it possible to verify profiles for consistency and completeness. Let's give that a try. First of all we need to run the latest main version of CRI-O, because v1.26.0 has not been released yet at time of writing. You can do that by either compiling it from the source code or by using the pre-built binary bundle via the get-script . The seccomp notifier feature of CRI-O is guarded by an annotation, which has to be explicitly allowed, for example by using a configuration drop-in like this: cat /etc/crio/crio.conf.d/02-runtimes.conf [crio.runtime] default_runtime = "runc" [crio.runtime.runtimes.runc] allowed_annotations = [ "io.kubernetes.cri-o.seccompNotifierAction" ] If CRI-O is up and running, then it should indicate that the seccomp notifier is available as well: sudo ./bin/crio --enable-metrics … INFO[…] Starting seccomp notifier watcher INFO[…] Serving metrics on :9090 via HTTP … We also enable the metrics, because they provide additional telemetry data about the notifier. Now we need a running Kubernetes cluster for demonstration purposes. For this demo, we mainly stick to the hack/local-up-cluster.sh approach to locally spawn a single node Kubernetes cluster. If everything is up and running, then we would have to define a seccomp profile for testing purposes. But we do not have to create our own, we can just use the RuntimeDefault profile which gets shipped with each container runtime. For example the RuntimeDefault profile for CRI-O can be found in the containers/common library. Now we need a test container, which can be a simple nginx pod like this: apiVersion : v1 kind : Pod metadata : name : nginx annotations : io.kubernetes.cri-o.seccompNotifierAction : "stop" spec : restartPolicy : Never containers : - name : nginx image : nginx:1.23.2 securityContext : seccompProfile : type : RuntimeDefault Please note the annotation io.kubernetes.cri-o.seccompNotifierAction , which enables the seccomp notifier for this workload. The value of the annotation can be either stop for stopping the workload or anything else for doing nothing else than logging and throwing metrics. Because of the termination we also use the restartPolicy: Never to not automatically recreate the container on failure. Let's run the pod and check if it works: kubectl apply -f nginx.yaml kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx 1/1 Running 0 3m39s 10.85.0.3 127.0.0.1 none none We can also test if the web server itself works as intended: curl 10.85.0.3 !DOCTYPE html html head titleWelcome to nginx!/title … While everything is now up and running, CRI-O also indicates that it has started the seccomp notifier: … INFO[…] Injecting seccomp notifier into seccomp profile of container 662a3bb0fdc7dd1bf5a88a8aa8ef9eba6296b593146d988b4a9b85822422febb … If we would now run a forbidden syscall inside of the container, then we can expect that the workload gets terminated. Let's give that a try by running chroot in the containers namespaces: kubectl exec -it nginx -- bash root@nginx:/# chroot /tmp chroot: cannot change root directory to '/tmp': Function not implemented root@nginx:/# command terminated with exit code 137 The exec session got terminated, so it looks like the container is not running any more: kubectl get pods NAME READY STATUS RESTARTS AGE nginx 0/1 seccomp killed 0 96s Alright, the container got killed by seccomp, do we get any more information about what was going on? kubectl describe pod nginx Name: nginx … Containers: nginx: … State: Terminated Reason: seccomp killed Message: Used forbidden syscalls: chroot (1x) Exit Code: 137 Started: Mon, 14 Nov 2022 12:19:46 +0100 Finished: Mon, 14 Nov 2022 12:20:26 +0100 … The seccomp notifier feature of CRI-O correctly set the termination reason and message, including which forbidden syscall has been used how often (1x ). How often? Yes, the notifier gives the application up to 5 seconds after the last seen syscall until it starts the termination. This means that it's possible to catch multiple forbidden syscalls within one test by avoiding time-consuming trial and errors. kubectl exec -it nginx -- chroot /tmp chroot: cannot change root directory to '/tmp': Function not implemented command terminated with exit code 125 kubectl exec -it nginx -- chroot /tmp chroot: cannot change root directory to '/tmp': Function not implemented command terminated with exit code 125 kubectl exec -it nginx -- swapoff -a command terminated with exit ...

Recap: Best practices for a container management platform In this chalk talk, we learned about five best practices for building a container platform management system. This talk covered key steps that help drive efficiency and effectiveness, from assembling the right teams to innovate, design, and operate your container management platform to choosing the right AWS and ISV tools. Behind the scenes: Making the diagrams I used a tool called Excalidraw to make the diagrams we used in our chalktalk.

A Sigstore section was added to the Open Source Security Foundation (OpenSSF)’s Landscape.