Fernando SRE

‘Kubectl edit’ is an indispensable command-line tool for Kubernetes users, offering the flexibility to modify resource definitions dynamically. This article aims to demystify ‘Kubectl edit,’ explaining its utility and showcasing real-world applications.

What is ‘Kubectl Edit’?

‘Kubectl edit’ is a command that facilitates live edits to Kubernetes resource configurations, such as pods, services, deployments, or other resources defined in YAML files. It’s akin to wielding a magic wand, allowing you to tweak your resources without the hassle of creating or applying new configuration files.

Why is ‘Kubectl Edit’ Valuable?

• Real-Time Configuration Changes: It enables immediate adjustments, making it invaluable for troubleshooting or adapting to evolving requirements.

• Quick Fixes: It’s perfect for addressing issues or misconfigurations swiftly, without the need for resource deletion and recreation.

• Experimentation: Ideal for experimenting and fine-tuning settings, helping you discover the best configurations for your applications.

Basic Syntax

The basic syntax for ‘Kubectl edit’ is straightforward:

kubectl edit <resource-type> <resource-name>

• <resource-type>: The type of Kubernetes resource you want to edit (e.g., pod, service, deployment).
• <resource-name>: The name of the specific resource to edit.

Real-Life Examples

Let’s explore practical examples to understand how ‘Kubectl edit’ can be effectively utilized in real scenarios.

Example 1: Modifying a Pod Configuration

Imagine needing to adjust the resource requests and limits for a pod named ‘my-pod.’ Execute:

kubectl edit pod my-pod

This command opens the pod’s configuration in your default text editor. You’ll see a YAML file similar to this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  ...
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  ...

In this file, focus on the resources section under containers. Here, you can modify the CPU and memory settings. For instance, to increase the CPU request to 500m and the memory limit to 256Mi, you would change the lines to:

   resources:
      requests:
        memory: "64Mi"
        cpu: "500m"
      limits:
        memory: "256Mi"
        cpu: "500m"

After making these changes, save and close the editor. Kubernetes will apply these updates to the pod ‘my-pod.’

Example 2: Updating a Deployment

To modify a deployment’s replicas or image version, use ‘kubectl edit’:

kubectl edit deployment my-deployment

This command opens the deployment’s configuration in your default text editor. You’ll see a YAML file similar to this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  ...
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: my-image:v1.0
        ports:
        - containerPort: 80
  ...

In this file, focus on the following sections:

• To change the number of replicas, modify the replicas field. For example, to scale up to 5 replicas:

spec:
  replicas: 5

• To update the image version, locate the image field under containers. For instance, to update to version v2.0 of your image:

 containers:
    - name: my-container
      image: my-image:v2.0

Save and close the editor and Kubernetes will apply these updates to the deployment ‘my-deployment.’

Example 3: Adjusting a Service

To fine-tune a service’s settings, such as changing the service type to LoadBalancer, you would use the command:

kubectl edit service my-service

The command will open the service’s configuration in your default text editor. You’ll likely see a YAML file similar to this:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  ...
spec:
  type: ClusterIP
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
  ...

In this file, focus on the spec section:

• To change the service type to LoadBalancer, modify the type field. For example:

spec:
  type: LoadBalancer

This change will alter the service type from ClusterIP to LoadBalancer, enabling external access to your service.

After making these changes, save and close the editor. Kubernetes will apply these updates to the service ‘my-service.’

Real-World Example: Debugging and Quick Fixes

If a pod is crashing due to a misconfigured environment variable, something I’ve seen happen countless times, use ‘kubectl edit’ to quickly access and correct the pod’s configuration, significantly reducing the downtime.

kubectl edit pod crashing-pod

Executing such a command will open the pod’s configuration in your default text editor, and you’ll likely see a YAML file similar to this:

apiVersion: v1
kind: Pod
metadata:
  name: crashing-pod
  ...
spec:
  containers:
  - name: my-container
    image: my-image
    env:
      - name: ENV_VAR
        value: "incorrect_value"
    ...

In this file, focus on the env section under containers. Here, you can find and correct the misconfigured environment variable. For instance, if ENV_VAR is set incorrectly, you would change it to the correct value:

    env:
      - name: ENV_VAR
        value: "correct_value"

After making this change, save and close the editor. Kubernetes will apply the update, and the pod should restart without the previous configuration issue.

It’s Not Magic: Understand What Happens Behind the Scenes

When you save changes made in the editor through kubectl edit, Kubernetes doesn’t simply apply these changes magically. Instead, a series of orchestrated steps occur to ensure that your modifications are implemented correctly and safely. Let’s demystify this process:

1. Parsing and Validation: First, Kubernetes parses the edited YAML or JSON file to ensure it’s correctly formatted. It then validates the changes against the Kubernetes API’s schema for the specific resource type. This step is crucial to prevent configuration errors from being applied.

2. Resource Versioning: Kubernetes keeps track of the version of each resource configuration. When you save your changes, Kubernetes checks the resource version in your edited file against the current version in the cluster. This is to ensure that you’re editing the latest version of the resource and to prevent conflicts.

3. Applying Changes: If the validation is successful and the version matches, Kubernetes proceeds to apply the changes. This is done by updating the resource’s definition in the Kubernetes API server.

4. Rolling Updates and Restart: Depending on the type of resource and the nature of the changes, Kubernetes may perform a rolling update. For example, if you edited a Deployment, Kubernetes would start creating new pods with the updated configuration and gradually terminate the old ones, ensuring minimal disruption.

5. Feedback Loop: Kubernetes controllers continuously monitor the state of resources. If the applied changes don’t match the desired state (for instance, if a new pod fails to start), Kubernetes attempts to rectify this by reverting to a previous, stable configuration.

6. Status Update: Finally, the status of the resource is updated to reflect the changes. You can view this status using commands like ‘kubectl get’ or ‘kubectl describe’ to ensure that your changes have been applied and are functioning as expected.

By understanding these steps, you gain insight into the robust and resilient nature of Kubernetes’ configuration management. It’s not just about making changes; it’s about making them in a controlled, reliable manner.

In Closing

‘Kubectl edit’ is a powerful tool for optimizing Kubernetes resources, offering simplicity and efficiency. With the examples provided, you’re now equipped to confidently fine-tune settings, address issues, and experiment with configurations, ensuring the smooth and efficient operation of your Kubernetes applications.

In the Kubernetes realm, ensuring the security and proper assignment of roles and permissions within a cluster is paramount. Kubernetes administrators often face the challenge of verifying whether a particular user or service account has the necessary permissions to act. This is where the often underutilized ‘kubectl auth can-i’ command comes into play, offering a straightforward way to check access rights without executing the actual operation.

The Power of ‘kubectl auth can-i’: 

The ‘kubectl auth can-i’ command is a part of the Kubernetes command-line tool that allows you to query the Kubernetes RBAC (Role-Based Access Control) to check if a user, group, or service account can perform a specific action. It’s an invaluable command for DevOps engineers and Kubernetes administrators to verify and troubleshoot permissions.

Basic Usage: To use ‘kubectl auth can-i’, you simply follow the syntax:

kubectl auth can-i <verb> <resource>

For example, if you want to check if your current user can create deployments in the default namespace, you would use:

kubectl auth can-i create deployments

Checking Permissions for a Service Account: 

Let’s say you have a service account named ‘monitoring-sa’ in the ‘monitoring’ namespace, and you want to check if it has permission to list endpoints (which is crucial for service discovery in monitoring solutions). You could run:

kubectl auth can-i list endpoints --namespace monitoring --as system:serviceaccount:monitoring:monitoring-sa

This command will return ‘yes’ if the service account has the required permissions, or ‘no’ if it does not.

Advanced Examples: 

You can also use ‘kubectl auth can-i’ to check permissions for verbs that are not part of the standard Kubernetes API. For instance, if you’re using Custom Resource Definitions (CRDs), you can check permissions for these as well:

kubectl auth can-i create mycustomresources.mydomain.com --as system:serviceaccount:monitoring:monitoring-sa

Besides checking permissions for standard Kubernetes API actions and Custom Resource Definitions (CRDs), ‘kubectl auth can-i’ can be used to verify permissions for specific API subresources. Subresources like ‘status’ and ‘scale’ are important for certain operations and can have separate permissions from the main resource.

For instance, if you want to check if a service account has the permission to update the status of a deployment, which is a common requirement for continuous deployment setups, you could use:

kubectl auth can-i update deployments/status --namespace production --as system:serviceaccount:default:deploy-sa

This command will check if the ‘deploy-sa’ service account in the ‘default’ namespace has permission to update the status subresource of deployments in the production namespace.

Another advanced use case is checking permissions for pod exec, which is crucial for debugging:

kubectl auth can-i create pod/exec --namespace development --as system:serviceaccount:default:debug-sa

This will verify if the ‘debug-sa’ service account in the ‘default’ namespace is allowed to execute commands in pods within the ‘development’ namespace. This is particularly useful when setting up service accounts for CI/CD pipelines that need to perform diagnostic commands in running pods.

By providing multiple advanced examples, we can demonstrate the versatility of the ‘kubectl auth can-i’ command in managing complex permission scenarios in Kubernetes.

Looking Ahead: 

The ‘kubectl auth can-i’ command is a simple yet powerful tool to help manage and verify permissions within a Kubernetes cluster. It’s an essential command for anyone responsible for the security and integrity of their Kubernetes environment.

Remember, always verify before you deploy!

Kubernetes offers robust tools for managing application configurations and safeguarding sensitive data: ConfigMaps and Secrets. This article provides hands-on examples to help you grasp these concepts.

What are ConfigMaps?

ConfigMaps in Kubernetes are designed to manage non-sensitive configuration data. They are generally created using YAML files that specify the configuration parameters.

Example: Environment Variables

Consider an application that requires a database URL and an API key. You can use a ConfigMap to set these as environment variables. Here’s a sample YAML file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_URL: jdbc:mysql://localhost:3306/db
  API_KEY: key123

Mounting ConfigMaps as Volumes

ConfigMaps can also be mounted as volumes, making them accessible to pods as files. This is useful for configuration files or scripts.

Example: Mount as Volume

To mount a ConfigMap as a volume, you can modify the pod specification like this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
  volumes:
  - name: config-volume
    configMap:
      name: app-config

What are Secrets?

Secrets are used for storing sensitive information like passwords and API tokens securely. It’s important to note that the data in Secrets should be encoded in base64 for an added layer of security.

Example: Secure API Token

To store an API token securely, you can create a Secret like this:

apiVersion: v1
kind: Secret
metadata:
  name: api-secret
data:
  API_TOKEN: base64_encoded_token

To generate a base64-encoded token, you can use the following command:

echo -n 'your_actual_token' | base64

In Summary

ConfigMaps and Secrets are indispensable tools in Kubernetes for managing configuration data and sensitive information. Understanding how to use them effectively is crucial for any Kubernetes deployment.

Kubernetes has become a cornerstone in the container orchestration world, and being adept at maneuvering through the Kubernetes environment is crucial for DevOps professionals.

Among the various tools at our disposal, kubectl stands out as an essential command-line tool for interacting with clusters.

kubectl exec

The kubectl exec command is utilized to run commands in a specific container within a Pod.

When you execute kubectl exec, it creates a new terminal session inside the container which allows for both interactive and non-interactive command execution.

Example: Suppose you have a running Pod hosting a web service and you wish to check the contents of a specific directory. You could use kubectl exec to run the ls command in the container, listing the files in that directory.

kubectl attach

On the other hand, kubectl attach allows you to attach to a running process within a container.

Unlike kubectl exec, kubectl attach connects to an existing terminal session, allowing you to observe the standard output and error of the running process.

Example: If you have a Pod running an application that writes logs to standard output, you could use kubectl attach to view these logs in real time.

Summarizing:

While kubectl exec spawns a new terminal session, kubectl attach connects to an existing session.

kubectl exec is more versatile for executing arbitrary commands, whereas kubectl attach is useful for interacting with running processes and observing their real-time behavior.

The key takeaway is understanding when to use kubectl exec versus kubectl attach based on the task at hand.

Kubernetes has made a significant impact as a container orchestration tool, but it’s crucial to understand that its utility doesn’t end there. One of its most compelling features is the ability to extend its API with Custom Resource Definitions (CRDs), especially since Kubernetes version 1.7. This article delves into what CRDs are, why they are essential, and how they work in conjunction with controllers to simplify complex tasks within a Kubernetes cluster.

.- What are Custom Resource Definitions (CRDs)?

CRDs act as extensions of the Kubernetes API, allowing users to create new types of resources without adding another API server. Simply put, CRDs serve as vehicles to extend the Kubernetes ecosystem. They are vital for enriching Kubernetes functionalities beyond its basic scope.

.- Basic Kubernetes Functionality Without CRDs

In an out-of-the-box Kubernetes setup, users can define deployments that spawn replica sets, which in turn create pods for running containers. Users can also set up services and ingress controllers for network access to these containers. However, this native functionality has limitations, such as lacking an in-built storage solution.

.- The Need for Extending Kubernetes

The real power of Kubernetes lies in its extensibility. Almost every third-party tool or service designed for Kubernetes operates through CRDs, which extend your cluster’s functionalities. For instance, if you are implementing a service mesh like Istio, it will extend your cluster with several CRDs like VirtualServices and Gateways.

.- The Role of Controllers in CRDs

CRDs by themselves are not functional. When a custom resource is created, the Kubernetes API only signals an event stating that a resource is created. Controllers respond to these events. They watch for specific changes to custom resources and take action accordingly, thereby breathing life into CRDs.

.- Benefits of Using CRDs and Controllers

The duo of CRDs and controllers can significantly simplify many tasks. They move the heavy lifting from the client-side to the server-side, reducing complexity and eliminating the need for client-side templating solutions. As a result, end-users can define their applications in a more Kubernetes-native way, without diving into the lower-level details.

Example: Creating a Simple CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myresources.mycompany.com
spec:
  group: mycompany.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                key:
                  type: string

.- Understanding Kubernetes Operators

After we’ve laid down the groundwork by discussing CRDs and controllers, it’s essential to touch upon Kubernetes Operators, which bring the two together in a well-organized manner. In essence, an Operator is a method of packaging, deploying, and managing a Kubernetes application.

An Operator extends Kubernetes to automate the management of the entire lifecycle of a particular application, API, or resource. It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate common tasks. For example, an Operator could manage a database cluster, handling tasks such as backups, updates, and scaling.

.- Using an Operator to Manage a Database Cluster

Imagine you have a PostgreSQL database running within your Kubernetes cluster. You could deploy a PostgreSQL Operator that automatically handles routine tasks like backups, updates, or even scaling. This Operator would use CRDs to understand custom resources that define the desired state for these tasks and use a controller to ensure the current state matches the desired state.

Here is a simplified YAML example defining a PostgresCluster custom resource, which could be managed by a PostgreSQL Operator:

Example: Using an Operator to Manage a Database Cluster

apiVersion: postgresql.org/v1
kind: PostgresCluster
metadata:
  name: my-postgres-cluster
spec:
  replicas: 3
  version: "12"
  backup:
    enabled: true
    schedule: "0 0 * * *"

.- Recommendations and Best Practices

While CRDs and controllers offer immense utility, it’s critical to rely on established practices and tools when implementing them. Do not attempt to write your controllers manually unless you are very experienced; instead, rely on tools like Operator SDK for creating operators for your CRDs.

.- Wrapping Up

We’ve explored the powerful features Kubernetes provides beyond its basic functionalities. Custom Resource Definitions (CRDs) allow us to extend the Kubernetes API, enabling more tailored operations within our cluster. Controllers breathe life into these CRDs by reacting to events and ensuring that the state of our resources aligns with our specifications.

Moreover, we touched upon Kubernetes Operators, which encapsulate both CRDs and controllers, along with domain-specific logic, to manage complex applications effortlessly. Operators serve as the cherry on top in the Kubernetes extensibility model, automating routine tasks and simplifying cluster management even further.

By embracing CRDs, controllers, and operators, we can exploit Kubernetes’ full potential and create an environment that’s not only flexible but also significantly easier to manage. As Kubernetes continues to evolve, leveraging these elements will undoubtedly make our journey in the cloud-native world much smoother.

The Horizontal Pod Autoscaler (HPA) is pivotal in Kubernetes. It’s like our trusty assistant, automatically adjusting the number of pods in a deployment according to observed metrics like CPU usage. However, there are moments when it encounters hurdles. One such instance is when you stumble upon error messages such as:

Name:                                                  widget-app-sun
Namespace:                                             development
...
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 55%
Min replicas:                                          1
Max replicas:                                          3
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age                 From                       Message
  ----     ------                        ----                ----                       -------
  Warning  FailedComputeMetricsReplicas  20m (x20 over 9m)   horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       12s (x29 over 9m)   horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API

What’s the Scoop?

This cryptic message is essentially HPA’s way of saying, “I’m having a hard time fetching those CPU metrics I need.” But why? Here are a few culprits:

Perhaps Metrics-server isn’t installed or isn’t operating correctly.
Maybe Metrics-server is present, but it’s struggling to fetch metrics from the nodes.
It could be a misconfiguration on HPA’s end.
Sometimes, network policies or RBAC restrictions come into play, obstructing access to the metrics API.

The Detective Work: Troubleshooting Steps

.- Is Metrics-server Onboard?

kubectl get deployments -n kube-system | grep metrics-server

metrics-server           1/1     1            1           221d

If it’s missing in action, it’s time to deploy it. Helm is a handy tool for this. https://artifacthub.io/packages/helm/metrics-server/metrics-server

.- How’s Metrics-server Feeling Today?

kubectl get pods -n kube-system -l k8s-app=metrics-server

NAME                              READY   STATUS    RESTARTS      AGE
metrics-server-5f9f776df5-zlg42   1/1     Running   6 (71d ago)   221d

Make sure it’s running smoothly. If it’s throwing a tantrum, dive into its logs:

kubectl logs metrics-server-5f9f776df5-zlg42 -n kube-system

I0730 17:07:50.422754       1 secure_serving.go:266] Serving securely on [::]:10250
I0730 17:07:50.425140       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0730 17:07:50.425155       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController

.- A Peek into Metrics-server’s Config

Sometimes, it needs some flags to communicate correctly, especially if your cluster has a unique CNI or is lounging on a special cloud provider. You might need to check and adjust flags like:

--kubelet-preferred-address-types or --kubelet-insecure-tls

.- The Network or RBAC Culprits

Are there any stringent network policies that are hindering the conversation between the metrics-server and the API server or the kubelets? Or maybe, metrics-server doesn’t have the right RBAC permissions to access metrics?

Peek into network policies in the kube-system namespace:

kubectl get networkpolicy -n kube-system

And don’t forget to inspect the ClusterRole:

kubectl describe clusterrole | grep metrics-server -A10

Name:         system:metrics-server
Labels:       objectset.rio.cattle.io/hash=9a6f488150c249811b9df07e116280789628963e
Annotations:  objectset.rio.cattle.io/applied:
                H4sIAAAAAAAA/4yRwY6bMBCGX6WasyEhSQkg9VD10ENvPfRScRjsSXABG80Yom7Eu69MotVKq93syRr/+j7711wBR/uHWKx3UAE3qFOcQuvZPmGw3qVdIan1mzkDBZ11Bir40U8SiH...

.- Version Harmony: HPA & Metrics-server

Compatibility matters! Ensure HPA and metrics-server are on the same page. Sometimes, a version mismatch might be the root cause.

Here’s how to check your Kubernetes version:

kubectl version

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

And let’s not forget about the metrics-server:

kubectl describe deployment metrics-server -n kube-system | grep Image:

Image:      rancher/mirrored-metrics-server:v0.6.2

Troubleshooting in Kubernetes can be challenging, but with a systematic approach, many issues, like the HPA metrics problem, can be resolved. It’s essential to understand the components involved and to remain adaptable. As Kubernetes continues to evolve, so too should our methods for diagnosing and fixing problems.