Infrastructure
A look at Kubernetes Operator Implementation at Licious — Part 2
Implementing Kubernetes Operators for Streamlined Application Management
In Part 1 of the article, we discussed the fundamental concept of operators in programming. we laid the groundwork by explaining what operators are, their significance in programming, and the diverse range of tasks they can accomplish.
Now, let’s delve deeper into how we implemented the Operator at Licious to automate the management of custom resources.
Building an Operator at Licious
At Licious, we recognized the need for automating the management of our Kubernetes-based applications. To achieve this, we decided to implement a Kubernetes Operator tailored to our specific requirements. Let’s walk through the key components of our implementation:
1. Choosing the Right Framework:
When embarking on our journey to develop the Kubernetes Operator, we evaluated several frameworks and libraries available in the Kubernetes ecosystem. After careful consideration, we opted to build our Operator using the Java programming language, leveraging the Kubernetes Java client library for seamless interaction with the Kubernetes API. Go is another good alternative to Java.
2. Project Structure and Components:
Our Operator project follows a modular structure, comprising several key components:
Main Class (LiciousOperatorApplication): The entry point of our Operator application, responsible for initializing the Operator, leader election and starting the reconciliation loop.
In the main method, we initialise the Kubernetes client and create an Operator instance. We specify leader election configuration to ensure high availability and reliability. Next, we initialise dependent resources such as Deployments, Autoscalers, Pod Disruption Budgets (PDBs), Secrets, Services, and Service Monitors. Finally, we register our custom reconciler and start the Operator.
Dependent Resource Handlers: Custom classes responsible for managing specific dependent resources, such as Deployment,Service etc. Each dependent resource contains a constructor that registers the dependent resource with the reconciler. These handlers encapsulate the logic for creating, updating, or deleting dependent resources based on changes detected in the Liciousapps resource. Each
Reconciler (LiciousappsReconciler): Handles the reconciliation logic for the Liciousapps custom resource, ensuring that the desired state specified in the resource is enforced within the cluster.
In our implementation, dependent resources such as Deployments, Autoscalers, Pod Disruption Budgets (PDBs), Secrets, Services, and Service Monitors act as observers. They register themselves with the reconciler, which acts as the subject. Let’s explore how this works:
Dependency Registration: To facilitate dependency management, our reconciler provides a method registerDependentResource that allows dependent resources to register themselves with the reconciler. This method is typically called during the initialization phase of each dependent resource. By maintaining a list of dependent resources (k8sDR), the reconciler can efficiently orchestrate the reconciliation process, ensuring that each dependent resource is synchronized with the desired state of the custom resource.
In the LiciousappsReconciler class, we maintain a list of dependent resources. During initialization, each dependent resource is instantiated and registered with the reconciler using the registerDependentResource method.
Each dependent resource extends the KubernetesDependentResource class, which requires them to implement the desired method. This method encapsulates the logic for managing the respective Kubernetes resource based on the desired state defined in the custom resource (Liciousapps).
Reconciliation and Dependency Management: In the context of Kubernetes Operators, reconciliation is the process by which the Operator ensures that the actual state of resources matches the desired state specified in the custom resources. To achieve this, our Operator reconciler implements the reconcile method, which orchestrates the reconciliation process for the custom resources it manages.
Reconciliation Method: The reconcile method is called periodically by the Operator to synchronize the state of resources. Within this method, we first ensure that the status field of the custom resource is initialized. Then, we add the identity of the reconciler to the list of entities responsible for reconciliation, ensuring visibility into the reconciliation process. Next, we iterate over each registered dependent resource (k8sDR) and invoke the reconcile method on each one, passing the custom resource and context as parameters. This allows each dependent resource to perform its reconciliation logic based on the current state of the custom resource.
Key Features Enabled by Operators for Dependent Resources:
Dynamic Resource Provisioning: Operators can dynamically provision and configure dependent resources based on the desired state specified in custom resources.
Automated Updates: Operators automatically detect changes to custom resources and initiate updates to dependent resources accordingly, ensuring they remain in sync with the application’s requirements.
Error Handling: Operators handle errors gracefully, allowing for effective troubleshooting and remediation in case of resource creation or update failures.
Customization: Operators can be customized to support various types of dependent resources, enabling tailored management of applications within Kubernetes clusters.
The Reconciliation Loop
While this example focused on a single reconciler method, it’s crucial to understand that the reconciler runs continuously as a control loop. The Operator will keep watching for changes to the Liciousapps resources, re-evaluating the current state, and reconciling the Mapping resources as needed.
This allows the Operator to constantly ensure that the actual state converges with the desired state specified by the Liciousapps resources, providing self-healing capabilities. If a custom resource is ever modified unexpectedly, the reconciler will detect the drift from the desired state and correct it on the next reconciliation cycle.
Event-Driven Architecture
Our Operator follows an event-driven architecture, where changes to custom resources trigger corresponding actions within the system. For example, when a new Liciousapps resource is created or updated, the reconciliation loop is invoked, and dependent resource handlers are triggered to synchronize the cluster's state with the desired configuration.
Error Handling and Resilience
To ensure the reliability and robustness of our Operator, we have implemented comprehensive error handling mechanisms. Any errors encountered during the reconciliation process are logged and handled gracefully, preventing cascading failures and ensuring the continued operation of the Operator.
3. Deploying Kubernetes Operators
Deploying Kubernetes Operators involves defining Custom Resource Definitions (CRDs), creating Custom Resources (CRs), and deploying the Operator itself. These are typically done using YAML files:
Custom Resource Definitions (CRDs): CRDs define the schema for custom resources that the Operator manages. They specify the structure and validation rules for custom resources. Example YAML file:
# Generated by Fabric8 CRDGenerator, manual edits might get overwritten!
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: liciousapps.licious.app
spec:
group: licious.app
names:
kind: Liciousapps
plural: liciousapps
singular: liciousapps
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
properties:
spec:
properties:
DeploymentSpec:
properties:
ContainerSpecs:
properties:
image:
properties:
pullPolicy:
type: string
repository:
type: string
tag:
type: string
type: object
port:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
resources:
properties:
limits:
properties:
cpu:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
memory:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
type: object
requests:
properties:
cpu:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
memory:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
type: object
type: object
secretName:
type: string
type: object
autoscaling:
properties:
enabled:
type: boolean
maxReplicas:
type: integer
minReplicas:
type: integer
name:
type: string
targetCPUUtilizationPercentage:
type: integer
type: object
poddisruptionbudget:
properties:
maxUnavailable:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
name:
type: string
type: object
pvc:
items:
properties:
mountPath:
type: string
name:
type: string
persistentVolumeClaim:
properties:
claimName:
type: string
type: object
type: object
type: array
serviceAccount:
properties:
annotations:
additionalProperties:
type: string
type: object
create:
type: boolean
name:
type: string
role_arn:
type: string
type: object
strategy:
properties:
rollingUpdate:
properties:
maxSurge:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
maxUnavailable:
anyOf:
- type: integer
- type: string
x-kubernetes-int-or-string: true
type: object
type:
type: string
type: object
volumes:
items:
properties:
configMap:
properties:
items:
properties:
key:
type: string
path:
type: string
type: object
name:
type: string
type: object
mountPath:
type: string
name:
type: string
type: object
type: array
type: object
GatewaySpec:
properties:
mapping:
properties:
spec:
properties:
cors:
properties:
headers:
type: string
methods:
type: string
origins:
type: string
type: object
host:
type: string
prefix:
type: string
timeout_ms:
type: integer
type: object
type: object
service:
properties:
port:
type: integer
type:
type: string
type: object
type: object
type: object
status:
properties:
reconciledBy:
items:
type: string
type: array
type: object
type: object
served: true
storage: true
subresources:
status: {}Custom Resources (CRs): CRs are instances of custom resources defined by CRDs. They represent the desired state of resources that the Operator manages. Example YAML file:
apiVersion: licious.app/v1alpha1
kind: Liciousapps
metadata:
name: helloworld
namespace: operator-test
spec:
imageRef: hello-world-app:latestOperator Deployment: The Operator is typically deployed as a Kubernetes deployment or StatefulSet. Example YAML file for deploying the Operator:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: licious-operator-deployment
namespace: licious-operator
labels:
k8s-app: licious-operator
spec:
progressDeadlineSeconds: 420
revisionHistoryLimit: 1
replicas: 1
selector:
matchLabels:
k8s-app: licious-operator
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
k8s-app: licious-operator
spec:
serviceAccountName: licious-operator-sa
containers:
- name: licious-operator-dev
image: licious-operator:v16
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: http
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 60
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 60
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 250m
memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: licious-operator-service
namespace: licious-operator
labels:
k8s-svc: licious-operator
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
k8s-app: licious-operator
type: ClusterIP
Now that we have a working Operator in place, it’s essential to monitor its performance, validate its functionality, and continue iterating on its capabilities to meet evolving requirements. Through testing, tuning, and documentation, we can ensure that the Operator operates seamlessly within our environment and delivers tangible benefits in terms of efficiency, reliability, and agility.
The true power of the reconciler pattern lies in its ability to abstract away low-level Kubernetes API details. Rather than dealing directly with Deployments, Services, and other built-in resource types, this Operator manages the application lifecycle through higher-level, domain-specific custom resources like Liciousapps and Mapping or Ingress.Developers can now declare what they want the desired state to be using these custom resources. The Operator’s reconciler, coupled with custom resource definitions, handles the tedious tasks of creating, updating, and managing the underlying Kubernetes resources required to materialize that desired state.
The example we walked through today is just the tip of the iceberg. As you create your own Operators, you’ll find the reconciler pattern becoming an indispensable tool for encoding operational knowledge, enforcing conventions, and providing a simpler avenue for managing applications on Kubernetes.
Additional Capabilities of Kubernetes Operators
In addition to the functionalities discussed, Kubernetes Operators offer a plethora of capabilities that empower organizations to streamline application management within Kubernetes clusters:
Custom Resource Management: Operators can manage custom resources tailored to specific application requirements, enabling organizations to define and enforce complex configurations with ease.
Stateful Application Management: Operators excel in managing stateful applications by automating tasks such as data migration, replication, and backup, ensuring data integrity and high availability.
Multi-Cluster Management: Operators can extend their reach across multiple Kubernetes clusters, providing centralized management and orchestration capabilities for distributed applications.
Integration with Ecosystem Tools: Operators can integrate seamlessly with ecosystem tools such as monitoring, logging, and CI/CD pipelines, enhancing observability and enabling end-to-end automation workflows.
Policy Enforcement: Operators can enforce policies and compliance standards within Kubernetes clusters, ensuring adherence to regulatory requirements and organizational guidelines.
In conclusion, Kubernetes Operators are invaluable tools for automating application management tasks, enhancing scalability, and ensuring reliability within Kubernetes clusters. By leveraging Operators, organizations can achieve higher efficiency, lower operational overheads, and enhanced agility in managing complex applications. As we continue to harness the power of Operators at Licious, we look forward to unlocking new possibilities and delivering exceptional experiences to our customers.
Thank you for joining us on this journey!
