mirror of
https://github.com/apricote/apricote.de.git
synced 2026-01-13 20:51:02 +00:00
update post for multiple parts
This commit is contained in:
parent
fad04ff7e0
commit
c97e19779c
7 changed files with 445 additions and 379 deletions
66
content/posts/ccm-from-scratch/_index.md
Normal file
66
content/posts/ccm-from-scratch/_index.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
author: "Julian Tölle"
|
||||
title: "Building a Kubernetes cloud-controller-manager"
|
||||
date: "2023-09-06"
|
||||
summary: |
|
||||
A tutorial series to showcase the steps needed to build a Kubernetes Cloud Controller Manager.
|
||||
Intended to be your guide into better Kubernetes-compatibility for your own Infrastructure-as-a-Service Cloud.
|
||||
description: |
|
||||
This tutorial series tries to showcase the steps needed to build a first iteration of a Kubernetes Cloud Controller Manager.
|
||||
It can be adapted to any Cloud Provider and may be your guide into better Kubernetes-compatibility for your own Infrastructure-as-a-Service Cloud.
|
||||
tags: ["kubernetes"]
|
||||
ShowBreadCrumbs: true
|
||||
cover:
|
||||
image: cover.webp
|
||||
responsiveImages: true
|
||||
caption: dall-e with prompt "ship stearing towards clouds, a captain behind the steering wheel, cartoon, wide angle"
|
||||
---
|
||||
|
||||
|
||||
|
||||
{{< collapse summary="Disclaimers" openByDefault=true >}}
|
||||
##### Employer
|
||||
|
||||
I [work for Hetzner Cloud]({{< ref "/work" >}}) and this project was done during working hours as part of the _Lab Days_.
|
||||
During Lab Days, employees can choose to work on topics unrelated to their work.
|
||||
|
||||
##### Production Readiness
|
||||
|
||||
The code written in this series is not ready for production.
|
||||
You should not actually use the code I provide for your Kubernetes Cluster on Hetzner Cloud.
|
||||
Please use the official [`hcloud-cloud-controller-manager`](https://github.com/hetznercloud/hcloud-cloud-controller-manager) instead.
|
||||
{{</ collapse >}}
|
||||
|
||||
## Introduction
|
||||
|
||||
### What is a Cloud Controller Manager?
|
||||
|
||||
[Source](https://kubernetes.io/docs/concepts/architecture/cloud-controller/)
|
||||
|
||||
- Controllers: Loop that reconciles reality with the desired state
|
||||
- Controller Manager: Binary that runs a bunch of controllers
|
||||
- Cloud Controller Manager: Part of the Kubernetes Control Plane, responsible for exposing Cloud Provider functionality inside the Kubernetes cluster.
|
||||
- Controllers in CCMs:
|
||||
- Node Metadata
|
||||
- Match Kubernetes `Node` to Cloud API Server (/Instance/Droplet/...) and set `spec.providerID`
|
||||
- Label Kubernetes `Nodes` with metadata of the Node, including the [Instance Type](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type), Topology ([Region](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion), [Zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone))
|
||||
- Figure out the Network addresses of the `Node` and set them in `status.addresses`.
|
||||
- Node Lifecycle
|
||||
- The `kubelet` is usually responsible for creating the `Node` object in Kubernetes on its first startup.
|
||||
- This does not work well for the removal of the node, as the `kubelet` might not get a chance to do this. Instead, the Node Lifecycle controller regularly checks if the Node was deleted in the Cloud API, and if it was it also deletes the `Node` object in Kubernetes.
|
||||
- Service
|
||||
- Watches `Service` objects with `type: LoadBalancer`. Creates Cloud Load Balancers and configures them according to the `Service` and `Node` objects
|
||||
- Route
|
||||
- In Kubernetes, every Pod gets an IP address. This IP address needs to be available from every other pods network (by default). This is usually implemented as an Overlay Network through your CNI like Calico or Cilium.
|
||||
- If you are already using some kind of Private Networking functionality of your Cloud Provider, then you can use this instead to get rid of the additional Overlay Network and let the Cloud Provider handle the connectivity.
|
||||
- This is implemented by configuring Routes in the Private Network Routing Table that send all traffic to the `Nodes` `spec.podCIDR` to the `Nodes` private IP.
|
||||
|
||||
### What is the Hetzner Cloud?
|
||||
|
||||
[Source](https://hetzner.com/cloud)
|
||||
|
||||
- IaaS with: Servers, Load Balancers, Private Networks
|
||||
|
||||
### Parts
|
||||
|
||||
This project got way longer than I expected, so I split everything up into 5 parts:
|
||||
|
Before Width: | Height: | Size: 345 KiB After Width: | Height: | Size: 345 KiB |
204
content/posts/ccm-from-scratch/part-1.md
Normal file
204
content/posts/ccm-from-scratch/part-1.md
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
---
|
||||
author: "Julian Tölle"
|
||||
title: "Part 1: Setting up the Boilerplate"
|
||||
date: "2023-09-06"
|
||||
description: |
|
||||
Goal: Get a compilable binary running k/cloud-provider with no controllers yet
|
||||
summary: |
|
||||
Goal: Get a compilable binary running k/cloud-provider with no controllers yet
|
||||
|
||||
tags: ["kubernetes"]
|
||||
ShowToc: true
|
||||
ShowReadingTime: true
|
||||
ShowBreadCrumbs: true
|
||||
---
|
||||
|
||||
This is part 1 out of 5 of the series [Building a Kubernetes cloud-controller-manager]({{< ref "." >}}).
|
||||
|
||||
### Dependencies
|
||||
|
||||
As all ~~good~~ software does, we [stand on the shoulders of giants](https://xkcd.com/2347/). Let us review the two main dependencies that our code will use:
|
||||
|
||||
#### `k8s.io/cloud-provider`
|
||||
|
||||
> Repository: https://github.com/kubernetes/cloud-provider
|
||||
|
||||
Official Library by Kubernetes Project to implement cloud-controller-managers.
|
||||
Historically part of `kubelet` but extracted a long time ago to build out-of-tree CCMs.
|
||||
Handles communication with Kubernetes, CCM implementors only need to implement a narrowly-scoped interface for each functionality.
|
||||
|
||||
> This series will use Kubernetes `v1.28` throughout.
|
||||
> If you use a different version, you may need to adjust your code.
|
||||
> The `k8s.io/cloud-provider` release series associated with Kubernetes `v1.28` is `v0.28.x`.
|
||||
|
||||
#### `github.com/hetznercloud/hcloud-go`
|
||||
|
||||
> Repository: https://github.com/hetznercloud/hcloud-go
|
||||
|
||||
Official Go SDK by Hetzner Cloud, currently maintained by my team.
|
||||
Exposes all functionality of the Hetzner Cloud API in (mostly) idiomatic Go.
|
||||
|
||||
### Repo initialization
|
||||
|
||||
```shell
|
||||
mkdir ccm-from-scratch
|
||||
cd ccm-from-scratch
|
||||
git init
|
||||
|
||||
I like to make an empty initial commit, makes the occasional
|
||||
rebase of the first few commits so much easier.
|
||||
git commit --allow-empty -m "feat: init repo"
|
||||
|
||||
# Optional
|
||||
git remote add git@github.com:apricote/ccm-from-scratch.git
|
||||
git push
|
||||
```
|
||||
|
||||
### `main.go`
|
||||
|
||||
> Source: https://github.com/kubernetes/cloud-provider/tree/v0.28.2/sample
|
||||
|
||||
Thankfully, `k8s.io/cloud-provider` provides an example entrypoint to get us started with out CCM. Lets review the code to make sure we understand what is happening:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
// [Imports]
|
||||
|
||||
// The main function is called as the entrypoint of our Go binary.
|
||||
func main() {
|
||||
// k8s.io/cloud-provider has an elaborate way to read the configuration from flags.
|
||||
// I found this very tedious to debug, but at least we do not have to do this ourselves.
|
||||
ccmOptions, err := options.NewCloudControllerManagerOptions()
|
||||
if err != nil {
|
||||
klog.Fatalf("unable to initialize command options: %v", err)
|
||||
}
|
||||
|
||||
// Can be used to add custom flags to the binary, we dont need this.
|
||||
fss := cliflag.NamedFlagSets{}
|
||||
|
||||
// This initializes the CLI command. The arguments are interesting, so lets take a closer look:
|
||||
command := app.NewCloudControllerManagerCommand(
|
||||
// The options we initialized earlier.
|
||||
ccmOptions,
|
||||
// This is a custom function that needs to return a [cloudprovider.Interface],
|
||||
// we will get to this soon.
|
||||
cloudInitializer,
|
||||
// This defines which controllers are started, if wanted,
|
||||
// one can add additional controller loops heroe
|
||||
app.DefaultInitFuncConstructors,
|
||||
// Kubernetes v1.28 renamed the controllers to more sensible names, this
|
||||
// map makes it so that old command-line arguments (--controllers) still work
|
||||
names.CCMControllerAliases(),
|
||||
// Our optional additional flags
|
||||
fss,
|
||||
// A [<-chan struct{}] that can be used to shut down the CCM on demand,
|
||||
// we do not need this.
|
||||
wait.NeverStop,
|
||||
)
|
||||
// Actually run the command to completion.
|
||||
code := cli.Run(command)
|
||||
os.Exit(code)
|
||||
}
|
||||
```
|
||||
|
||||
Now, this does not compile right now. We use the undefined `cloudInitalizer` method. The method signature we need to implement is `(*config.CompletedConfig) cloudprovider.Interface`. The sample code includes this method, but I found it overly complex for our small CCM, so we will implement it ourselves. Lets take a closer look at the interface we need to return in the next section.
|
||||
|
||||
|
||||
|
||||
|
||||
### `cloudprovider.Interface`
|
||||
|
||||
> Source: https://github.com/kubernetes/cloud-provider/blob/v0.28.2/cloud.go#L42-L69
|
||||
|
||||
This is the entrypoint into the functionality we can (and will!) implement for our CCM. The interface includes an initializer, two cloud-provider metadata methods and a bunch of Getter functions to other interfaces that implement the actual functionality.
|
||||
|
||||
Before we can write the `cloudInitializer` method from above, lets prepare a struct that implements the interface:
|
||||
|
||||
```go
|
||||
package ccm
|
||||
// ccm/cloud.go
|
||||
|
||||
import (
|
||||
cloudprovider "k8s.io/cloud-provider"
|
||||
)
|
||||
|
||||
type CloudProvider struct {}
|
||||
|
||||
// Let's try to assign our struct to a var of the interface we try to implement.
|
||||
// This way we get nice feedback from our IDE if we break the contract.
|
||||
var _ cloudprovider.Interface = CloudProvider{}
|
||||
|
||||
|
||||
// Can be used to setup our own controllers or goroutines that need to talk to
|
||||
//Kubernetes. Not needed for our implementation, so we will leave it empty.
|
||||
func (c CloudProvider) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) {}
|
||||
|
||||
// An arbitrary name for our provider. We will use "ccm-from-scratch" for this example.
|
||||
func (c CloudProvider) ProviderName() string { return "ccm-from-scratch" }
|
||||
|
||||
// I have not yet figured out what Cluster ID is actually supposed to do. We can disable it.
|
||||
func (c CloudProvider) HasClusterID() bool { return false }
|
||||
|
||||
// The following methods all expose additional interfaces. This allows us to
|
||||
// modularly choose what we want to support. As long as we return "false" as the
|
||||
// second argument, the associated functionality is disabled for our CCM.
|
||||
// We will get to implementing [InstancesV2], [LoadBalancer] & [Routes] in later parts of this series.
|
||||
func (c CloudProvider) LoadBalancer() (cloudprovider.LoadBalancer, bool) { return nil, false }
|
||||
func (c CloudProvider) Instances() (cloudprovider.Instances, bool) { return nil, false }
|
||||
func (c CloudProvider) InstancesV2() (cloudprovider.InstancesV2, bool) { return nil, false }
|
||||
func (c CloudProvider) Zones() (cloudprovider.Zones, bool) { return nil, false }
|
||||
func (c CloudProvider) Clusters() (cloudprovider.Clusters, bool) { return nil, false }
|
||||
func (c CloudProvider) Routes() (cloudprovider.Routes, bool) { return nil, false }
|
||||
```
|
||||
|
||||
Now that we have a struct implementing the interface, lets create our `cloudInitializer` method. We will actually do this in the same file as our struct:
|
||||
|
||||
```go
|
||||
// ccm/cloud.go
|
||||
|
||||
// Just create a new struct for now, we will add some more stuff to this later.
|
||||
func NewCloudProvider(_ *config.CompletedConfig) cloudprovider.Interface {
|
||||
return CloudProvider{}
|
||||
}
|
||||
```
|
||||
|
||||
And now we can pass this method in our `main()`:
|
||||
|
||||
```go
|
||||
package main
|
||||
// main.go
|
||||
import (
|
||||
// Existing imports
|
||||
|
||||
"github.com/apricote/ccm-from-scratch/ccm"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// [ other code ]
|
||||
// This initializes the CLI command. The arguments are interesting, so lets take a closer look:
|
||||
command := app.NewCloudControllerManagerCommand(
|
||||
ccmOptions,
|
||||
|
||||
// This was previously "cloudInitializer"
|
||||
ccm.NewCloudProvider,
|
||||
//
|
||||
|
||||
app.DefaultInitFuncConstructors,
|
||||
names.CCMControllerAliases(),
|
||||
fss,
|
||||
wait.NeverStop,
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### Starting our CCM
|
||||
|
||||
We now have all the pieces in place to try to run our program.
|
||||
|
||||
For now, we can use `go run` to test our binary. The next part of this series will implement a proper development environment.
|
||||
|
||||
```shell
|
||||
$ go run .
|
||||
TODO figure out actual arguments necessary and output
|
||||
```
|
||||
|
|
@ -1,264 +1,20 @@
|
|||
---
|
||||
author: "Julian Tölle"
|
||||
title: "Building a Kubernetes cloud-controller-manager"
|
||||
title: "Part 2: Development Environment with Terraform, k3s and Skaffold"
|
||||
date: "2023-09-06"
|
||||
description: |
|
||||
Tutorial on building a Kubernetes cloud-controller-manager from Scratch.
|
||||
In this post we outline the background and scope of the tutorial and setup the boilerplate.
|
||||
[SideQuest] Goal: Get an environment to deploy our binary to, to test any controllers we implement
|
||||
summary: |
|
||||
[SideQuest] Goal: Get an environment to deploy our binary to, to test any controllers we implement
|
||||
tags: ["kubernetes"]
|
||||
ShowToc: true
|
||||
ShowReadingTime: true
|
||||
cover:
|
||||
image: cover.webp
|
||||
responsiveImages: true
|
||||
caption: dall-e with prompt "ship stearing towards clouds, a captain behind the steering wheel, cartoon, wide angle"
|
||||
ShowBreadCrumbs: true
|
||||
---
|
||||
|
||||
### Disclaimers
|
||||
> If this topic does not interest you, you can safely skip ahead! While we use what we built in the following parts, you can still understand what is going on without reading this article.
|
||||
|
||||
#### Work
|
||||
|
||||
I [work for Hetzner Cloud](../../work.md) and this project was done during working hours as part of the _Lab Days_.
|
||||
During Lab Days, employees can choose to work on topics unrelated to their work.
|
||||
|
||||
#### Production Readiness
|
||||
|
||||
The code written in this series is not ready for production.
|
||||
You should not actually use the code I provide for your Kubernetes Cluster on Hetzner Cloud.
|
||||
Please use the official [`hcloud-cloud-controller-manager`](https://github.com/hetznercloud/hcloud-cloud-controller-manager) instead.
|
||||
|
||||
## Scope
|
||||
|
||||
The series tries to showcase the steps needed to build a first iteration of a Kubernetes Cloud Controller Manager.
|
||||
It can be adapted to any Cloud Provider and may be your guide into better Kubernetes-compatibility for your own Infrastructure-as-a-Service Cloud.
|
||||
|
||||
## Introduction
|
||||
|
||||
### What is a Cloud Controller Manager?
|
||||
|
||||
> Source: https://kubernetes.io/docs/concepts/architecture/cloud-controller/
|
||||
|
||||
- Controllers: Loop that reconciles reality with the desired state
|
||||
- Controller Manager: Binary that runs a bunch of controllers
|
||||
- Cloud Controller Manager: Part of the Kubernetes Control Plane, responsible for exposing Cloud Provider functionality inside the Kubernetes cluster.
|
||||
- Controllers in CCMs:
|
||||
- Node Metadata
|
||||
- Match Kubernetes `Node` to Cloud API Server (/Instance/Droplet/...) and set `spec.providerID`
|
||||
- Label Kubernetes `Nodes` with metadata of the Node, including the [Instance Type](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type), Topology ([Region](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion), [Zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone))
|
||||
- Figure out the Network addresses of the `Node` and set them in `status.addresses`.
|
||||
- Node Lifecycle
|
||||
- The `kubelet` is usually responsible for creating the `Node` object in Kubernetes on its first startup.
|
||||
- This does not work well for the removal of the node, as the `kubelet` might not get a chance to do this. Instead, the Node Lifecycle controller regularly checks if the Node was deleted in the Cloud API, and if it was it also deletes the `Node` object in Kubernetes.
|
||||
- Service
|
||||
- Watches `Service` objects with `type: LoadBalancer`. Creates Cloud Load Balancers and configures them according to the `Service` and `Node` objects
|
||||
- Route
|
||||
- In Kubernetes, every Pod gets an IP address. This IP address needs to be available from every other pods network (by default). This is usually implemented as an Overlay Network through your CNI like Calico or Cilium.
|
||||
- If you are already using some kind of Private Networking functionality of your Cloud Provider, then you can use this instead to get rid of the additional Overlay Network and let the Cloud Provider handle the connectivity.
|
||||
- This is implemented by configuring Routes in the Private Network Routing Table that send all traffic to the `Nodes` `spec.podCIDR` to the `Nodes` private IP.
|
||||
|
||||
### What is the Hetzner Cloud?
|
||||
|
||||
> Source: https://hetzner.com/cloud
|
||||
|
||||
- IaaS with: Servers, Load Balancers, Private Networks
|
||||
|
||||
## (Part 1) Setting up the Boilerplate
|
||||
|
||||
> Goal: Get a compilable binary running k/cloud-provider with no controllers yet
|
||||
|
||||
### Dependencies
|
||||
|
||||
As all ~~good~~ software does, we [stand on the shoulders of giants](https://xkcd.com/2347/). Let us review the two main dependencies that our code will use:
|
||||
|
||||
#### `k8s.io/cloud-provider`
|
||||
|
||||
> Repository: https://github.com/kubernetes/cloud-provider
|
||||
|
||||
Official Library by Kubernetes Project to implement cloud-controller-managers.
|
||||
Historically part of `kubelet` but extracted a long time ago to build out-of-tree CCMs.
|
||||
Handles communication with Kubernetes, CCM implementors only need to implement a narrowly-scoped interface for each functionality.
|
||||
|
||||
> This series will use Kubernetes `v1.28` throughout.
|
||||
> If you use a different version, you may need to adjust your code.
|
||||
> The `k8s.io/cloud-provider` release series associated with Kubernetes `v1.28` is `v0.28.x`.
|
||||
|
||||
#### `github.com/hetznercloud/hcloud-go`
|
||||
|
||||
> Repository: https://github.com/hetznercloud/hcloud-go
|
||||
|
||||
Official Go SDK by Hetzner Cloud, currently maintained by my team.
|
||||
Exposes all functionality of the Hetzner Cloud API in (mostly) idiomatic Go.
|
||||
|
||||
### Repo initialization
|
||||
|
||||
```shell
|
||||
mkdir ccm-from-scratch
|
||||
cd ccm-from-scratch
|
||||
git init
|
||||
|
||||
I like to make an empty initial commit, makes the occasional
|
||||
rebase of the first few commits so much easier.
|
||||
git commit --allow-empty -m "feat: init repo"
|
||||
|
||||
# Optional
|
||||
git remote add git@github.com:apricote/ccm-from-scratch.git
|
||||
git push
|
||||
```
|
||||
|
||||
### `main.go`
|
||||
|
||||
> Source: https://github.com/kubernetes/cloud-provider/tree/v0.28.2/sample
|
||||
|
||||
Thankfully, `k8s.io/cloud-provider` provides an example entrypoint to get us started with out CCM. Lets review the code to make sure we understand what is happening:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
// [Imports]
|
||||
|
||||
// The main function is called as the entrypoint of our Go binary.
|
||||
func main() {
|
||||
// k8s.io/cloud-provider has an elaborate way to read the configuration from flags.
|
||||
// I found this very tedious to debug, but at least we do not have to do this ourselves.
|
||||
ccmOptions, err := options.NewCloudControllerManagerOptions()
|
||||
if err != nil {
|
||||
klog.Fatalf("unable to initialize command options: %v", err)
|
||||
}
|
||||
|
||||
// Can be used to add custom flags to the binary, we dont need this.
|
||||
fss := cliflag.NamedFlagSets{}
|
||||
|
||||
// This initializes the CLI command. The arguments are interesting, so lets take a closer look:
|
||||
command := app.NewCloudControllerManagerCommand(
|
||||
// The options we initialized earlier.
|
||||
ccmOptions,
|
||||
// This is a custom function that needs to return a [cloudprovider.Interface],
|
||||
// we will get to this soon.
|
||||
cloudInitializer,
|
||||
// This defines which controllers are started, if wanted,
|
||||
// one can add additional controller loops heroe
|
||||
app.DefaultInitFuncConstructors,
|
||||
// Kubernetes v1.28 renamed the controllers to more sensible names, this
|
||||
// map makes it so that old command-line arguments (--controllers) still work
|
||||
names.CCMControllerAliases(),
|
||||
// Our optional additional flags
|
||||
fss,
|
||||
// A [<-chan struct{}] that can be used to shut down the CCM on demand,
|
||||
// we do not need this.
|
||||
wait.NeverStop,
|
||||
)
|
||||
// Actually run the command to completion.
|
||||
code := cli.Run(command)
|
||||
os.Exit(code)
|
||||
}
|
||||
```
|
||||
|
||||
Now, this does not compile right now. We use the undefined `cloudInitalizer` method. The method signature we need to implement is `(*config.CompletedConfig) cloudprovider.Interface`. The sample code includes this method, but I found it overly complex for our small CCM, so we will implement it ourselves. Lets take a closer look at the interface we need to return in the next section.
|
||||
|
||||
|
||||
|
||||
|
||||
### `cloudprovider.Interface`
|
||||
|
||||
> Source: https://github.com/kubernetes/cloud-provider/blob/v0.28.2/cloud.go#L42-L69
|
||||
|
||||
This is the entrypoint into the functionality we can (and will!) implement for our CCM. The interface includes an initializer, two cloud-provider metadata methods and a bunch of Getter functions to other interfaces that implement the actual functionality.
|
||||
|
||||
Before we can write the `cloudInitializer` method from above, lets prepare a struct that implements the interface:
|
||||
|
||||
```go
|
||||
package ccm
|
||||
// ccm/cloud.go
|
||||
|
||||
import (
|
||||
cloudprovider "k8s.io/cloud-provider"
|
||||
)
|
||||
|
||||
type CloudProvider struct {}
|
||||
|
||||
// Let's try to assign our struct to a var of the interface we try to implement.
|
||||
// This way we get nice feedback from our IDE if we break the contract.
|
||||
var _ cloudprovider.Interface = CloudProvider{}
|
||||
|
||||
|
||||
// Can be used to setup our own controllers or goroutines that need to talk to
|
||||
//Kubernetes. Not needed for our implementation, so we will leave it empty.
|
||||
func (c CloudProvider) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) {}
|
||||
|
||||
// An arbitrary name for our provider. We will use "ccm-from-scratch" for this example.
|
||||
func (c CloudProvider) ProviderName() string { return "ccm-from-scratch" }
|
||||
|
||||
// I have not yet figured out what Cluster ID is actually supposed to do. We can disable it.
|
||||
func (c CloudProvider) HasClusterID() bool { return false }
|
||||
|
||||
// The following methods all expose additional interfaces. This allows us to
|
||||
// modularly choose what we want to support. As long as we return "false" as the
|
||||
// second argument, the associated functionality is disabled for our CCM.
|
||||
// We will get to implementing [InstancesV2], [LoadBalancer] & [Routes] in later parts of this series.
|
||||
func (c CloudProvider) LoadBalancer() (cloudprovider.LoadBalancer, bool) { return nil, false }
|
||||
func (c CloudProvider) Instances() (cloudprovider.Instances, bool) { return nil, false }
|
||||
func (c CloudProvider) InstancesV2() (cloudprovider.InstancesV2, bool) { return nil, false }
|
||||
func (c CloudProvider) Zones() (cloudprovider.Zones, bool) { return nil, false }
|
||||
func (c CloudProvider) Clusters() (cloudprovider.Clusters, bool) { return nil, false }
|
||||
func (c CloudProvider) Routes() (cloudprovider.Routes, bool) { return nil, false }
|
||||
```
|
||||
|
||||
Now that we have a struct implementing the interface, lets create our `cloudInitializer` method. We will actually do this in the same file as our struct:
|
||||
|
||||
```go
|
||||
// ccm/cloud.go
|
||||
|
||||
// Just create a new struct for now, we will add some more stuff to this later.
|
||||
func NewCloudProvider(_ *config.CompletedConfig) cloudprovider.Interface {
|
||||
return CloudProvider{}
|
||||
}
|
||||
```
|
||||
|
||||
And now we can pass this method in our `main()`:
|
||||
|
||||
```go
|
||||
package main
|
||||
// main.go
|
||||
import (
|
||||
// Existing imports
|
||||
|
||||
"github.com/apricote/ccm-from-scratch/ccm"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// [ other code ]
|
||||
// This initializes the CLI command. The arguments are interesting, so lets take a closer look:
|
||||
command := app.NewCloudControllerManagerCommand(
|
||||
ccmOptions,
|
||||
|
||||
// This was previously "cloudInitializer"
|
||||
ccm.NewCloudProvider,
|
||||
//
|
||||
|
||||
app.DefaultInitFuncConstructors,
|
||||
names.CCMControllerAliases(),
|
||||
fss,
|
||||
wait.NeverStop,
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### Starting our CCM
|
||||
|
||||
We now have all the pieces in place to try to run our program.
|
||||
|
||||
For now, we can use `go run` to test our binary. The next part of this series will implement a proper development environment.
|
||||
|
||||
```shell
|
||||
$ go run .
|
||||
TODO figure out actual arguments necessary and output
|
||||
```
|
||||
|
||||
## (Part 2, Sidequest) Development Environment with Terraform, k3s and Skaffold
|
||||
|
||||
> Goal: Get an environment to deploy our binary to, to test any controllers we implement
|
||||
|
||||
- If this topic does not interest you, you can safely skip ahead! While we use what we built in the following parts, you can still understand what is going on without reading this article.
|
||||
This is part 2 out of 5 of the series [Building a Kubernetes cloud-controller-manager]({{< ref "." >}}).
|
||||
|
||||
Lets start by listing the requirements for our development environment:
|
||||
- Single command to start the environment
|
||||
|
|
@ -269,8 +25,8 @@ Lets start by listing the requirements for our development environment:
|
|||
I have chosen the following technologies to implement this:
|
||||
|
||||
- Terraform Infrastructure-as-Code
|
||||
- Using the [`terraform-provider-hcloud`](https://github.com/hetznercloud/terraform-provider-hcloud) to manage the Hetzner Cloud resources
|
||||
- Using the `kubernetes` & `helm` providers to add the required resources to the cluster
|
||||
- Using the [`terraform-provider-hcloud`](https://github.com/hetznercloud/terraform-provider-hcloud) to manage the Hetzner Cloud resources
|
||||
- Using the `kubernetes` & `helm` providers to add the required resources to the cluster
|
||||
- k3s through k3sup for the Kubernetes cluster
|
||||
- Skaffold for the deployment & debugging
|
||||
|
||||
|
|
@ -773,128 +529,3 @@ If everything worked, you should now have a running `ccm-from-scratch` deploymen
|
|||
$ kubectl get pods -n kube-system -l app=ccm-from-scratch
|
||||
TODO output
|
||||
```
|
||||
|
||||
## (Part 3) Controller: Node
|
||||
|
||||
> Goal: Implement the Node controller with all features, deploy & verify
|
||||
|
||||
Now that our side-quest is finished, we can finally go back to the part why you are all here: Implementing the controllers of our cloud-controller-manager.
|
||||
|
||||
This part focuses on the Node controller. To [recap](TODO Link to part 1) its functionality:
|
||||
- _Node_
|
||||
- Match Kubernetes `Node` to Cloud API Server (/Instance/Droplet/...) and set `spec.providerID`
|
||||
- Label Kubernetes `Nodes` with metadata of the Node, including the [Instance Type](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type), Topology ([Region](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion), [Zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone))
|
||||
- Figure out the Network addresses of the `Node` and set them in `status.addresses`.
|
||||
- _Node Lifecycle_
|
||||
- The `kubelet` is usually responsible for creating the `Node` object in Kubernetes on its first startup.
|
||||
- This does not work well for the removal of the node, as the `kubelet` might not get a chance to do this. Instead, the Node Lifecycle controller regularly checks if the Node was deleted in the Cloud API, and if it was it also deletes the `Node` object in Kubernetes.
|
||||
|
||||
Luck has is, `k8s.io/cloud-provider` already implements most of the logic, so we can focus on the integration with our Cloud Provider (Hetzner Cloud in my case).
|
||||
Similar to the general `Cloud` interface from [Part 1](TODO LINK), this post focuses on the [InstancesV2](https://pkg.go.dev/k8s.io/cloud-provider@v0.28.2#InstancesV2) interface.
|
||||
|
||||
Lets start by stubbing out the interface in a new file:
|
||||
|
||||
```go
|
||||
package ccm
|
||||
// ccm/instances_v2.go
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
cloudprovider "k8s.io/cloud-provider"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
)
|
||||
|
||||
type InstancesV2 struct {}
|
||||
|
||||
// Let's try to assign our struct to a var of the interface we try to implement.
|
||||
// This way we get nice feedback from our IDE if we break the contract.
|
||||
var _ cloudprovider.InstancesV2 = InstancesV2{}
|
||||
|
||||
func (i *InstancesV2) InstanceExists(ctx context.Context, node *corev1.Node) (bool, error) {
|
||||
// We return true for now, to avoid CCM removing any Nodes from the cluster.
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (i *InstancesV2) InstanceShutdown(ctx context.Context, node *corev1.Node) (bool, error) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
func (i *InstancesV2) InstanceMetadata(ctx context.Context, node *corev1.Node) (*cloudprovider.InstanceMetadata, error) {
|
||||
return nil, nil
|
||||
}
|
||||
```
|
||||
|
||||
### Creating our Cloud Provider Client
|
||||
|
||||
Before we can make any requests to our Cloud Provider API, we need to read some credentials and create the client.
|
||||
As mentioned in Part 1, we are going to use [hcloud-go](https://github.com/hetznercloud/hcloud-go), but you can use whatever client you like for your case.
|
||||
Because we also need the client in other controllers, we will create it in `ccm.NewCloudProvider()` and save it to the `CloudProvider` struct.
|
||||
|
||||
```go
|
||||
package ccm
|
||||
// ccm/cloud.go
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/hetznercloud/hcloud-go/hcloud"
|
||||
)
|
||||
|
||||
type CloudProvider struct {
|
||||
client *hcloud.Client
|
||||
}
|
||||
|
||||
func NewCloudProvider(_ *config.CompletedConfig) cloudprovider.Interface {
|
||||
// In Part 2 we setup a Kubernetes Secret with our Hetzner Cloud API token, that is mounted as
|
||||
// the environment variable HCLOUD_TOKEN in our Pod.
|
||||
token := os.Getenv("HCLOUD_TOKEN")
|
||||
|
||||
client := hcloud.NewClient(
|
||||
hcloud.WithToken(token),
|
||||
// Setting a custom user agent
|
||||
hcloud.WithApplication("ccm-from-scratch", ""),
|
||||
)
|
||||
|
||||
|
||||
return CloudProvider{client: client}
|
||||
}
|
||||
```
|
||||
|
||||
We can now initialize the `InstancesV2` struct and pass the client to it:
|
||||
|
||||
```go
|
||||
package ccm
|
||||
|
||||
// ccm/instances_v2.go
|
||||
type InstancesV2 struct {
|
||||
client *hcloud.Client
|
||||
}
|
||||
|
||||
// ccm/cloud.go
|
||||
func (c CloudProvider) InstancesV2() (cloudprovider.InstancesV2, bool) {
|
||||
return InstancesV2{client: c.client}, true
|
||||
}
|
||||
```
|
||||
|
||||
When we start our program now, it should detect that the InstancesV2 interface is supported and start the Node & NodeLifecycle controllers.
|
||||
|
||||
### InstanceMetadata
|
||||
|
||||
// TODO: Start with everything inline, refactor it out in the next section
|
||||
|
||||
### Refactoring
|
||||
|
||||
### InstanceExists
|
||||
|
||||
### InstanceShutdown
|
||||
|
||||
### Testing (in prod!)
|
||||
|
||||
## (Part 4) Controller: Service (Load Balancers)
|
||||
|
||||
> Goal: Setup Load Balancers so traffic reaches our cluster, set up an ingress in the dev env that uses the LB
|
||||
|
||||
## (Part 5) Controller: Route
|
||||
|
||||
> Goal: Use the Hetzner Cloud Networks instead of Overlay Networks for Pod CIDR connectivity
|
||||
133
content/posts/ccm-from-scratch/part-3.md
Normal file
133
content/posts/ccm-from-scratch/part-3.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
---
|
||||
author: "Julian Tölle"
|
||||
title: "Part 3: Implementing Controller: Node"
|
||||
date: "2023-09-06"
|
||||
description: |
|
||||
Tutorial on building a Kubernetes cloud-controller-manager from Scratch.
|
||||
In this post we outline the background and scope of the tutorial and setup the boilerplate.
|
||||
summary: |
|
||||
Tutorial on building a Kubernetes cloud-controller-manager from Scratch.
|
||||
In this post we outline the background and scope of the tutorial and setup the boilerplate.
|
||||
tags: ["kubernetes"]
|
||||
ShowToc: true
|
||||
ShowReadingTime: true
|
||||
ShowBreadCrumbs: true
|
||||
---
|
||||
|
||||
This is part 3 out of 5 of the series [Building a Kubernetes cloud-controller-manager]({{< ref "." >}}).
|
||||
|
||||
|
||||
> Goal: Implement the Node controller with all features, deploy & verify
|
||||
|
||||
Now that our side-quest is finished, we can finally go back to the part why you are all here: Implementing the controllers of our cloud-controller-manager.
|
||||
|
||||
This part focuses on the Node controller. To [recap](TODO Link to part 1) its functionality:
|
||||
- _Node_
|
||||
- Match Kubernetes `Node` to Cloud API Server (/Instance/Droplet/...) and set `spec.providerID`
|
||||
- Label Kubernetes `Nodes` with metadata of the Node, including the [Instance Type](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type), Topology ([Region](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion), [Zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone))
|
||||
- Figure out the Network addresses of the `Node` and set them in `status.addresses`.
|
||||
- _Node Lifecycle_
|
||||
- The `kubelet` is usually responsible for creating the `Node` object in Kubernetes on its first startup.
|
||||
- This does not work well for the removal of the node, as the `kubelet` might not get a chance to do this. Instead, the Node Lifecycle controller regularly checks if the Node was deleted in the Cloud API, and if it was it also deletes the `Node` object in Kubernetes.
|
||||
|
||||
Luck has is, `k8s.io/cloud-provider` already implements most of the logic, so we can focus on the integration with our Cloud Provider (Hetzner Cloud in my case).
|
||||
Similar to the general `Cloud` interface from [Part 1](TODO LINK), this post focuses on the [InstancesV2](https://pkg.go.dev/k8s.io/cloud-provider@v0.28.2#InstancesV2) interface.
|
||||
|
||||
Lets start by stubbing out the interface in a new file:
|
||||
|
||||
```go
|
||||
package ccm
|
||||
// ccm/instances_v2.go
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
cloudprovider "k8s.io/cloud-provider"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
)
|
||||
|
||||
type InstancesV2 struct {}
|
||||
|
||||
// Let's try to assign our struct to a var of the interface we try to implement.
|
||||
// This way we get nice feedback from our IDE if we break the contract.
|
||||
var _ cloudprovider.InstancesV2 = InstancesV2{}
|
||||
|
||||
func (i *InstancesV2) InstanceExists(ctx context.Context, node *corev1.Node) (bool, error) {
|
||||
// We return true for now, to avoid CCM removing any Nodes from the cluster.
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (i *InstancesV2) InstanceShutdown(ctx context.Context, node *corev1.Node) (bool, error) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
func (i *InstancesV2) InstanceMetadata(ctx context.Context, node *corev1.Node) (*cloudprovider.InstanceMetadata, error) {
|
||||
return nil, nil
|
||||
}
|
||||
```
|
||||
|
||||
### Creating our Cloud Provider Client
|
||||
|
||||
Before we can make any requests to our Cloud Provider API, we need to read some credentials and create the client.
|
||||
As mentioned in Part 1, we are going to use [hcloud-go](https://github.com/hetznercloud/hcloud-go), but you can use whatever client you like for your case.
|
||||
Because we also need the client in other controllers, we will create it in `ccm.NewCloudProvider()` and save it to the `CloudProvider` struct.
|
||||
|
||||
```go
|
||||
package ccm
|
||||
// ccm/cloud.go
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/hetznercloud/hcloud-go/hcloud"
|
||||
)
|
||||
|
||||
type CloudProvider struct {
|
||||
client *hcloud.Client
|
||||
}
|
||||
|
||||
func NewCloudProvider(_ *config.CompletedConfig) cloudprovider.Interface {
|
||||
// In Part 2 we setup a Kubernetes Secret with our Hetzner Cloud API token, that is mounted as
|
||||
// the environment variable HCLOUD_TOKEN in our Pod.
|
||||
token := os.Getenv("HCLOUD_TOKEN")
|
||||
|
||||
client := hcloud.NewClient(
|
||||
hcloud.WithToken(token),
|
||||
// Setting a custom user agent
|
||||
hcloud.WithApplication("ccm-from-scratch", ""),
|
||||
)
|
||||
|
||||
|
||||
return CloudProvider{client: client}
|
||||
}
|
||||
```
|
||||
|
||||
We can now initialize the `InstancesV2` struct and pass the client to it:
|
||||
|
||||
```go
|
||||
package ccm
|
||||
|
||||
// ccm/instances_v2.go
|
||||
type InstancesV2 struct {
|
||||
client *hcloud.Client
|
||||
}
|
||||
|
||||
// ccm/cloud.go
|
||||
func (c CloudProvider) InstancesV2() (cloudprovider.InstancesV2, bool) {
|
||||
return InstancesV2{client: c.client}, true
|
||||
}
|
||||
```
|
||||
|
||||
When we start our program now, it should detect that the InstancesV2 interface is supported and start the Node & NodeLifecycle controllers.
|
||||
|
||||
### InstanceMetadata
|
||||
|
||||
// TODO: Start with everything inline, refactor it out in the next section
|
||||
|
||||
### Refactoring
|
||||
|
||||
### InstanceExists
|
||||
|
||||
### InstanceShutdown
|
||||
|
||||
### Testing (in prod!)
|
||||
17
content/posts/ccm-from-scratch/part-4.md
Normal file
17
content/posts/ccm-from-scratch/part-4.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
---
|
||||
author: "Julian Tölle"
|
||||
title: "Part 4: Implementing Controller: Service (Load Balancers)"
|
||||
date: "2023-09-06"
|
||||
description: |
|
||||
Goal: Setup Load Balancers so traffic reaches our cluster, set up an ingress in the dev env that uses the LB
|
||||
summary: |
|
||||
Goal: Setup Load Balancers so traffic reaches our cluster, set up an ingress in the dev env that uses the LB
|
||||
tags: ["kubernetes"]
|
||||
ShowToc: true
|
||||
ShowReadingTime: true
|
||||
ShowBreadCrumbs: true
|
||||
---
|
||||
|
||||
This is part 4 out of 5 of the series [Building a Kubernetes cloud-controller-manager]({{< ref "." >}}).
|
||||
|
||||
>
|
||||
15
content/posts/ccm-from-scratch/part-5.md
Normal file
15
content/posts/ccm-from-scratch/part-5.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
---
|
||||
author: "Julian Tölle"
|
||||
title: "Part 5: Implementing Controller: Route"
|
||||
date: "2023-09-06"
|
||||
description: |
|
||||
Goal: Use the Hetzner Cloud Networks instead of Overlay Networks for Pod CIDR connectivity
|
||||
summary: |
|
||||
Goal: Use the Hetzner Cloud Networks instead of Overlay Networks for Pod CIDR connectivity
|
||||
tags: ["kubernetes"]
|
||||
ShowToc: true
|
||||
ShowReadingTime: true
|
||||
ShowBreadCrumbs: true
|
||||
---
|
||||
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue