Openstack Kubernetes Deployment

This Terraform example provides one stop approach to deploy the CogStack platform with its core components and observability stack in an OpenStack environment. It is specifically designed to simplify and automate the provisioning and configuration needed to run CogStack reliably and securely.

This example:

• Provisions Ubuntu VMs in openstack
• Installs Docker on the VMs using Cloud-Init to manage containers easily
• Installs Kubernetes using k3s using Cloud-init

Usage

Requirements

• Terraform - Install Terraform
• Openstack Cloud environment

1. Get the configuration files

Get the Terraform files for this example (the ZIP contains all deployment-examples; use the openstack-kubernetes folder for this guide).

Download all deployment examples (ZIP)

Alternatively you can view the file contents here:

k3s-cluster terraform files

This terraform configuration provisions VMs and installs k3s.

main.tfoutputs.tfproviders.tfvariables.tf

module "openstack_cogstack_infra" {
  source = "../../../deployment/terraform/modules/openstack-kubernetes-infra"
  host_instances = [
    {
      name          = "cogstack-k3s",
      is_controller = true,
      # floating_ip = {
      #   use_floating_ip = true,
      #   address         = "10.10.10.10"
      # }
    },
    {
      name          = "cogstack-k3s-node-2"
      flavour       = "2cpu4ram"
      volume_size   = 20
      is_controller = false
    },
  ]
  allowed_ingress_ips_cidr = var.allowed_ingress_ips_cidr
  ubuntu_immage_name       = var.ubuntu_immage_name
  # generate_random_name_prefix = false
  # prefix                      = "dev"
  # network = {
  #   network_id = "some-id"
  # }
}

output "created_infra" {
  value = module.openstack_cogstack_infra
}

output "kubeconfig_file" {
  value = module.openstack_cogstack_infra.kubeconfig_file
}

terraform {
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 3.0.0"
    }
  }
}

provider "openstack" {
  insecure                      = true
  enable_logging                = true
  auth_url                      = var.openstack_environment.openstack_auth_url
  application_credential_id     = var.openstack_environment.openstack_application_credential_id
  application_credential_secret = var.openstack_environment.openstack_application_credential_secret
  region                        = var.openstack_environment.region
}

variable "openstack_environment" {
  type = object({
    openstack_auth_url                      = string,
    openstack_application_credential_secret = string,
    openstack_application_credential_id     = string,
    region                                  = string
  })
  description = <<EOT
openstack_application_credential_id     = OpenStack application credential ID. Scoped to a project
openstack_application_credential_secret = OpenStack application credential secret. Scoped to a project
openstack_auth_url    = OpenStack Auth URL. Scoped accross projects
EOT
}

variable "allowed_ingress_ips_cidr" {
  description = "The CIDR block to grant access to cogstack services to. For example, grant access to internal users in the VPN"
  type        = string
}

variable "ubuntu_immage_name" {
  type        = string
  description = "Name of an available Machine Image running ubuntu in the openstack environment"
}

kubernetes-deployment terraform files

This terraform configuration deploys CogStack services to the cluster.

dashboard.tfingress.ymlmain.tfoutput.tfvariables.tf

resource "helm_release" "kubernetes_dashboard" {
  depends_on   = [kubernetes_namespace.kubernetes_dashboard]
  name         = "kubernetes-dashboard"
  repository   = "https://kubernetes.github.io/dashboard/"
  chart        = "kubernetes-dashboard"
  namespace    = "kubernetes-dashboard"
  timeout      = 600
  atomic       = true
  force_update = true
  #   values = [
  #     <<EOT
  # app:
  #   ingress:

  #     enabled: true
  #     path: /
  #     hosts: 
  #     useDefaultIngressClass: true
  #     tls:
  #       enabled: false
  #     issuer:
  #       scope: disabled

  # EOT
  #   ]
}
resource "kubernetes_namespace" "kubernetes_dashboard" {
  metadata {
    name = "kubernetes-dashboard"
  }
}

resource "kubernetes_service_account" "admin_user" {
  depends_on = [kubernetes_namespace.kubernetes_dashboard]
  metadata {
    namespace = "kubernetes-dashboard"
    name      = "admin-user"
  }
}

resource "kubernetes_cluster_role_binding" "admin_user" {
  depends_on = [kubernetes_namespace.kubernetes_dashboard]
  metadata {
    name = "admin-user"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  subject {
    kind      = "ServiceAccount"
    name      = "admin-user"
    namespace = "kubernetes-dashboard"
  }
}

resource "kubernetes_secret" "admin_user" {
  depends_on = [kubernetes_namespace.kubernetes_dashboard]
  metadata {
    name      = "admin-user"
    namespace = "kubernetes-dashboard"

    annotations = {
      "kubernetes.io/service-account.name" = "admin-user"
    }
  }

  type = "kubernetes.io/service-account-token"
}

# TODO
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: kubernetes-dashboard-kong-proxy 
            port:
              number: 443
        path: /dashboard
        pathType: ImplementationSpecific

provider "helm" {
  kubernetes = {
    config_path = var.kubeconfig_file
  }
}

provider "kubernetes" {
  config_path = var.kubeconfig_file
}

locals {
  services = {
    medcat_service = {
      path_prefix = "medcat-service"
    }
  }
}

module "cogstack_helm_services" {
  source                = "github.com/CogStack/cogstack-platform-toolkit//deployment/terraform/modules/cogstack-helm-services?ref=terraform-modules-v0.1.0"
  medcat_service_values = <<EOT
replicaCount: 2
EOT
}

locals {
  medcat_ingress_address = module.cogstack_helm_services.service_urls.medcat_service.ip_address
}

output "service_urls" {
  value = {
    medcat_service = {
      example_curl = "curl http://${local.medcat_ingress_address}/${local.services.medcat_service.path_prefix}/api/info"
      example_ui   = "http://${local.medcat_ingress_address}/${local.services.medcat_service.path_prefix}/docs"
    }
  }
  description = "Public URls to call services on"
}


output "dashboard" {
  value = {
    url          = ""
    access_token = kubernetes_secret.admin_user.data.token
  }
  sensitive = true
}

variable "kubeconfig_file" {
  description = "Path to the generated KUBECONFIG file used to connect to kubernetes"
}

2. Add required secrets for your environment

Create a terraform.tfvars file, based on terraform.tfvars.example, containing the secrets for your environment.

3. Run Terraform

cd k3s-cluster
terraform init
terraform apply --auto-approve

K3S_KUBECONFIG=$(terraform output -raw kubeconfig_file)

# Deploy services to kubernetes
cd ../kubernetes-deployment
export TF_VAR_kubeconfig_file=$K3S_KUBECONFIG
terraform init
terraform apply --auto-approve

Initial provisioning takes up to 10 minutes, where time is mostly downloading large docker images

4. Accessing the CogStack Platform

Once the deployment is complete and all services are running, you can access the CogStack platform and its components using the following URLs:

terraform output created_services

Optionally use the K8s cluster as normal with the CLI

After setting up the cluster, it is possible to interact directly with it using the kubectl CLI

The requirement is to get the KUBECONFIG file created by the terraform apply.

# Get KUBECONFIG
K3S_KUBECONFIG=$(terraform output -raw kubeconfig_file)

# SET KUBECONFIG
export KUBECONFIG=${K3S_KUBECONFIG}

You can then interact with kubernetes via the CLI for example:

# Run Medcat service
helm install my-medcat oci://registry-1.docker.io/cogstacksystems/medcat-service-helm --wait --timeout 10m0s

# Find public url
kubectl get ingress

Access the k8s dashboard using

terraform output dashboard # Find the access token
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443