Skip to content

Instantly share code, notes, and snippets.

@so0k

so0k/00-README.md

Last active May 26, 2018 08:03
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save so0k/f4308160a9a2e749aa0b90715288e08b to your computer and use it in GitHub Desktop.
Save so0k/f4308160a9a2e749aa0b90715288e08b to your computer and use it in GitHub Desktop.
Download ZIP
Cog Multi Pod deployment with external Postgres
Raw
00-README.md

Deploying Cog on Kubernetes (on AWS)

Note: Use the excellent helm chart - ohaiwalt/cog-helm to get up quickly on k8s

Follow the guide below to understand the inner workings a bit better.

We will be creating 1 deployment for the cog server and 1 deployment for the relay server.

We chose to run Postgres outside the Kubernetes cluster and provide instructions how to expose all service ports on a single ELB.

Using an existing RDS instance, create fresh user and database on an existing RDS (Postgres)

$ psql -U $RDS_USERNAME -h $RDS_HOSTNAME -p $RDS_PORT -d postgres
postgres=> \l
postgres=> CREATE DATABASE cog;
postgres=> CREATE USER cog WITH PASSWORD 'mysupersecret';
postgres=> GRANT ALL PRIVILEGES ON DATABASE cog TO cog;
postgres=> \connect cog
cog=> CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
cog=> \q

Or provision a complete fresh RDS instance.

Update the secrets in 04-secrets.

On OSX, generate uuid with uuidgen but make sure to convert to lower case!

uuidgen | tr '[:upper:]' '[:lower:]'

generate token with openssl rand -hex 12

(or python -c 'import sys,os,binascii; sys.stdout.write(binascii.hexlify(os.urandom(16)).decode("utf-8"))')

Convert 04-secrets to a kubernetes secret called cog:

./06-env2secret.py -i 05-secrets -o 02-cog-secrets.yml -s cog

Deploy:

kubectl create -f 01-cog-service.yml

NOTE: Do NOT call your service "COG" - it will create an Environment variable COG_SERVICE_PORT inside every container, causing port conflicts and configuration issues!

kubectl create -f 02-cog-secrets.yml

kubectl create -f 03-cog-deployment.yml

Note: COG_HOST environment variable in Relay Deployment definition needs to be able to find the service as it will connect to MQTT message bus:

kubectl create -f 04-relay-deployment.yml

review relay logs:

kubectl logs `kubectl get po -l app=relay -o name | cut -d"/" -f2`

If you see this error:

client API version: 1.23, server API version: 1.22

review Docker Version (CoreOS stable is still on 1.10.3, use $DOCKER_API_VERSION to work around this)

If you see this error:

Dynamic configuration root dir not found.

or

msg="Got invocation request on /bot/commands/<relay-id>/ec2/instance-list"
msg="Creating environment 6bff5c76eaf64eb896bdc6736aff5f4e/ec2:0.1.0"
msg="Dynamic config not found. Checked: '/data/ec2/config.yaml' and '/data/ec2/config.yml'."
msg="Dynamic config not found. Checked: '/data/ec2/room_direct.yaml' and '/data/ec2/room_direct.yml'."
msg="Dynamic config not found. Checked: '/data/ec2/user_so0k.yaml' and '/data/ec2/user_so0k.yml'."

Note: When using Dynamic configurations controlled centrally on COG server, need to configure relay to allow RELAY_MANAGED_DYNAMIC_CONFIG

Add TCP Listeners to ELB

aws elb create-load-balancer-listeners --load-balancer-name k8s-elb --listeners "Protocol=TCP,LoadBalancerPort=4000,InstanceProtocol=TCP,InstancePort=34000"
aws elb create-load-balancer-listeners --load-balancer-name k8s-elb --listeners "Protocol=TCP,LoadBalancerPort=4001,InstanceProtocol=TCP,InstancePort=34001"
aws elb create-load-balancer-listeners --load-balancer-name k8s-elb --listeners "Protocol=TCP,LoadBalancerPort=4002,InstanceProtocol=TCP,InstancePort=34002"

Note: Use TLS termination on the ELB by setting the listener Protocol to HTTPS with a valid certificate (or use Let's encrypt).

Open ports on ELB Security Group (untested)

aws ec2 authorize-security-group-ingress --group-name sg-k8s-elb --protocol tcp --from-port 4000 --to-port 4002 --cidr "0.0.0.0/0"

Create CNAME for ELB in route53

cog.example.com -> elb.amazon-url

Get a shell

kubectl get po -l app=cog
kubectl exec -it cog-2663148853-e9bcb /bin/sh

Follow instructions from the cog book

Raw
01-cog-service.yml
kind: Service
apiVersion: v1
metadata:
name: cogapi
labels:
app: cog
spec:
type: NodePort
selector:
app: cog
ports:
- port: 4000
targetPort: 4000
nodePort: 34000
protocol: TCP
name: api
- port: 4001
targetPort: 4001
nodePort: 34001
protocol: TCP
name: trigger
- port: 4002
targetPort: 4002
nodePort: 34002
protocol: TCP
name: service
Raw
03-cog-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: cog
spec:
replicas: 1
template:
metadata:
labels:
app: cog
track: stable
spec:
containers:
- name: cog
image: operable/cog:0.16.2
imagePullPolicy: Always
ports:
- name: api
containerPort: 4000
protocol: TCP
- name: trigger
containerPort: 4001
protocol: TCP
- name: service
containerPort: 4002
protocol: TCP
- name: mqtt
containerPort: 1883
protocol: TCP
env:
- name: "COG_API_URL_HOST"
# where to reach cog API externally
value: "https://cog.example.com"
- name: "COG_API_URL_PORT"
value: "4000"
- name: "COG_TRIGGER_URL_HOST"
# where to reach trigger externally
value: "https://cog.example.com"
- name: "COG_TRIGGER_URL_PORT"
value: "4001"
- name: "COG_SERVICE_URL_HOST"
# where to reach service externally
value: "https://cog.example.com"
- name: "COG_SERVICE_URL_PORT"
value: "4002"
- name: "COG_MQTT_HOST"
value: "0.0.0.0"
- name: "COG_MQTT_PORT"
value: "1883"
- name: "COG_BOOTSTRAP_EMAIL_ADDRESS"
value: "cog@localhost"
- name: "COG_BOOTSTRAP_FIRST_NAME"
value: "Cog"
- name: "COG_BOOTSTRAP_LAST_NAME"
value: "Administrator"
- name: "COG_BOOTSTRAP_PASSWORD"
value: "changeme"
- name: "COG_BOOTSTRAP_USERNAME"
value: "admin"
- name: "COG_SLACK_ENABLED"
value: "1"
- name: "SLACK_API_TOKEN"
valueFrom:
secretKeyRef:
name: cog
key: slack-api-token
- name: "DATABASE_URL"
valueFrom:
secretKeyRef:
name: cog
key: database-url
- name: "RELAY_COG_TOKEN"
valueFrom:
secretKeyRef:
name: cog
key: relay-cog-token
- name: "RELAY_ID"
valueFrom:
secretKeyRef:
name: cog
key: relay-id
volumeMounts:
- name: cog-data
mountPath: "/data"
command:
- /home/operable/cog/scripts/docker-start
volumes:
- name: cog-data
emptyDir: {}
Raw
04-relay-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: relay
spec:
replicas: 1
template:
metadata:
labels:
app: relay
track: stable
spec:
containers:
- name: relay
image: operable/relay:0.16.1
imagePullPolicy: Always
securityContext:
privileged: true
env:
- name: "RELAY_COG_HOST"
value: "cogapi"
- name: "RELAY_DYNAMIC_CONFIG_ROOT"
value: "/data/relay/configs"
# pull dynamic config from cog deployment
- name: "RELAY_MANAGED_DYNAMIC_CONFIG"
value: "true"
- name: "RELAY_COG_REFRESH_INTERVAL"
value: "30s"
# - name: relay_log_level
# - value: debug
- name: "RELAY_DOCKER_CLEAN_INTERVAL"
value: "1m"
# CoreOS stable is on older Docker server
# - name: "DOCKER_API_VERSION"
# value: "1.22"
- name: "RELAY_COG_TOKEN"
valueFrom:
secretKeyRef:
name: cog
key: relay-cog-token
- name: "RELAY_ID"
valueFrom:
secretKeyRef:
name: cog
key: relay-id
volumeMounts:
- name: cog-data
mountPath: "/data"
- name: docker-socket
mountPath: "/var/run/docker.sock"
command:
- /usr/local/bin/relay
volumes:
- name: docker-socket
hostPath:
path: /var/run/docker.sock
- name: cog-data
emptyDir: {}
Raw
05-secrets
SLACK_API_TOKEN=xoxb-...
DATABASE_URL=ecto://cog:password@mypg.ap-southeast-1.rds.amazonaws.com:5432/cogdb
RELAY_ID=00000000-0000-0000-0000-000000000000
RELAY_COG_TOKEN=mysecrettoken
Raw
06-env2secret.py
#!/usr/bin/env python
import os,yaml,argparse
from base64 import b64encode
parser = argparse.ArgumentParser(description="convert .env to k8s secret")
parser.add_argument("-i","--input-file", default=".env",
help=".env file to parse (default=./.env)")
parser.add_argument("-o","--output-file",default="secret.yaml",
help="yaml file to generate (default=./secret.yaml)")
parser.add_argument("-s","--secret-name",default="mysecret",
help="name for secret (default=mysecret)")
args = parser.parse_args()
secret = {
"apiVersion": "v1",
"kind": "Secret",
"metadata": {
"name": args.secret_name
},
"type": "Opaque",
"data": {}
}
# add secrets from envfile
if not os.path.exists(args.input_file):
print("{filename} not found".format(filename=args.input_file))
else:
with open(args.input_file) as f:
for line in f:
line = line.strip()
if not line or line.startswith('#') or '=' not in line:
continue
k,v = line.split('=',1)
v = v.strip()
k = k.lower().replace('_','-')
k = ''.join(e for e in k if (e.isalnum() or e == '-'))
secret['data'][k]=b64encode(v)
# dump secret to yaml
with open(args.output_file,'w') as out:
yaml.dump(secret,out,default_flow_style=False)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment