Economics of DC/OS Starter Cluster on AWS

Yu##### … (with some tweaks and rants)

Ref: https://dcos.io/docs/1.8/administration/installing/cloud/aws/
Ref: https://downloads.dcos.io/dcos/stable/aws.html

DC/OS AWS CloudFormation Options

Master Agent Hourly Cost ($) Daily ($)
1 5 + 1 $1.862 (= 0.266*7) ~ $45
3 5 + 1 $2.394 (= 0.266*9) ~ $57.50

That’s about a COFFEE every hour. Not bad if you need it for few hours only (or PoC is funded by company).

A word of caution here – make sure that you completely destroy the stack using CloudFormation console. Your EC2 instances will come back to life if you simply kill them from “EC2 Dashboard”. One of my friend got over $800 invoice because he didn’t understand this awesome feature of CloudFormation.

So what to do if you are on a tighter budget (like self-funded learning adventure) and want to explore DC/OS?

How do I get DC/OS on AWS for cheap?

Option EC2 Hourly ($) Daily ($) Spot
Single m3.xlarge $0.266 ~ $6.38 ~$1.00
Mini m3.large + m3.xlarge $0.379 ~ $9.09 ~$1.40

FYI: SPOT gets 85-90% savings.

My suggestion would be Mini Install which is good enough for trying out simple utility & microservices use-cases or go for {1 master +5 agents} for big-data analytics use-case.

Ref: https://docs.mesosphere.com/1.8/administration/installing/custom/advanced/

Minimal Mesosphere DC/OS v1.8 (Mini)

… (with tweak for running a tiny marathon-lb on the slave node)

Prerequisites

  • Two VMs only! (for a resource constrained lab or laptop)
  • CentOS 7.x Minimal + SELINUX Disabled + Firewall Disabled + IPv6 Disabled
  • Docker 1.11.x with “–storage-driver=overlay”
  • Catch-all DNS *.a77.sudhaker.com => 192.168.20.77

Configuration

Host Role VM Size
mesos-mini-01 bootstrap + master + dcos-cli 2 CPU, 4GB RAM, 60GB HDD
mesos-mini-02 the slave (hybrid) 2 CPU, 4GB RAM, 60GB HDD

For AWS, mini-01 can be a m3.large or higher & mini-02 can be a m3.xlarge or higher!

Common script – To be run on every node

# add docker repo and install docker
cat > /etc/yum.repos.d/docker.repo << '__EOF__'
[docker]
name=Docker Repository - Centos $releasever
baseurl=http://yum.dockerproject.org/repo/main/centos/$releasever
enabled=1
gpgcheck=1
gpgkey=http://yum.dockerproject.org/gpg
__EOF__

yum install -y https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
yum install -y docker-engine-1.11.2 docker-engine-selinux-1.11.2 tar xz unzip curl ipset nfs-utils

yum clean all

groupadd nogroup

mkdir -p /etc/systemd/system/docker.service.d

cat > /etc/systemd/system/docker.service.d/override.conf << '__EOF__'
[Service] 
ExecStart= 
ExecStart=/usr/bin/docker daemon --storage-driver=overlay
__EOF__

systemctl daemon-reload
systemctl enable docker

systemctl start docker

# if the default DNS doesn't resolve
tee -a /etc/hosts << '__EOF__'

192.168.20.76 mesos-mini-01
192.168.20.77 mesos-mini-02
__EOF__

To be run on node1 {bootstrap + master + dcos-cli}

Generate bootstrap

mkdir /opt/dcos-setup && cd /opt/dcos-setup && curl -O https://downloads.dcos.io/dcos/stable/dcos_generate_config.sh

mkdir -p genconf

cat > genconf/ip-detect << '__EOF__'
#!/usr/bin/env bash
set -o nounset -o errexit
export PATH=/usr/sbin:/usr/bin:$PATH
echo $(ip addr show ens192 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | head -1)
__EOF__

chmod 755 genconf/ip-detect

### VERY IMPORTANT: validate that the following ip-detect is working
### if not then follow https://dcos.io/docs/1.8/administration/installing/custom/advanced/
###   to get a working script, must work consistently on each & every node
./genconf/ip-detect

cat > genconf/config.yaml << '__EOF__'
---
bootstrap_url: http://mesos-mini-01:8081       
cluster_name: dcos
exhibitor_storage_backend: static
master_discovery: static
master_list:
- 192.168.20.76
resolvers:
- 192.168.20.1
__EOF__

bash dcos_generate_config.sh

Launch bootstrap

docker pull nginx:alpine
docker run -d --restart=unless-stopped -p 8081:80 -v /opt/dcos-setup/genconf/serve:/usr/share/nginx/html:ro --name=dcos-bootstrap-nginx nginx:alpine

Install master

mkdir -p /tmp/dcos && cd /tmp/dcos && curl -O http://mesos-mini-01:8081/dcos_install.sh && bash dcos_install.sh master && cd -

Install dcos-cli (while we are waiting for master to come up).

mkdir -p ~/bin && cd ~/bin && curl -O https://downloads.dcos.io/binaries/cli/linux/x86-64/dcos-1.8/dcos && chmod 755 ~/bin/dcos && cd -

dcos config set core.dcos_url http://mesos-mini-01
dcos auth login

To be run on node2 {the hybrid slave}

mkdir -p /tmp/dcos && cd /tmp/dcos && curl -O http://mesos-mini-01:8081/dcos_install.sh && bash dcos_install.sh slave  && cd -

And then tweak it to support marathon-lb on it (thanks Vishnu for the ‘mesos-resource’ pointer).

systemctl kill -s SIGUSR1 dcos-mesos-slave
systemctl stop dcos-mesos-slave
unlink /var/lib/mesos/slave/meta/slaves/latest

Edit /var/lib/dcos/mesos-resources (add 80, 443) – whatever sed works (validate)

sed -i 's/\[{"begin"/\[{"begin": 80, "end": 80}, {"begin": 443, "end": 443}, {"begin"/' /var/lib/dcos/mesos-resources
OR
sed -i 's/\[{"end"/\[{"begin": 80, "end": 80}, {"begin": 443, "end": 443}, {"end"/' /var/lib/dcos/mesos-resources

Re-initialize the slave (with required ports – 80,443)

systemctl start dcos-mesos-slave

Install the tiny marathon-lb (0.1 CPU, 64mb RAM)

IMPORTANT: dcos package won’t let us install this minimal configuration, so we’ll use marathon!

Export marathon json

dcos package describe --app --render marathon-lb > marathon-lb.json

Edit the json, use sed or manually edit it

sed -i 's/"slave_public"/"*"/' marathon-lb.json
sed -i 's/"cpus": 2/"cpus": 0.1/' marathon-lb.json
sed -i 's/ "mem": 1024/ "mem": 64/' marathon-lb.json

Let marathon deploy this app

dcos marathon app add marathon-lb.json

Install the demo app (dockercloud/hello-world)

cat > dockercloud-hello-world.json << '__EOF__'
{
  "id": "dockercloud-hello-world",
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "dockercloud/hello-world",
      "network": "BRIDGE",
      "portMappings": [
        { "hostPort": 0, "containerPort": 80 }
      ],
      "forcePullImage":true
    }
  },
  "instances": 2,
  "cpus": 0.1,
  "mem": 128,
  "healthChecks": [{
      "protocol": "HTTP",
      "path": "/",
      "portIndex": 0,
      "timeoutSeconds": 10,
      "gracePeriodSeconds": 10,
      "intervalSeconds": 2,
      "maxConsecutiveFailures": 10
  }],
  "labels":{
    "HAPROXY_GROUP":"external",
    "HAPROXY_0_VHOST":"dockercloud-hello-world.a77.sudhaker.com"
  }
}
__EOF__

dcos marathon app add dockercloud-hello-world.json

Browse to => http://dockercloud-hello-world.a77.sudhaker.com

Minimal Mesosphere DC/OS v1.8 (Single)

… (with tweak for running a tiny marathon-lb)

Prerequisites

  • One VMs only! (for a resource constrained lab or laptop)
  • CentOS 7.x Minimal + SELINUX Disabled + Firewall Disabled + IPv6 Disabled
  • Docker 1.11.x with “–storage-driver=overlay”

Configuration

Host Role VM Size
mesos-single-01 bootstrap + master + dcos-cli 2 CPU, 4GB RAM, 60GB HDD

Update: I was told that this setup will seize to work with DCOS v1.9 due to some unavoidable conflict. Ouch!

Common script – To be run on every node

# add docker repo and install docker
cat > /etc/yum.repos.d/docker.repo << '__EOF__'
[docker]
name=Docker Repository - Centos $releasever
baseurl=http://yum.dockerproject.org/repo/main/centos/$releasever
enabled=1
gpgcheck=1
gpgkey=http://yum.dockerproject.org/gpg
__EOF__

yum install -y https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
yum install -y docker-engine-1.11.2 docker-engine-selinux-1.11.2 tar xz unzip curl ipset nfs-utils

yum clean all

groupadd nogroup

mkdir -p /etc/systemd/system/docker.service.d

cat > /etc/systemd/system/docker.service.d/override.conf << '__EOF__'
[Service] 
ExecStart= 
ExecStart=/usr/bin/docker daemon --storage-driver=overlay --insecure-registry mesos-mini-01:5000 -H fd:// 
__EOF__

systemctl daemon-reload
systemctl enable docker

systemctl start docker

# if the default DNS doesn't resolve
tee -a /etc/hosts << '__EOF__'

192.168.20.80 mesos-single
__EOF__

To be run on node1 {bootstrap + master + dcos-cli}

Generate bootstrap

mkdir /opt/dcos-setup && cd /opt/dcos-setup && curl -O https://downloads.dcos.io/dcos/stable/dcos_generate_config.sh

mkdir -p genconf

cat > genconf/ip-detect << '__EOF__'
#!/usr/bin/env bash
set -o nounset -o errexit
export PATH=/usr/sbin:/usr/bin:$PATH
echo $(ip addr show ens192 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | head -1)
__EOF__

chmod 755 genconf/ip-detect

### VERY IMPORTANT: validate that the following ip-detect is working
### if not then follow https://dcos.io/docs/1.8/administration/installing/custom/advanced/
###   to get a working script, must work consistently on each & every node
./genconf/ip-detect

cat > genconf/config.yaml << '__EOF__'
---
bootstrap_url: http://mesos-mini-01:8081       
cluster_name: dcos
exhibitor_storage_backend: static
master_discovery: static
master_list:
- 192.168.20.80
resolvers:
- 192.168.20.1
__EOF__

bash dcos_generate_config.sh

Launch bootstrap

docker pull nginx:alpine
docker run -d --restart=unless-stopped -p 8081:80 -v /opt/dcos-setup/genconf/serve:/usr/share/nginx/html:ro --name=dcos-bootstrap-nginx nginx:alpine

Install master

mkdir -p /tmp/dcos && cd /tmp/dcos && curl -O http://mesos-single:8081/dcos_install.sh && bash dcos_install.sh master && cd -

Install slave

export opt_mesos=$(ls -1d /opt/mesosphere/packages/mesos--*)
ln -s $opt_mesos/dcos.target.wants_slave/dcos-mesos-slave.service /etc/systemd/system
ln -s $opt_mesos/dcos.target.wants_slave/dcos-mesos-slave.service /etc/systemd/system/dcos.target.wants
mkdir -p /var/log/mesos
systemctl start dcos-mesos-slave

Install dcos-cli (while we are waiting for master to come up).

mkdir -p ~/bin && cd ~/bin && curl -O https://downloads.dcos.io/binaries/cli/linux/x86-64/dcos-1.8/dcos && chmod 755 ~/bin/dcos && cd -

dcos config set core.dcos_url http://mesos-single
dcos auth login