The magic behind
"one-click" deployments

Chaos Communication Camp 2023

About me

Hi, I'm Jadyn
she/her
I'm 22
I work in cybersecurity
I do things...

Mastodon: @jadyn@chaos.social
Web: jadyn.dev

Overview

  • "one-click" deployment?
  • Inner workings
  • Demonstration
  • Q&A

One-Click deployment

  • Fully automated deployment
  • What does that mean?
    • VM/Container
    • IPs
    • DNS
    • Users
    • Firewall
    • Software
    • Monitoring
    • ...

Inner workings

Deployment runner

  • root/sudo access to every machine
    → ssh
  • Access to deployment sources
    → Gitlab, GitHub, netbox, ...
  • Access to secret storage
    → Requires a gpg setup

→ Dedicated Host with a YubiKey

picture datacenter

Inner workings

Gather data

  • Hostname(s)
  • Networking
    Interfaces, IPs, Gateways, DNS servers
  • Hypervisor configs
    Cores, Memory, Disks, IDs, ...
  • Services
screenshot netbox device overview

Inner workings

Gather secrets

  • Passwords / Secrets
  • Storage
    → pass compatible format
  • Generation
    • Dynamic Secrets
    • Static Secrets

                                ansible
                                └── caffeinated-labs.net
                                    ├── hosts
                                    │   ├── autodeploy.jupiter.caffeinated-labs.net
                                    │   │   └── secrets_host_root_pw
                                    │   ├── firewall.jupiter.caffeinated-labs.net
                                    │   │   └── secrets_host_root_pw
                                    │   ├── jupiter.caffeinated-labs.net
                                    │   │   └── secrets_host_root_pw
                                    │   ├── netbox-new.jupiter.caffeinated-labs.net
                                    │   │   ├── netbox_config_secret
                                    │   │   ├── netbox_db_password
                                    │   │   ├── netbox_superuser_password
                                    │   │   └── secrets_host_root_pw
                                    │   ├── netbox.caffeinated-labs.net
                                    │   │   └── secrets_host_root_pw
                                    │   ├── netbox.jupiter.caffeinated-labs.net
                                    │   │   ├── netbox_config_secret
                                    │   │   ├── netbox_db_password
                                    │   │   ├── netbox_superuser_password
                                    │   │   └── secrets_host_root_pw
                                    │   └── nginx.jupiter.caffeinated-labs.net
                                    │       └── secrets_host_root_pw
                                    └── static
                                        ├── secrets_netbox_token
                                        └── secrets_proxmox_api

Inner workings

VM/Container

  • Spawn on hypervisor/
    Configure Hardware
  • Network configuration
  • Default passwords
screenshot proxmox

Inner workings

Basic Configuration

  • Install updates & default packages
  • Configure Firewall & ssh
  • Create/Manage users

                                PLAY [Provision Host]
                                TASK [Gathering Facts]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [apt update & upgrade]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [Install default debian packages]
                                ok: [cccamp23.jupiter.caffeinated-labs.net] 
                                    => (item={'name': 'vim'})
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item={'name': 'sudo'})
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item={'name': 'git'})

                                TASK [Install ufw]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [Deny all incoming traffic]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [Allow outgoing traffic]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [Allow internal traffic]
                                skipping: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [Setup up allowed external traffic]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item={'service': 'sshd', 'port': 22, 'proto': 'tcp'})

                                TASK [Setup service {{ service.name }}]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item={'id': 3, 'display': 'netbox (TCP/80, 443)'})

                                TASK [Reload firewall]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [ensure root password is set correctly]
                                changed: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [allow passwordless sudo for sudo group]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

Inner workings

DNS

  • Add DNS entries
    • A/AAAA
    • CNAME
    • SSHFP
    • ...

                            PLAY [Provison DNS]

                            TASK [Get SSHFP (RSA)]
                            ok: [cccamp23.jupiter.caffeinated-labs.net]

                            TASK [Get SSHFP (ECDSA)]
                            ok: [cccamp23.jupiter.caffeinated-labs.net]

                            TASK [Get SSHFP (ED25519)]
                            ok: [cccamp23.jupiter.caffeinated-labs.net]

                            TASK [Collect monitoring targets]
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'target': 'cccamp23.jupiter.caffeinated-labs.net', 'port': 9100})

                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'target': 'cccamp23.jupiter.caffeinated-labs.net', 'port': 9100})

                            TASK [Collect base records]
                            skipping: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item={'name': 'cccamp23.jupiter', 'type': 'A', 'content': '__omit_place_holder__3b7d895ee8b90d3b20644e2e16ba2e9ebf89b72a', 'when': False})
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'name': 'cccamp23.jupiter', 'type': 'AAAA', 'content': '2001:67c:b54:101:10:100::', 'when': True})
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'name': 'cccamp23.jupiter', 'type': 'SSHFP', 'content': '1 2 __omit_place_holder__3b7d895ee8b90d3b20644e2e16ba2e9ebf89b72a'})
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'name': 'cccamp23.jupiter', 'type': 'SSHFP', 'content': '3 2 __omit_place_holder__3b7d895ee8b90d3b20644e2e16ba2e9ebf89b72a'})
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'name': 'cccamp23.jupiter', 'type': 'SSHFP', 'content': '4 2 __omit_place_holder__3b7d895ee8b90d3b20644e2e16ba2e9ebf89b72a'})
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                                    => (item={'name': '_prometheus._tcp.prometheus.jupiter', 'type': 'SRV', 'content': '10 0 9100 cccamp23.jupiter.caffeinated-labs.net.', 'when': True})

                            TASK [Deploy DNS]
                            ok: [cccamp23.jupiter.caffeinated-labs.net -> localhost({{ inventory_hostname }})]
                            +cccamp23.jupiter.caffeinated-labs.net.         AAAA            2001:67c:b54:101:10:100::
                            +cccamp23.jupiter.caffeinated-labs.net.         SSHFP           1 2 d4216b67ba0ae84136f7c2696337a330662d96952f4f133bca03f701e2218c50
                            +cccamp23.jupiter.caffeinated-labs.net.         SSHFP           3 2 f98ffb902011d94b1e849014dfecb7773bbbac310d8ff113ba7eee316fd603ed
                            +cccamp23.jupiter.caffeinated-labs.net.         SSHFP           4 2 71265693c673c169f172c4c7e0e625ee2186a7cdc06757cdef99fcc8c7ab666c
                            +_prometheus._tcp.prometheus.jupiter.caffeinated-labs.net.              SRV             10 0 9100 cccamp23.jupiter.caffeinated-labs.net.

Inner workings

Services

  • Install and configure
    • Databases
    • Dependencies
    • Software
    • Configfiles
    • Useraccounts

                                PLAY [Setup netbox]

                                TASK [Gathering Facts]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.base.postgresql : Ensure postgres user is created]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.base.postgresql : Install postgresql and dependencies]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=postgresql)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=python3-psycopg2)

                                TASK [caffeinated_labs.base.postgresql : Create user]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=netbox)

                                TASK [caffeinated_labs.base.postgresql : Create database]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=netbox)

                                TASK [caffeinated_labs.base.redis : Install redis]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Ensure netbox user is created]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Ensure base directory for netbox is created]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Install netbox dependencies]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=python3)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=python3-pip)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=python3-venv)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=python3-dev)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=build-essential)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=libxml2-dev)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=libxslt1-dev)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=libffi-dev)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=libpq-dev)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=libssl-dev)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=zlib1g-dev)

                                TASK [caffeinated_labs.services.netbox : Ensure netbox binary exists and has the correct version]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Install netbox from github releases]
                                skipping: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Template config]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Template local_requirements.txt]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Template gunicorn]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

                                TASK [caffeinated_labs.services.netbox : Template systemd services]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=netbox.service)
                                ok: [cccamp23.jupiter.caffeinated-labs.net]
                                    => (item=netbox-rq.service)

                                TASK [caffeinated_labs.services.netbox : Flush handlers]

                                TASK [caffeinated_labs.services.netbox : Create superuser]
                                ok: [cccamp23.jupiter.caffeinated-labs.net]

Inner workings

Monitoring

  • Register host at monitoring services
    • Prometheus
    • Grafana
    • ...
prometheus webinterface

Inner workings

De-provisioning

  • Remove host from hypervisor
  • Delete DNS entries
  • un-register from monitoring
  • ...
screenshot deprovisoning

Demonstration

Watch it happen

Thanks!



Slides: jadyn.dev/talks

Code: caffeinated-labs.org

Q&A