Startup Infra for Small Self-hosted Projects

This repository provides Ansible playbooks to set up a minimal infrastructure for a simple self-hosted application. Ideal for small hobby projects. I made this repository a place to Learn about DevOps and Cloud Infrastructure. You have all the tutorial you need to get started.

Development Environment Setup

This project provides a convenient dispatcher command, ibt, to help you set up your local development environment and manage project tools. Additionally, a Gitpod configuration is available for a cloud-based, ready-to-use environment.

Unified Tools Bash script: ibt

The ibt command (Infra Bootstrap Tools) is a shell function that provides a unified interface to the main project scripts with subcommands and auto-completion support.

Subcommands:

• ibt setup [tool ...] — Install required tools and dependencies (see below)
• ibt stacks [args ...] — Manage and run infrastructure stacks
• ibt tools [args ...] — Use Docker-based aliases for Ansible, AWS CLI, etc.

Auto-completion:

Tab-completion is available for subcommands and for the setup tool list (e.g., ibt setup [TAB]).

To enable ibt in your shell:

source ./bin/bash/ibt.sh

You can add this line to your ~/.bashrc or ~/.bash_profile for persistence.

Example usage:

# Install pre-commit and Ansible
ibt setup pre-commit ansible

# List available stacks
ibt stacks list

# Use Docker-based Ansible
ibt tools dasb --version

Nix Shell

Prerequisites: This requires Nix to be installed on your system.

For a fully automated and reproducible development environment using Nix, you have two options:

Option 1: Using Traditional Nix Shell (Most Compatible)

Use this option if nix develop fails with experimental Nix feature 'nix-command' is disabled.

nix-shell bin/nix/shell.nix

# Or select a specific shell
nix-shell bin/nix/shell.nix --argstr shell flux

Shell overview:

• default: general development shell with Python, Docker, Git, and pre-commit
• ansible: adds Ansible tooling and installs Galaxy dependencies
• flux: adds jq, kubectl, helm, kind, and flux for Kubernetes/Flux work
• docs: adds Hugo and Go for website/documentation work
• full: includes everything when you need the broadest environment

The shells are ready to use after their hooks complete. Python-based shells create and activate a local virtual environment (.venv) to avoid conflicts with the Nix-provided Python.

Option 2: Using Nix Flakes (Optional)

Nix Flakes provide a more modern and reproducible approach, but require the nix-command and flakes experimental features:

# Enable flakes if not already enabled (add to ~/.config/nix/nix.conf or /etc/nix/nix.conf):
# experimental-features = nix-command flakes

# Enter the default general-purpose development environment
nix develop

Task-focused shells are also available:

# Infrastructure / Ansible work
nix develop .#ansible

# Flux / Kind / Kubernetes work
nix develop .#flux

# Website / Hugo work
nix develop .#docs

# Full kitchen-sink environment
nix develop .#full

GitHub Codespaces

The repository includes a .devcontainer/devcontainer.json that keeps the current base image (mcr.microsoft.com/devcontainers/base:ubuntu-24.04), installs Nix by default, and enables flakes (nix-command flakes) out of the box. Docker access is enabled as well.

You can use:

nix develop
nix develop .#ansible
nix develop .#flux

If you are in a different environment where flakes are not enabled, use:

nix-shell bin/nix/shell.nix

Gitpod

Alternatively, you can use Gitpod to get a pre-configured development environment in your browser. Click the button below to get started:

🚀 Quick Start

Got your prerequisites and secrets configured?

Deploy your entire infrastructure with a single command:

make up

This command will:

• Provision infrastructure on DigitalOcean using Terraform.
• Install Docker on all hosts.
• Initialize a Docker Swarm cluster.
• Deploy Caddy and Portainer applications.

To tear down the infrastructure:

make down

For a detailed step-by-step guide, including prerequisites and secret management options, please refer to our Full Getting Started Guide.

To understand the underlying Ansible concepts, check out Understanding Ansible Concepts.

Articles and Tutorials

The articles/tutorials are divided into sections.

• Introduction -> What are the tools to manage infrastructure? Perfect to learn the basis.
• How-tos -> Good takeaway from this project - Answers many questions you could encounter in the future
• Deepening Understanding -> Learn more about each application used in this setup (Portainer, Graphana, Caddy, etc.)
• Local Experimentation & Advanced Topics: Deeper dives into specific setups and components.

I used DigitalOcean for experiments because it is cheap, but any cloud provider should work, as we are mainly playing with Virtual Machines. You can even get $200 DigitalOcean free credit when starting

• Don't forget to delete everything after a tutorial if you don't want to add unnecessary costs

Tools Introduction

• WIP: 📚 1: What is Terraform and why you might need it.
• 📚 2: What is Terraform Cloud and why you might need it.
• WIP: 📚 3: What is Github Action and why you might need it.
• WIP: 📚 4: What is Ansible and why you might need it.
• WIP: 📚 5: What is Ansible AWX and why you might need it.

Learn the Tools

• 🌍 How to configure GitHub Environments with Terraform?
• 🏭 How to provision VM on Digital Ocean with Terraform?
• 🔏 How to create SSH keys with Terraform?
• 🗺️ How to create Ansible Inventory with Terraform?
• 👩 How to run an Ansible playbook using GitHub Action?

Local Experimentation & Advanced Topics

These articles cover setting up local test environments and exploring specific components in more detail. They are excellent for understanding the individual parts before or alongside deploying the full cloud infrastructure.

• 🧰 1: Design and Test Ansible playbook with Vagrant
  • Example code .articles/1_vagrant_101
• 🧰 2 Experimenting on Docker Swarm with Vagrant and Ansible
  • Example code .articles/2_docker_swarm_101
• WIP: 🧰 3: Automate Infrastructure provisioning with Ansible and Github action (This might be more of an advanced topic now, focusing on CI/CD aspects beyond the basic make up)

Learn about the applications used in this setup

• WIP: ☸️ 1: What is Portainer and why you might need it. (Covered by Portainer Management UI)
• WIP: ☸️ 2: What is Prometheus and why you might need it.
• WIP: ☸️ 3: What is Caddy and why you might need it. (Covered by Caddy Web Server)

Deep Dives

Below, I've included some explainers on how I solved a specific problem when I encountered it. This should demistify some of the magic happening in this repo.

Ansible

• 🏃‍♂️ 1: Speedup Ansible Playbook Pull-Requests by only running affected roles
• 🏃‍♂️ 2: How to rotate Docker Secrets with Ansible
• 🏃‍♂️ 3: How to implement Pull-Request locking for Ansible

Ansible Collection

This repository is also available as an Ansible Collection on Ansible Galaxy, allowing you to easily reuse the roles in your own Ansible projects.

Collection Name: xnok.infra_bootstrap_tools

Installation

To install this collection from Ansible Galaxy, use the following command:

ansible-galaxy collection install xnok.infra_bootstrap_tools

Usage

Once installed, you can use the roles from this collection in your playbooks. For example, to use the docker role:

- hosts: all
  become: yes
  roles:
    - role: xnok.infra_bootstrap_tools.docker
      # Optional: specify variables for the role
      # docker_users:
      #   - your_username

Refer to the README.md file within each role's directory (ansible/roles/[role_name]/README.md) for detailed information on specific roles, their variables, and dependencies.

You can find the collection on Ansible Galaxy: xnok.infra_bootstrap_tools

Release Management

This monorepo uses Changesets for automated version management and releases. Different package types (Ansible collection, Python package, Docker stacks) are automatically published when versions are bumped.

For detailed information about the release process, see RELEASE.md.

Quick Start

To create a changeset for your changes:

npx changeset add

For more details on package types, publishing workflows, and troubleshooting, see the full release documentation.

Architecture

Handy toolchain

Do you want to go fast? Too lazy to set up your local environment?

Then, use the tools from a Docker container. I included a simple Toochain in this repository and a useful alias for it.

Use common infrastructure tools in docker with:

• docker_tools_alias.sh

source ./bin/docker_tools_alias.sh

use dasb for ansible in docker
use dap for ansible-playbook in docker
use daws for awscli in docker
use dpk for packer in docker
use dtf for terraform in docker
use dbash for bash in docker

Tools Showcase

This project leverages several tools to streamline development and improve code quality. Here's a brief overview of some of them:

• Pre-commit: We use pre-commit hooks to automate linting, formatting, and other checks before code is committed. This helps maintain code consistency and catch errors early. Configuration can be found in .pre-commit-config.yaml.
• 1Password CLI: For securely managing sensitive information like API keys and passwords, the 1Password CLI can be integrated into your workflow. The setup script provides an option to install it.
• Boilerplate: Gruntwork Boilerplate is used to generate répétitive code structures, ensuring consistency and saving time. You can find boilerplate templates in the .boilerplates directory.
• Hugo: The project documentation website (what you're likely reading if you're on the website!) is built using Hugo, a fast and flexible static site generator. The website content is in the website/ directory.

Scale Up

With docker swarm and portainer it because easy to manager multiple nodes.