# **Labtop v0.02 - Homelab GitOps & Config Management** ## Project Navigation [[Nexus - Homelab]] - [[Homelab - Labtop|Labtop: init]] - Labtop v.0.02 (this project) - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Overview|Overview]] - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Project Goals, Stories, Tasks|Project Goals, Stories, and Tasks]] - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#**Labtop v0.02 - Homelab GitOps & Config Management**#Technical Requirements Details|Technical Requirements]] - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Definitions & Glossary|Definitions & Glossary]] - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Project Approach and Timeline|Project Approach & Timeline]] - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Sprint Log|Sprint Plans]] - [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Log|Log]] ## Overview My overarching goal for homelab projects is to enable a transition to a DevOps career. The next step in achieving this goal is to develop a directory and file structure that mirrors professional DevOps teams on my server which enables management and configurations of current and future deployments. This project entails creating a directory structure configured as Git repositories and having them hosted with GitHub. Achieving this will enable GitOps CI/CD workflows across any project on the homelab going forward. Additionally, this project will be thoughtfully planned, documented, and executed following common [[Agile]] methodology practices. --- ## Project Goals, Stories, Tasks This section describes my goals, or [[Epic|epics]], for this project. Each epic will relate to [[Story|user stories]] to clarify. In turn, each story will require the completion of one or more tasks. ### Epic 1: Infrastructure Setup and testing Prepare the foundational elements needed before automation. #### Story 1.1 Validate Git and GitHub Connection [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Log|Log]] As a DevOps engineer, I want to ensure that my GitHub SSH authentication is working so that I can securely push and pull repositories. ##### Tasks: - [x] Confirm SSH authentication to [[GitHub]] (`ssh -T [email protected]`) - [x] Clone a test repo and push (`git clone` and `git push`). #### Story 1.2 Prepare a testing environment. [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Log|Log]] As a DevOps engineer, I want to set up a sandbox testing environment so that I can test automation scripts safely. ##### Tasks: - [ ] Set up a VM or container to test automation scripts. - [ ] Researching [[DevContainer]] - [ ] Dependency, [[Docker]], and [[Container]] - [ ] Dependency: supported [[Integrated Development Environment|IDE]] VSCode - [ ] [[Docker]] to run on laptop. - [ ] Install docker desktop - [ ] pull image - [ ] run image - [ ] access container - [ ] How will the script be developed? - [ ] Local IDE, project file mounted to container? - [ ] Local IDE, project file pushed to GH remote repo and pulled to container for testing? - [ ] Container code editor ? - Not advised will lose all work if container goes down. - [ ] How do I organize files on local? - [ ] Laptop currently follows PARA framework for directory structure. - [ ] Project file path 1. projects > homelab > Labtop > v0.02 > scripts > dev (test, stage, prod) - [ ] Docker path: ?? should this be a dev container? i don't thinkso? maybe? - [ ] Configure a snapshot or rollback mechanism. - [ ] `docker commit` operates as snapshot - [ ] researching Roll Back procedure - [ ] docker commit essentially creates a new image reflecting the state of the current image. This is the snapshot that enables a rollback to a fresh state. - [ ] Unclear if mounted volumes would also rollback. ### DESCOPED ~~Epic 2: Directory Structure Initialization and Automation~~ Create a DevOps-aligned directory structure with an automated setup. #### Story 2.1: Define the Target Directory Structure. As a DevOps engineer, I want to define a clean home directory structure so that I can manage projects, configurations, and automations efficiently. ##### Tasks: - [ ] Research industry standard DevOps directory structures. - My research has shown that there really isn't a standard structure. - What matters is good structure within project repositories. - What do the greats use? - [ ] Design the folder structure for repositories, scripts, logs, backups, etc. ```shell /home/<user>/ ├── devops-lab/ # Main workspace for homelab projects │ ├── infra/ # Infrastructure as Code (Terraform, Ansible, etc.) │ │ ├── terraform/ # Terraform scripts │ │ ├── ansible/ # Ansible playbooks │ │ ├── kubernetes/ # Kubernetes manifests │ │ ├── helm/ # Helm charts │ ├── ci-cd/ # Continuous Integration / Deployment │ │ ├── github-actions/ # GitHub Actions YAML │ │ ├── gitlab-ci/ # GitLab pipelines │ │ ├── jenkins/ # Jenkins configurations │ ├── monitoring/ # Monitoring and Logging setup │ │ ├── grafana/ # Grafana dashboards │ │ ├── prometheus/ # Prometheus configuration │ │ ├── logstash/ # Logstash rules │ ├── environments/ # Environment-specific configurations │ │ ├── dev/ # Development │ │ ├── test/ # Testing │ │ ├── prod/ # Production │ ├── scripts/ # Automation scripts (Bash, Python, etc.) │ ├── dotfiles/ # Dotfiles for personal environment (Bash, Zsh, etc.) │ ├── homelab-docs/ # Documentation │ │ ├── README.md # Overview of homelab setup │ │ ├── network-architecture.md # Homelab network design │ │ ├── security.md # Security policies and SSH setup │ ├── research/ # Research and experiments │ ├── sandbox/ # Temporary test environment │ ├── backup/ # Backup scripts and configurations │ ├── .gitignore # Ignore sensitive files │ ├── docker-compose.yml # If using Docker Compose │ ├── Makefile # Standardized automation (optional) ``` - [ ] Write a documentation file (`README.md`) explaining the structure. - [[Directory README]] --- #### DESCOPED ~~Story 2.2 Automate Directory Creation with Bash Script~~ I want to create my directory structure automatically so that I can quickly set up new machines and keep everything consistent. ##### Tasks: - [ ] Write a Bash Script to create directories - [ ] Add permissions and ownership settings - [ ] Test the script on a fresh system - [ ] Store the script in a Git repository. ### DESCOPED ~~Epic 3: GitOps and Environment Management~~ Implement **GitOps principles** by tracking configurations in Git & defining environments. #### Story 3.1: Initialize Git Repo for Homelab configs I want to track my directory structure in Git so taht I can version control and sync changes between systems. ##### Tasks - [ ] Init a Git repo: `homelab-configs.git` - [ ] create a `.gitignore` file to exclude senistive files - [ ] Set up a `README.md` explaining the repo's purpose - [ ] Push to GH #### Story 3.2: Set up Separate Environments (Dev, Test, Prod) As a DevOps Engineer I want to manage Dev, Test, and Prod environments separately so that I can test changes before applying them to production. ##### Tasks - [ ] Investigate DevContainers. - Does this replace Dev and Test environment separation? - [ ] Decide between Git branches vs Separate Repositories for Environments - [ ] If separate repositories then create structure like: - [ ] Alternatively implement Git Branching where Main branch is the production branch and Dev, and Test branches managed by `git checkout -b dev` for development which gets a PR to Test branch and then a PR to Main/Prod. - [ ] Script a deployment script for pulling and applying updates (This feature to be handled by CI/CD pipeline in the future) #### Story 3.3: Automate Environment Syncing As a DevOps engineer, I want to ensure my homelab automatically updates so that all my configurations are consistent across environments. ##### Tasks - [ ] Set up GH Actions workflow - [ ] Write a `cron` job to pull changes periodically - [ ] Test syncing across environments. ### DESCOPED ~~Epic 4: Configuration Management and Secrets management~~ Use dotfiles, secrets management, and automation to maintain system configs. #### Story 4.1 Manage dotfiles in Git As a DevOps engineer, I want to track my dotfiles (`.bashrc`, `.vimrc`, etc) so that I can apply my configurations easily on any machine. ##### Tasks - [ ] Create a **separate Git repo (`homelab-dotfiles.git`)**. - [ ] Use a **bare Git repository** or a dotfile manager (`stow` or `chezmoi`). - [ ] Set up **symlinks** to keep dotfiles in place. #### Story 4.2: Securely Manage Secrets As a DevOps Engineer, I want to securely store environment variables and API keys so that my sensitive credentials are protected. ##### Tasks - [ ] Research **GPG encryption** or **Hashicorp Vault** for secrets management. - [ ] Store encrypted files in `homelab-vault/` in a separate **private Git repo**. - [ ] Use `.gitignore` to prevent secret leaks. #### Story 4.3: Define "Configuration Drift" Monitoring and Detection As a DevOps engineer, I want to detect configuration drift so that my homelab remains in sync with Git. ##### Tasks - Define Configuration Drift Criteria - [ ] Identify which system files and directories should be monitored (e.g., `/etc/`, `/home/user/git/homelab-configs/`). - Implement Git-Based Drift Detection - [ ] Write a basic script that runs `git status --porcelain` to detect uncommitted changes. - [ ] Automate the script to log output in a `drift.log` file. - Set Up Automated Monitoring - [ ]  Create a **cron job** to run the script every 10 minutes. - [ ] Configure it to send a **console notification or email alert** when drift is detected. --- ## Technical Requirements Details ### Hardware: The labtop server A remote client machine. ### Software: - Linux Server OS, Git, Bash Terminal - All software in its current state is fine. ### Services - SSH - Git installed on host - GitHub with SSH authentication access. (was established in the last project but not tested.) --- ## Definitions & Glossary ### Definition of Ready - A task is ready to be worked on when: - All prior tasks in sequence are completed. - Any dependencies (e.g., software installation, SSH authentication) are in place. - A clear acceptance criterion is defined for the task. ### Definition of Done A task is considered done when: - The defined success metric for the task is achieved. - The result is tested and validated (e.g., Git commits and pushes verified). - Any required documentation (such as a README update) is completed. - The system state matches the expected outcome without requiring manual intervention. ### Glossary of terms - [[Agile]] | [[Automation]] - [[Backlog]] - [[DevOps]] - [[Epic]] - [[Git]] | [[GitHub]] | [[GitOps]] - [[Initiatives]] - [[Retrospective]] | [[Rollback]] - [[Scope]] | [[Snapshot]] | [[Sprint]] - [[Task]] - [[Story|User Story]] --- ## Project Approach and Timeline My approach in completing this project follows industry norms for Agile teams. ### Organization The project is structured using an Agile framework, where work is divided into: - Initiatives: Broad directional or strategy goals - Epics: High-level goals - User Stories: Functional requirement derived from the higher level goals - Tasks: Actionable work items needed to implement User Stories. ### Planning & Backlog Management To maintain a structured workflow, I will adopt a Sprint-based execution model with an actively managed backlog of pending work. - Backlog Grooming: Before each sprint begins, I will review and refine upcoming tasks, ensuring they are well-defined, properly estimated, and prioritized based on project needs. - The backlog for Homelab projects (including Labtop) are in the [[Nexus - Homelab]] page. - Task Estimation: I will use a point system where 1 point represents an estimated workday of effort, with the smallest unit being 0.5 points. ### Sprint Execution: Sprints provide a time-boxed structure for completing tasks within the project. - Each sprint has a fixed duration of 2 weeks (10 working days), ensuring a consistent rhythm for progress. - Each sprint has a maximum capacity of 10 points, aligning effort with available time. - At the start of each sprint, tasks from the backlog are selected based on priority and available capacity. - For the project, only one task is worked on at a time, and tasks must be completed sequentially to maintain workflow continuity. - At the end of each sprint, a Retrospective will be conducted to assess progress, identify improvements, and refine the next sprint. - Future Sprint Planning: Insights from retrospectives will be incorporated into backlog refinement and planning for the next sprint. --- ## Sprint Log ### Sprint 01 19-Feb to 05-Mar 202 >[! warning] Project was overhauled due to technical acumen limitations. >- Sprint moving forward to end date with ad hoc projects >- EPIC 3 and 4 moved to backlog. >- Revising to include initiatives | Status | Sprint | Epic.Story | Task | Story Points (est) | Story Points (actual) | | ------------ | ------ | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------ | --------------------- | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 1.1 Validate Git and GitHub Connection\|1.1]] | Confirm SSH authentication to GitHub | 0.5 | <0.5 | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 1.1 Validate Git and GitHub Connection\|1.1]] | Clone a test repo and push | 0.5 | <0.5 | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 1.2 Prepare a testing environment.\|1.2]] | Set up a VM or container to test automation scripts | 1.0 | 2.0 | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 1.2 Prepare a testing environment.\|1.2]] | Configure a snapshot or rollback mechanism. | 1.0 | 2.0 | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 2.1 Define the Target Directory Structure.\|2.1]] | Research standard DevOps directory structures. | 0.5 | 1.0 | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 2.1 Define the Target Directory Structure.\|2.1]] | Design a folder structure for repositories, scripts, logs, and backups. | 0.5 | 0.5 | | Done | 01 | [[Homelab - Labtop v0.02 - Homelab GitOps & Config Management#Story 2.1 Define the Target Directory Structure.\|2.1]] | Write a `README.md` documenting the structure. | 1.0 | 0.5 | | Descoped | 01 | 2.2 | Write a Bash script (`setup_structure.sh`) to create directories. | 2.0 | | | Descoped | 01 | 2.2 | Set directory permissions and ownership (`chmod` & `chown`). | 0.5 | | | Descoped | 01 | 2.2 | Test the script in a **clean environment** (e.g., a new VM). | 0.5 | | | Descoped | 01 | 2.2 | Store the script in a Git repository. | 0.5 | | | Descoped | 01 | 3.1 | Initialize a Git repository (`homelab-configs.git`). | 0.5 | | | Descoped<br> | 01 | 3.1 | Create a `.gitignore` file to exclude sensitive files. | 0.5 | | | Descoped | 01 | 3.1 | Set up a `README.md` explaining the repo’s purpose and Push the repo to GitHub. | 0.5 | | | Working | 01 | [[PostgreSQL]] | Ad Hoc, deploy and configure postgresql on server for remote access | 0.5 | 0.5 | | Waiting | 01 | Retro | | 0.5 | | #### Sprint 01 Team Just me. #### Sprint 01 Retro ==Due: Weds March 5th== ### Sprint 02 To be planned | Status | Sprint | Epic.Story | Task | Story Points (est) | Story Points (actual) | | ------ | ------ | ---------- | ---- | ------------------ | --------------------- | | | | | | | | <hr> ## Insights From Project Again needing to rescope my project because it is not fully aligned very well to my roadmap document for which I should be studying. - I am not fully utilizing my time effectively because I am learning things that I do not need to know at this stage. - Scripting will come later, after the fundamentals - DevOps level of anythjing will come later, after the fundamentals. - Given my current working situation, time to drop down and focus on Linux. - Successful outcome is earning LPIC-1 --- ## Log [[Nexus - Homelab#Labtop Project Backlog|Labtop Backlog]] #### 28-FEB - modifed second client `.ssh confg` - See [[Homelab - Deploying PostgreSQL]] : adhoc side quest, needed to practice for interviews going forward. #### 27-FEB - modified client `~/.ssh/config` file with the following host alias ```bash Host labtop HostName 192.... ``` - Now i won't need to remember the ip address. - that is so lazy - installed [[PostgreSQL]] to the server. - installing PGADMIN on client device/local to access. #### 21-FEB-2025 - `ubtubtu_test_env` container is up and running. - Added VIM to environment. - Researching project directory best practices - IE where do I work on the automation scripts and how do I mount that one directory to the test environment so the OS can see the file to execute from local disk - OR should i use GIT and push the file to remote and then pull it down in the container? This is how I imagine I will deploy it to the server when the time comes - Researching `docker commit` command for the rollback mechanism. #### 20-FEB-2025 - researching creating sandbox environment. Emphasis is staying close to DevOps practices. - Researching VM's - Though, I am sure my laptop has compute, both my dev laptop and the labtop are macbook air models of different vintages. - Researching Containers - Learned about [[Docker]] and the [[Container]] concept and use case to run as testing environments. - Learned about [[DevContainer]] to set up to ensure reproducibility. - Added backlog item re: DevContainer - Both concepts anchor to [[Container]], and both concepts are like infrastructure as code. - Used ChatGPT a lot today to explain differences because I was getting confused. - Working on spinning up a docker container with linux in it as this is the task for this sprint. - Docker is now on the new laptop. - Did some work within ChatGPT to focus this project by creating [[Initiatives|initiatives]] and rescoping. #### 19-FEB-2025 - began sprint 1 - Task 1.1 - Confirm SSH authentication to GitHub - done. - Tested SSH connection. Frustratingly complicated passphrase. - Added to backlog user story for YubiKey use instead of passphrase. - Task 1.1 Clone a test repo and push - To accomplish this, a repo was created on remote with a readme.md file. - Then, I cloned the repo down to local (server) via `git clone [email protected]:nvbergen/test.git` - Once the repo was on local machine, I used VIM to edit the readme file by adding some simple text and then pushed back up to remote. - Re-wrote [[Nexus - Homelab]], [[Homelab - Labtop|labtop: init]], and most of this file to be more clear. #### 18-FEB-2025 - Adopted Agile Methodology and planned the project utilizing Epics, Stories, and Tasks. - The project execution will be timed in sprints that are 10 days (2 weeks) in length. Just like on my last team Sprints open and close on a single day of the week. - The Sprint will open on a Wednesday - ChatGPT was utilized to aid in concept discovery and to author specific tasks to meet overall goals for which I had input. #### 17-FEB-2025 - Rescoped this item to be its own project for learning purposes. - Applied project charter framework to designing this project. <hr> >[!summary-top] Project Summary & Key Takeaways >_This summary will appear at the top if using the Cornell style note tools. >Only 3 lines really appear so try to condense as much as possible. >Summary info goes here when completed._