[{"_1":2,"_152":-5,"_153":-5},"loaderData",{"_3":4,"_13":14,"_21":22,"_131":132},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","routes/_layout",{"_15":-5,"_16":17,"_18":17,"_19":11,"_20":11},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_23":24,"_25":-5,"_26":11,"_27":11,"_28":29,"_30":31,"_32":-5,"_33":11,"_34":35,"_84":-5,"_85":86,"_124":-5,"_125":11,"_126":17,"_127":11,"_128":-5,"_129":130},"canonicalUrl","https://hub.docker.com/r/ilyalesikov/kustomize/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","ilyalesikov","reponame","kustomize","cloudBuildSource","isAutobuildsAvailable","repo",{"_36":37,"_38":39,"_41":42,"_43":44,"_45":11,"_46":47,"_48":49,"_50":51,"_53":31,"_28":29,"_54":55,"_56":40,"_57":58,"_59":60,"_61":62,"_63":-5,"_64":58,"_65":66,"_67":11,"_68":29,"_69":70,"_75":11,"_76":77,"_81":29,"_82":83},"categories",[],"content_types",[40],"image","date_registered","2020-01-12T12:59:07.139827Z","description","","is_private","last_updated","2020-10-13T00:22:02.600376Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[52],"application/vnd.docker.container.image.v1+json","name","pull_count",1467,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","> Demonstration of complete, fully-featured CI/CD and cloud automation for microservices, done with GCP/GKE\n\n## Features\n\n* Multistage deployments (staging, prod)\n* Canary deployments\n* Horizontal pod/instance autoscaling\n* Rollbacks, self-healing\n* Distributed tracing, monitoring, logging, profiling, debugging\n\nSetup/deployment is heavily automated so it will be easy for you to deploy it by yourself using [GCP account with Free Trial](https://cloud.google.com/free)\n\n## Contents\n1. [Features](#features)\n1. [Software](#software)\n1. [How it works](#how)\n1. [Quick start](#quick-start)\n1. [Looking around](#looking-around)\n1. [Cleanup](#cleanup)\n1. [Implementing this in the real-world](#real-world)\n1. [Known issues](#known-issues)\n1. [Halp?](#halp)\n\n## Software\n\n| What | For |\n|-----------------------------------------------------------------|-------------------------------------------------------|\n| Terraform, Terragrunt | Cloud automation |\n| Kubernetes (GKE), Kustomize | Container orchestration |\n| Google Cloud Build | CI |\n| ArgoCD, Argo Rollouts | CD |\n| Google Stackdriver | Monitoring, logging, tracing,
profiling, debugging |\n| Cloud KMS, Container Registry,
Storage and other GCP goodies | |\n\nAlso we are using [10 microservices from Google](./third-party/microservices) with built-in instrumentation for Stackdriver\n\n## How it works (simplified)\n\n![Diagram](./res/diagram.png)\n\n## Quick start\n\n1. You need [GCP account with Free Trial](https://cloud.google.com/free) activated\n1. You need [GitHub account](https://github.com/join)\n1. Fork this repo (we can't setup GCB triggers for repositories you don't own)\n1. You need Docker installed (any OS)\n\n1. Run and attach to the docker container:\n ```bash\n # Change this to the owner of the forked \"gke-demo\" repo, don't leave it like this\n GITHUB_USERNAME=ilya-lesikov\n\n # Run container with all the tooling we need:\n # NOTE: you can change \"TF_VAR_project_id\" in this command to point to the existing GCP project\n docker run -d --name gke-demo \\\n -e TF_VAR_project_id=gke-demo-$GITHUB_USERNAME \\\n -e TF_VAR_github_demo_owner=$GITHUB_USERNAME \\\n ilyalesikov/gke-demo\n\n # Attach to the container\n docker exec -it gke-demo bash\n ```\n\n1. Prepare for cloud provisioning (this is run from the inside of the container):\n ```bash\n # Clone the repo you forked\n git clone --recursive https://github.com/${TF_VAR_github_demo_owner}/gke-demo\n\n # Run this and follow the instructions on your screen.\n # This will authorize us to access your GCP account and the \"gke-demo\" repo you forked.\n ./gke-demo/scripts/prepare.sh && source /root/.bashrc\n ```\n\n1. Provision our cloud infrastructure with Terraform/Terragrunt:\n > On any transient errors (e.g. SSL/TLS errors or `remote server closed connection`) just rerun the `terragrunt` command. Terragrunt handles _most_ of these automatically, but Terraform sucks so much it'll need 10 wrappers to be truly reliable\n ```bash\n cd gke-demo/terraform/environments\n terragrunt apply-all --terragrunt-non-interactive\n ```\n\n1. Build and deploy **all** of our applications:\n ```bash\n git tag -d release_all\n git push --delete origin release_all\n git tag release_all\n git push origin release_all # This will trigger our CI/CD\n ```\n\n1. Opening this page should start creation of Monitoring workspace and will activate Stackdriver: https://console.cloud.google.com/monitoring/dashboards\n\n1. Now just wait for the build to complete: https://console.cloud.google.com/cloud-build/builds\n\n

\n\n1. Works now!\n\n## Looking around\n\nFirst, switch to our production cluster:\n```bash\nkubectl config use-context \"gke_${TF_VAR_project_id}_europe-west2-a_cluster-demo-prod\"\n```\n\nCheck if our app is synced and healthy:\n\n```bash\nargocd app get hipstershop-prod\n```\n\n![app-get](./res/app-get.png)\n\n---\n\nList our canary rollouts:\n\n```bash\nkubectl argo rollouts list rollouts\n```\n\n

\n\n---\n\nCheck out details for some particular rollout/microservice:\n\n```bash\nkubectl argo rollouts get rollout adservice\n```\n\n

\n\n---\n\nWe even have a neat web-interface to manage our applications lifecycle, do rollbacks, etc:\n\n```bash\nIP=\"$(argocd context | awk 'NR==2 {print $3}')\"\nPASS=\"$(kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f2)\"\nprintf '\\nThe web-interface is here: https://%s, username is \"admin\", password is \"%s\"\\n\\n' \"$IP\" \"$PASS\"\n```\n\n![argocd-webui](./res/argocd-webui.png)\n\n---\n\nNow buy something in our \"Hipstershop\" application to produce some data for Stackdriver:\n\n```bash\nIP=\"$(kubectl get service frontend-external | awk 'NR==2 {print $4}')\"\nprintf '\\nApplication is here: http://%s\\n\\n' \"$IP\"\n```\n\n![hipstershop](./res/hipstershop.png)\n\n### Stackdriver\n\n#### Monitoring\n\nSimple k8s monitoring dashboard:\n\nhttps://console.cloud.google.com/monitoring/dashboards/resourceList/kubernetes\n\n![dashboard](./res/dashboard.png)\n\nThere are lots of metrics out of the box, thanks to GCP, GKE, Kubernetes, Istio and instrumentation on applications side:\n\nhttps://console.cloud.google.com/monitoring/metrics-explorer\n\n![metrics](./res/metrics.png)\n\n#### Distributed tracing\n\nhttps://console.cloud.google.com/traces/list\n\n![trace](./res/trace.png)\n\n#### Profiling\n\nhttps://console.cloud.google.com/profiler\n\n![profiler](./res/profiler.png)\n\n#### Debugging\n\nhttps://console.cloud.google.com/debug\n\n![debugger](./res/debugger.png)\n\n## Cleanup\n\nThis should destroy everything, except Terraform remote state bucket and enabled services/APIs:\n```bash\ncd /git/gke-demo/terraform/environments\nterragrunt destroy-all --terragrunt-non-interactive\n```\n\nAs an alternative, this will completely delete the project, cleaning up everything we've created:\n```bash\ngcloud projects delete $TF_VAR_project_id\n```\n\nStop and remove the container with the tooling from your system (run this outside the container):\n```bash\ndocker rm -f gke-demo\n```\n\nYou might want to remove GCB application from your GitHub account too.\n\n## Implementing this in the real world\n\nThis project has some nice (and useful in production systems) things implemented, but this is nevertheless a demonstration. What I would do differently if this would be making me $$$:\n\n1. Dump Google Cloud ~~Abomination~~Build. The worst thing in this demo I worked with so far. In the end it feels more like a one big ugly shell script split into chunks, each of them executed in a separate container. When it works, it works, but... it doesn't even have dependencies between the builds and no sane way to handle concurrency. Almost no builders, existing ones are as sophisticated as `RUN apt install terraform, ENTRYPOINT terraform`.\n\n Check out Concourse, Drone CI, Spinnaker, or if you are going 100% Kubernetes try something like Argo stack, it was a breeze to work with ArgoCD/Rollouts.\n\n1. The repo should be split at least in two — one for the shared infrastructure automation code (e.g. Terraform), the other one for microservices. I would say that you better split your microservices in different repos too, this will allow for cleaner CI, though I heard about people using monorepos. There is still some glue needed to avoid versioning mess and race conditions in your CI/CD when you have multiple microservices developed, tested and deployed simultaneously.\n\n1. You'll need to streamline developers workflow on their local machines with something like Minikube and Skaffold. Developer should be able to deploy microservices and accompanying software (DBs) that is needed to properly develop/test his own microservice on his local machine, to minimize testing in staging environment (it's much slower and more expensive). It will be a sort of a replacement for `docker-compose.yml` files in the root of application repo that helps you deploy DBs and stuff and maybe even microservices you are heavily depend on.\n\n1. Versions are pinned as precisely as they could in all the places, this is just to keep this demo working without much maintenance. Updating these pinned versions programmatically is always a big PITA, but this is a right way and you need to figure it out. As a simplified workflow you can pin to the minor (not patch) version, so that patch updates applied automatically. This way you'll need to go through your code and repin minor versions manually from time to time. And of course uncontrolled patch updates can break things sometimes, but that will happen rarely, so it's a kind of a trade-off between reliable and simple.\n\n1. For all of this to actually be reliable and resilient you need comprehensive testing on many levels, including E2E and load testing.\n\n1. You might not want to instantly and without any confirmation deploy to production every thing that passed staging environment.\n\n1. Industry-standard Prometheus might be a better choice than proprietary Stackdriver.\n\n1. I didn't use more traditional CMS like Ansible, since all my needs were covered by Terraform and Kubernetes. It still might be useful when working with VMs, but with hosted K8S I didn't really need that.\n\n1. There might be something else ~~that I didn't know~~ ~~that I forgot~~ that I didn't know.\n\n1. `$ grep -RE 'TODO|FIXME'`\n\n## Known issues\n\n1. `Missing required GCS remote state configuration project` \\\n Reason: sometimes Terragrunt can't parse few keys (e.g. `project`) in `remote_state.config`. \\\n Workaround: `/git/gke-demo/scripts/terragrunt-cleanup.sh`\n\n2. Terragrunt/Terraform fails during init phase \\\n Workaround: `/git/gke-demo/scripts/terragrunt-reinit.sh` \\\n If didn't help: `/git/gke-demo/scripts/terragrunt-cleanup.sh`\n\n3. `connection reset by peer`, `connection closed`, SSL/TLS errors \\\n Reason: Terraform sucks \\\n Workaround: rerun failed command\n\n## Halp? \n\nI tested it many times, but I could have missed something. \\\nIf you experience any problems, let me know and leave an issue, thanks.\n","has_starred","hub_user","immutable_tags_settings",{"_71":11,"_72":73},"enabled","rules",[74],".*","is_automated","permissions",{"_78":17,"_79":11,"_80":11},"read","write","admin","user","storage_size",151688078,"orgData","recentTags",[87],{"_88":40,"_89":90,"_91":92,"_93":94,"_95":96,"_46":113,"_114":90,"_115":29,"_116":52,"_53":117,"_118":119,"_120":104,"_121":113,"_122":62,"_123":17},"content_type","creator",8690530,"full_size",75841273,"id",82336031,"images",[97],{"_98":99,"_100":101,"_102":44,"_103":104,"_105":106,"_107":108,"_109":44,"_110":-5,"_111":92,"_59":62,"_112":-5},"architecture","amd64","digest","sha256:0cb2465c3a9810bef1396ed2e5ff149686be20750e0b37a59131f89d868d8e69","features","last_pulled","2026-09-21T05:56:51.515566007Z","last_pushed","2020-01-16T11:54:19Z","os","linux","os_features","os_version","size","variant","2020-10-13T00:22:02.280278Z","last_updater","last_updater_username","media_type","latest","repository",8303526,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"routes/_layout.r.$namespace.$reponame.tags",{"_34":133,"_140":141,"_129":150,"_151":11},{"_36":134,"_38":135,"_41":42,"_43":44,"_45":11,"_46":47,"_48":49,"_50":136,"_53":31,"_28":29,"_54":55,"_56":40,"_57":58,"_59":60,"_61":62,"_63":-5,"_64":58,"_65":66,"_67":11,"_68":29,"_69":137,"_75":11,"_76":139,"_81":29,"_82":83},[],[40],[52],{"_71":11,"_72":138},[74],{"_78":17,"_79":11,"_80":11},"tags",{"_142":60,"_143":-5,"_144":-5,"_145":146},"count","next","previous","results",[147],{"_88":40,"_89":90,"_91":92,"_93":94,"_95":148,"_46":113,"_114":90,"_115":29,"_116":52,"_53":117,"_118":119,"_120":104,"_121":113,"_122":62,"_123":17},[149],{"_98":99,"_100":101,"_102":44,"_103":104,"_105":106,"_107":108,"_109":44,"_110":-5,"_111":92,"_59":62,"_112":-5},[],"canDeleteTags","actionData","errors"]

ilyalesikov/kustomize

Tags for ilyalesikov/kustomize