The DevOps Blog
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

390 lines
16KB

  1. .. title: Your First Minikube Helm Deployment
  2. .. date: 2019-02-10
  3. .. updated: 2019-06-21
  4. .. status: published
  5. .. tags: minikube, kubernetes, ingress, helm, prometheus, grafana,
  6. .. category: kubernetes
  7. .. slug: your-first-minikube-helm-deployment
  8. .. authors: Elijah Lazkani
  9. .. description: Deploying your first minikube helm charts.
  10. .. type: text
  11. In the last post, we have configured a basic *minikube* cluster. In this post we will deploy a few items we will need in a cluster and maybe in the future, experiment with it a bit.
  12. .. TEASER_END
  13. Prerequisite
  14. ============
  15. During this post and probably during future posts, we will be using *helm* to deploy to our *minikube* cluster. Some offered by the helm team, others by the community and maybe our own. We need to install ``helm`` on our machine. It should be as easy as downloading the binary but if you can find it in your package manager go that route.
  16. Deploying Tiller
  17. ================
  18. Before we can start with the deployments using ``helm``, we need to deploy *tiller*. It's a service that manages communications with the client and deployments.
  19. .. code:: text
  20. $ helm init --history-max=10
  21. Creating ~/.helm
  22. Creating ~/.helm/repository
  23. Creating ~/.helm/repository/cache
  24. Creating ~/.helm/repository/local
  25. Creating ~/.helm/plugins
  26. Creating ~/.helm/starters
  27. Creating ~/.helm/cache/archive
  28. Creating ~/.helm/repository/repositories.yaml
  29. Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
  30. Adding local repo with URL: http://127.0.0.1:8879/charts
  31. $HELM_HOME has been configured at ~/.helm.
  32. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
  33. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
  34. To prevent this, run ``helm init`` with the --tiller-tls-verify flag.
  35. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
  36. *Tiller* is deployed, give it a few minutes for the pods to come up.
  37. Deploy Prometheus
  38. =================
  39. We often need to monitor multiple aspects of the cluster easily. Sometimes maybe even write our applications to (let's say) publish metrics to prometheus. And I said 'let's say' because technically we offer an endpoint that a prometheus exporter will consume regularly and publish to the prometheus server. Anyway, let's deploy prometheus.
  40. .. code:: text
  41. $ helm install stable/prometheus-operator --name prometheus-operator --namespace kube-prometheus
  42. NAME: prometheus-operator
  43. LAST DEPLOYED: Sat Feb 9 18:09:43 2019
  44. NAMESPACE: kube-prometheus
  45. STATUS: DEPLOYED
  46. RESOURCES:
  47. ==> v1/Secret
  48. NAME TYPE DATA AGE
  49. prometheus-operator-grafana Opaque 3 4s
  50. alertmanager-prometheus-operator-alertmanager Opaque 1 4s
  51. ==> v1beta1/ClusterRole
  52. NAME AGE
  53. prometheus-operator-kube-state-metrics 3s
  54. psp-prometheus-operator-kube-state-metrics 3s
  55. psp-prometheus-operator-prometheus-node-exporter 3s
  56. ==> v1/Service
  57. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  58. prometheus-operator-grafana ClusterIP 10.107.125.114 80/TCP 3s
  59. prometheus-operator-kube-state-metrics ClusterIP 10.99.250.30 8080/TCP 3s
  60. prometheus-operator-prometheus-node-exporter ClusterIP 10.111.99.199 9100/TCP 3s
  61. prometheus-operator-alertmanager ClusterIP 10.96.49.73 9093/TCP 3s
  62. prometheus-operator-coredns ClusterIP None 9153/TCP 3s
  63. prometheus-operator-kube-controller-manager ClusterIP None 10252/TCP 3s
  64. prometheus-operator-kube-etcd ClusterIP None 4001/TCP 3s
  65. prometheus-operator-kube-scheduler ClusterIP None 10251/TCP 3s
  66. prometheus-operator-operator ClusterIP 10.101.253.101 8080/TCP 3s
  67. prometheus-operator-prometheus ClusterIP 10.107.117.120 9090/TCP 3s
  68. ==> v1beta1/DaemonSet
  69. NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  70. prometheus-operator-prometheus-node-exporter 1 1 0 1 0 3s
  71. ==> v1/Deployment
  72. NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
  73. prometheus-operator-operator 1 1 1 0 3s
  74. ==> v1/ServiceMonitor
  75. NAME AGE
  76. prometheus-operator-alertmanager 2s
  77. prometheus-operator-coredns 2s
  78. prometheus-operator-apiserver 2s
  79. prometheus-operator-kube-controller-manager 2s
  80. prometheus-operator-kube-etcd 2s
  81. prometheus-operator-kube-scheduler 2s
  82. prometheus-operator-kube-state-metrics 2s
  83. prometheus-operator-kubelet 2s
  84. prometheus-operator-node-exporter 2s
  85. prometheus-operator-operator 2s
  86. prometheus-operator-prometheus 2s
  87. ==> v1/Pod(related)
  88. NAME READY STATUS RESTARTS AGE
  89. prometheus-operator-prometheus-node-exporter-fntpx 0/1 ContainerCreating 0 3s
  90. prometheus-operator-grafana-8559d7df44-vrm8d 0/3 ContainerCreating 0 2s
  91. prometheus-operator-kube-state-metrics-7769f5bd54-6znvh 0/1 ContainerCreating 0 2s
  92. prometheus-operator-operator-7967865bf5-cbd6r 0/1 ContainerCreating 0 2s
  93. ==> v1beta1/PodSecurityPolicy
  94. NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
  95. prometheus-operator-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
  96. prometheus-operator-kube-state-metrics false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false secret
  97. prometheus-operator-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
  98. prometheus-operator-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
  99. prometheus-operator-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
  100. prometheus-operator-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
  101. ==> v1/ConfigMap
  102. NAME DATA AGE
  103. prometheus-operator-grafana-config-dashboards 1 4s
  104. prometheus-operator-grafana 1 4s
  105. prometheus-operator-grafana-datasource 1 4s
  106. prometheus-operator-etcd 1 4s
  107. prometheus-operator-grafana-coredns-k8s 1 4s
  108. prometheus-operator-k8s-cluster-rsrc-use 1 4s
  109. prometheus-operator-k8s-node-rsrc-use 1 4s
  110. prometheus-operator-k8s-resources-cluster 1 4s
  111. prometheus-operator-k8s-resources-namespace 1 4s
  112. prometheus-operator-k8s-resources-pod 1 4s
  113. prometheus-operator-nodes 1 4s
  114. prometheus-operator-persistentvolumesusage 1 4s
  115. prometheus-operator-pods 1 4s
  116. prometheus-operator-statefulset 1 4s
  117. ==> v1/ClusterRoleBinding
  118. NAME AGE
  119. prometheus-operator-grafana-clusterrolebinding 3s
  120. prometheus-operator-alertmanager 3s
  121. prometheus-operator-operator 3s
  122. prometheus-operator-operator-psp 3s
  123. prometheus-operator-prometheus 3s
  124. prometheus-operator-prometheus-psp 3s
  125. ==> v1beta1/Role
  126. NAME AGE
  127. prometheus-operator-grafana 3s
  128. ==> v1beta1/Deployment
  129. NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
  130. prometheus-operator-kube-state-metrics 1 1 1 0 3s
  131. ==> v1/Alertmanager
  132. NAME AGE
  133. prometheus-operator-alertmanager 3s
  134. ==> v1/ServiceAccount
  135. NAME SECRETS AGE
  136. prometheus-operator-grafana 1 4s
  137. prometheus-operator-kube-state-metrics 1 4s
  138. prometheus-operator-prometheus-node-exporter 1 4s
  139. prometheus-operator-alertmanager 1 4s
  140. prometheus-operator-operator 1 4s
  141. prometheus-operator-prometheus 1 4s
  142. ==> v1/ClusterRole
  143. NAME AGE
  144. prometheus-operator-grafana-clusterrole 4s
  145. prometheus-operator-alertmanager 3s
  146. prometheus-operator-operator 3s
  147. prometheus-operator-operator-psp 3s
  148. prometheus-operator-prometheus 3s
  149. prometheus-operator-prometheus-psp 3s
  150. ==> v1/Role
  151. NAME AGE
  152. prometheus-operator-prometheus-config 3s
  153. prometheus-operator-prometheus 2s
  154. prometheus-operator-prometheus 2s
  155. ==> v1beta1/RoleBinding
  156. NAME AGE
  157. prometheus-operator-grafana 3s
  158. ==> v1beta2/Deployment
  159. NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
  160. prometheus-operator-grafana 1 1 1 0 3s
  161. ==> v1/Prometheus
  162. NAME AGE
  163. prometheus-operator-prometheus 2s
  164. ==> v1beta1/ClusterRoleBinding
  165. NAME AGE
  166. prometheus-operator-kube-state-metrics 3s
  167. psp-prometheus-operator-kube-state-metrics 3s
  168. psp-prometheus-operator-prometheus-node-exporter 3s
  169. ==> v1/RoleBinding
  170. NAME AGE
  171. prometheus-operator-prometheus-config 3s
  172. prometheus-operator-prometheus 2s
  173. prometheus-operator-prometheus 2s
  174. ==> v1/PrometheusRule
  175. NAME AGE
  176. prometheus-operator-alertmanager.rules 2s
  177. prometheus-operator-etcd 2s
  178. prometheus-operator-general.rules 2s
  179. prometheus-operator-k8s.rules 2s
  180. prometheus-operator-kube-apiserver.rules 2s
  181. prometheus-operator-kube-prometheus-node-alerting.rules 2s
  182. prometheus-operator-kube-prometheus-node-recording.rules 2s
  183. prometheus-operator-kube-scheduler.rules 2s
  184. prometheus-operator-kubernetes-absent 2s
  185. prometheus-operator-kubernetes-apps 2s
  186. prometheus-operator-kubernetes-resources 2s
  187. prometheus-operator-kubernetes-storage 2s
  188. prometheus-operator-kubernetes-system 2s
  189. prometheus-operator-node.rules 2s
  190. prometheus-operator-prometheus-operator 2s
  191. prometheus-operator-prometheus.rules 2s
  192. NOTES:
  193. The Prometheus Operator has been installed. Check its status by running:
  194. kubectl --namespace kube-prometheus get pods -l "release=prometheus-operator"
  195. Visit https://github.com/coreos/prometheus-operator for instructions on how
  196. to create & configure Alertmanager and Prometheus instances using the Operator.
  197. At this point, prometheus has been deployed to the cluster. Give it a few minutes for all the pods to come up. Let's keep on working to get access to the rest of the consoles offered by the prometheus deployment.
  198. Prometheus Console
  199. ==================
  200. Let's write an ingress configuration to expose the prometheus console. First off we need to list all the service deployed for prometheus.
  201. .. code:: text
  202. $ kubectl get service prometheus-operator-prometheus -o yaml -n kube-prometheus
  203. apiVersion: v1
  204. kind: Service
  205. metadata:
  206. creationTimestamp: "2019-02-09T23:09:55Z"
  207. labels:
  208. app: prometheus-operator-prometheus
  209. chart: prometheus-operator-2.1.6
  210. heritage: Tiller
  211. release: prometheus-operator
  212. name: prometheus-operator-prometheus
  213. namespace: kube-prometheus
  214. resourceVersion: "10996"
  215. selfLink: /api/v1/namespaces/kube-prometheus/services/prometheus-operator-prometheus
  216. uid: d038d6fa-2cbf-11e9-b74f-48ea5bb87c0b
  217. spec:
  218. clusterIP: 10.107.117.120
  219. ports:
  220. - name: web
  221. port: 9090
  222. protocol: TCP
  223. targetPort: web
  224. selector:
  225. app: prometheus
  226. prometheus: prometheus-operator-prometheus
  227. sessionAffinity: None
  228. type: ClusterIP
  229. status:
  230. loadBalancer: {}
  231. As we can see from the service above, its name is ``prometheus-operator-prometheus`` and it's listening on port ``9090``. So let's write the ingress configuration for it.
  232. .. code:: yaml
  233. ---
  234. apiVersion: extensions/v1beta1
  235. kind: Ingress
  236. metadata:
  237. name: prometheus-dashboard
  238. namespace: kube-prometheus
  239. annotations:
  240. nginx.ingress.kubernetes.io/rewrite-target: /
  241. spec:
  242. rules:
  243. - host: prometheus.kube.local
  244. http:
  245. paths:
  246. - path: /
  247. backend:
  248. serviceName: prometheus-operator-prometheus
  249. servicePort: 9090
  250. Save the file as ``kube-prometheus-ingress.yaml`` or some such and deploy.
  251. .. code:: text
  252. $ kubectl apply -f kube-prometheus-ingress.yaml
  253. ingress.extensions/prometheus-dashboard created
  254. And then add the service host to our ``/etc/hosts``.
  255. .. code:: text
  256. 192.168.39.78 prometheus.kube.local
  257. Now you can access http://prometheus.kube.local from your browser.
  258. Grafana Console
  259. ===============
  260. Much like what we did with the prometheus console previously, we need to do the same to the grafana dashboard.
  261. First step, let's check the service.
  262. .. code:: text
  263. $ kubectl get service prometheus-operator-grafana -o yaml -n kube-prometheus
  264. apiVersion: v1
  265. kind: Service
  266. metadata:
  267. creationTimestamp: "2019-02-09T23:09:55Z"
  268. labels:
  269. app: grafana
  270. chart: grafana-1.25.0
  271. heritage: Tiller
  272. release: prometheus-operator
  273. name: prometheus-operator-grafana
  274. namespace: kube-prometheus
  275. resourceVersion: "10973"
  276. selfLink: /api/v1/namespaces/kube-prometheus/services/prometheus-operator-grafana
  277. uid: cffe169b-2cbf-11e9-b74f-48ea5bb87c0b
  278. spec:
  279. clusterIP: 10.107.125.114
  280. ports:
  281. - name: service
  282. port: 80
  283. protocol: TCP
  284. targetPort: 3000
  285. selector:
  286. app: grafana
  287. release: prometheus-operator
  288. sessionAffinity: None
  289. type: ClusterIP
  290. status:
  291. loadBalancer: {}
  292. We get ``prometheus-operator-grafana`` and port ``80``. Next is the ingress configuration.
  293. .. code:: yaml
  294. ---
  295. apiVersion: extensions/v1beta1
  296. kind: Ingress
  297. metadata:
  298. name: prometheus-grafana
  299. namespace: kube-prometheus
  300. annotations:
  301. nginx.ingress.kubernetes.io/rewrite-target: /
  302. spec:
  303. rules:
  304. - host: grafana.kube.local
  305. http:
  306. paths:
  307. - path: /
  308. backend:
  309. serviceName: prometheus-operator-grafana
  310. servicePort: 80
  311. Then we deploy.
  312. .. code:: text
  313. kubectl apply -f kube-grafana-ingress.yaml
  314. ingress.extensions/prometheus-grafana created
  315. And let's not forget ``/etc/hosts``.
  316. .. code:: text
  317. 192.168.39.78 grafana.kube.local
  318. And the grafana dashboard should appear if you visit http://grafana.kube.local.