Jekyll2024-02-07T20:42:05+00:00https://safeith.com/feed.xmlSAFEITHHow I do it?Hojjatblog@safeith.comFix MacBook Air Auto Wakeup After Suspending In Debian2020-02-15T12:54:00+00:002020-02-15T12:54:00+00:00https://safeith.com/debian/2020/02/15/fix-macbook-air-auto-wakeup-after-suspending-in-debian<p>Recently my wife gave her old MacBook Air to me, and I started to test some of the Linux distributions, and finally, I decided to continue with Debian. Debian works for me as well as macOS, and I can say it is more powerful than macOS as the unofficial OS for this hardware. I just had a small issue with Debian on this Macbook Air device, and it was “auto wakeup after suspending” and I fixed it with creating a Systemd unit</p>
<p>I only need to create this Systemd unit</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>vim /etc/systemd/system/suspend-fix.service</code></pre></figure>
<p>Whit this content</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>Unit]
<span class="nv">Description</span><span class="o">=</span>Fix <span class="k">for </span>the <span class="nb">suspend </span>issue
<span class="o">[</span>Service]
<span class="nv">Type</span><span class="o">=</span>oneshot
<span class="nv">ExecStart</span><span class="o">=</span>/bin/sh <span class="nt">-c</span> <span class="s2">"echo XHC1 > /proc/acpi/wakeup && echo LID0 > /proc/acpi/wakeup"</span>
<span class="o">[</span>Install]
<span class="nv">WantedBy</span><span class="o">=</span>multi-user.target</code></pre></figure>
<p>And finally I enabled it</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>systemctl <span class="nb">enable</span> <span class="nt">--now</span> suspend-fix.service</code></pre></figure>
<hr />
<p>Source: <a href="https://www.reddit.com/r/Fedora/comments/66rwng/here_is_a_solution_to_the_problem_of_macbook_air/">reddit.com</a></p>Hojjatblog@safeith.comRecently my wife gave her old MacBook Air to me, and I started to test some of the Linux distributions, and finally, I decided to continue with Debian. Debian works for me as well as macOS, and I can say it is more powerful than macOS as the unofficial OS for this hardware. I just had a small issue with Debian on this Macbook Air device, and it was “auto wakeup after suspending” and I fixed it with creating a Systemd unitLet’s Encrypt SSL for cPanel hostname2019-12-28T10:38:00+00:002019-12-28T10:38:00+00:00https://safeith.com/ssl/2019/12/28/letsencrypt-ssl-for-cpanel-hostname<p>In this short story, I will share my experience about using the Let’s Encrypt SSL certificate for cPanel hostname. As you know, cPanel provides a free SSL certificate for cPanel hostname as default. But some times it now works for Iranian domain, so you can use the following steps to have a valid SSL certificate for your cPanel services.</p>
<h2 id="install-the-certbot-from-epel">Install the Certbot from EPEL</h2>
<p>Run following command to install <code class="language-plaintext highlighter-rouge">certbot</code> from <code class="language-plaintext highlighter-rouge">epel</code> repo</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">yum <span class="nb">install</span> <span class="nt">--enablerepo</span><span class="o">=</span>epel certbot</code></pre></figure>
<h2 id="create-deploy-hook-script-for-certbot">Create deploy-hook script for Certbot</h2>
<p>Create <code class="language-plaintext highlighter-rouge">hostname-ssl.sh</code> file</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">vim /usr/local/bin/hostname-ssl.sh
<span class="c"># Copy following lines on it</span>
<span class="c">#!/bin/sh</span>
<span class="nb">set</span> <span class="nt">-e</span>
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/privkey.pem /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/cert.pem <span class="o">></span> /var/cpanel/ssl/cpanel/cpanel.pem
/bin/chown cpanel:cpanel /var/cpanel/ssl/cpanel/cpanel.pem
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/privkey.pem <span class="o">></span> /var/cpanel/ssl/exim/exim.key
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/cert.pem <span class="o">></span> /var/cpanel/ssl/exim/exim.crt
/bin/chown mailnull:mail /var/cpanel/ssl/exim/exim.<span class="k">*</span>
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/privkey.pem <span class="o">></span> /var/cpanel/ssl/ftp/ftpd-rsa-key.pem
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/cert.pem <span class="o">></span> /var/cpanel/ssl/ftp/ftpd-rsa.pem
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/privkey.pem /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/cert.pem <span class="o">></span> /var/cpanel/ssl/ftp/pure-ftpd.pem
/bin/chown root:wheel /var/cpanel/ssl/ftp/<span class="k">*</span>
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/privkey.pem <span class="o">></span> /var/cpanel/ssl/dovecot/dovecot.key
/bin/cat /etc/letsencrypt/live/<span class="nv">$HOSTNAME</span>/cert.pem <span class="o">></span> /var/cpanel/ssl/dovecot/dovecot.crt
/bin/chown root:wheel /var/cpanel/ssl/dovecot/dovecot.<span class="k">*</span>
/bin/systemctl restart cpanel.service
/bin/systemctl restart exim.service
/bin/systemctl restart pure-ftpd.service
/bin/systemctl restart dovecot.service</code></pre></figure>
<p>Now make it executable</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">chmod</span> +x /usr/local/bin/hostname-ssl.sh</code></pre></figure>
<h2 id="issue-a-certificate-for-cpanel-hostname">Issue a certificate for cPanel hostname</h2>
<p>With the following command you will be able to issue a Let’s Encrypt valid certificate for cPanel HOSTNAME</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">certbot <span class="nt">--debug</span> certonly <span class="nt">-a</span> webroot <span class="nt">--agree-tos</span> <span class="nt">--webroot-path</span><span class="o">=</span>/usr/local/apache/htdocs <span class="nt">--deploy-hook</span><span class="o">=</span>/usr/local/bin/hostname-ssl.sh <span class="nt">--renew-by-default</span> <span class="nt">-d</span> <span class="nv">$HOSTNAME</span></code></pre></figure>
<h2 id="certificate-renew-cron-job">Certificate renew cron job</h2>
<p>For the certificate, auto-renew add the following cron job</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">00 02 <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> certbot renew</code></pre></figure>
<hr />
<p>Source: <a href="https://forums.cpanel.net/threads/cpanel-ssl-certs-lets-encrypt.522041/">cPanel Forums</a></p>Hojjatblog@safeith.comIn this short story, I will share my experience about using the Let’s Encrypt SSL certificate for cPanel hostname. As you know, cPanel provides a free SSL certificate for cPanel hostname as default. But some times it now works for Iranian domain, so you can use the following steps to have a valid SSL certificate for your cPanel services.Kubernetes cluster from scratch2019-05-25T05:49:07+00:002019-05-25T05:49:07+00:00https://safeith.com/kubernetes/2019/05/25/kubernetes-cluster-from-scratch<p><strong><span style="color: red;">Note: This document is deprecated</span></strong></p>
<p>In this document we will create a kubernetes cluster with Ubuntu 18.04 servers and Hyperkube, It’s a multi master cluster.</p>
<h2 id="requirements">Requirements</h2>
<ol>
<li>Two Ubuntu 18.04 as master</li>
<li>Two Ubuntu 18.4 as worker</li>
<li>Minimum 2 core vCPU per server</li>
<li>Minimum 2 GB RAM per server</li>
</ol>
<h2 id="preconfiguration">PreConfiguration</h2>
<p>Setup DNS resolver for servers in own client</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">nano /etc/hosts
<span class="c"># Add your servers like following lines</span>
10.10.0.2 k8s-controller-1
10.10.0.3 k8s-controller-2
10.10.0.4 k8s-worker-1
10.10.0.5 k8s-worker-1</code></pre></figure>
<p>Setup date/time in all servers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in </span>k8s-controller-1 k8s-controller-2 k8s-worker-1 k8s-worker-2
<span class="k">do
</span>ssh root@<span class="nv">$i</span> <span class="s2">"timedatectl set-local-rtc 0; timedatectl set-timezone UTC"</span>
<span class="k">done</span></code></pre></figure>
<p>Setup DNS resolver for servers in all servers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in </span>k8s-controller-1 k8s-controller-2 k8s-worker-1 k8s-worker-2
<span class="k">do
</span><span class="nb">grep</span> <span class="nt">-w</span> k8s /etc/hosts | ssh root@<span class="nv">$i</span> <span class="s2">"tee -a /etc/hosts"</span>
<span class="k">done</span></code></pre></figure>
<p>Setup bridge netfilter and IP forwarding</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in </span>k8s-controller-1 k8s-controller-2 k8s-worker-1 k8s-worker-2
<span class="k">do
</span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"net.bridge.bridge-nf-call-iptables=1</span><span class="se">\n\</span><span class="s2">
net.bridge.bridge-nf-call-ip6tables=1</span><span class="se">\n\</span><span class="s2">
net.ipv4.ip_forward=1"</span> <span class="se">\</span>
| ssh root@<span class="nv">$i</span> <span class="s2">"tee /etc/sysctl.d/kubernetes.conf && </span><span class="se">\</span><span class="s2">
modprobe br_netfilter && sysctl -p --system"</span>
<span class="k">done</span></code></pre></figure>
<h2 id="create-cerficates">Create cerficates</h2>
<p>Login to first master with ssh</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh root@k8s-controller-1 <span class="nt">-p</span> 22</code></pre></figure>
<p>Create openssl configuration</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CONTROLLER1_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-1 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">CONTROLLER2_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-2 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">SERVICE_IP</span><span class="o">=</span><span class="s2">"10.96.0.1"</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /etc/kubernetes/pki
<span class="nb">cd</span> /etc/kubernetes/pki
<span class="nb">cat</span> <span class="o">></span> openssl.cnf <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[ req ]
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_ca ]
basicConstraints = critical, CA:TRUE
keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign
[ v3_req_server ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
[ v3_req_client ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
[ v3_req_apiserver ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names_cluster
[ v3_req_etcd ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names_etcd
[ alt_names_cluster ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = k8s-controller-1
DNS.6 = k8s-controller-2
# DNS.7 = </span><span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="k">}</span><span class="sh">
IP.1 = </span><span class="k">${</span><span class="nv">CONTROLLER1_IP</span><span class="k">}</span><span class="sh">
IP.2 = </span><span class="k">${</span><span class="nv">CONTROLLER2_IP</span><span class="k">}</span><span class="sh">
IP.3 = </span><span class="k">${</span><span class="nv">SERVICE_IP</span><span class="k">}</span><span class="sh">
# IP.4 = </span><span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_IP</span><span class="k">}</span><span class="sh">
[ alt_names_etcd ]
DNS.1 = k8s-controller-1
DNS.2 = k8s-controller-2
IP.1 = </span><span class="k">${</span><span class="nv">CONTROLLER1_IP</span><span class="k">}</span><span class="sh">
IP.2 = </span><span class="k">${</span><span class="nv">CONTROLLER2_IP</span><span class="k">}</span><span class="sh">
EOF</span></code></pre></figure>
<p>Create kubernetes CA certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> ca.key
<span class="nb">chmod </span>0600 ca.key
openssl req <span class="nt">-x509</span> <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-nodes</span> <span class="nt">-key</span> ca.key <span class="nt">-days</span> 3650 <span class="nt">-out</span> ca.crt <span class="se">\</span>
<span class="nt">-subj</span> <span class="s2">"/CN=kubernetes-ca"</span> <span class="nt">-extensions</span> v3_ca <span class="se">\</span>
<span class="nt">-config</span> ./openssl.cnf</code></pre></figure>
<p>Create kube apiserver certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> kube-apiserver.key
<span class="nb">chmod </span>0600 kube-apiserver.key
openssl req <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-key</span> kube-apiserver.key <span class="nt">-subj</span> <span class="s2">"/CN=kube-apiserver"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> ca.crt <span class="nt">-CAkey</span> ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> kube-apiserver.crt <span class="nt">-days</span> 365 <span class="se">\</span>
<span class="nt">-extensions</span> v3_req_apiserver <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create apiserver kubelet client certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> apiserver-kubelet-client.key
<span class="nb">chmod </span>0600 apiserver-kubelet-client.key
openssl req <span class="nt">-new</span> <span class="nt">-key</span> apiserver-kubelet-client.key <span class="se">\</span>
<span class="nt">-subj</span> <span class="s2">"/CN=kube-apiserver-kubelet-client/O=system:masters"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> ca.crt <span class="nt">-CAkey</span> ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> apiserver-kubelet-client.crt <span class="nt">-days</span> 365 <span class="se">\</span>
<span class="nt">-extensions</span> v3_req_client <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create admin client certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> admin.key
<span class="nb">chmod </span>0600 admin.key
openssl req <span class="nt">-new</span> <span class="nt">-key</span> admin.key <span class="nt">-subj</span> <span class="s2">"/CN=kubernetes-admin/O=system:masters"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> ca.crt <span class="nt">-CAkey</span> ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> admin.crt <span class="nt">-days</span> 365 <span class="nt">-extensions</span> v3_req_client <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create service account key</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> sa.key
openssl ec <span class="nt">-in</span> sa.key <span class="nt">-outform</span> PEM <span class="nt">-pubout</span> <span class="nt">-out</span> sa.pub
<span class="nb">chmod </span>0600 sa.key
openssl req <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-key</span> sa.key <span class="se">\</span>
<span class="nt">-subj</span> <span class="s2">"/CN=system:kube-controller-manager"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> ca.crt <span class="nt">-CAkey</span> ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> sa.crt <span class="nt">-days</span> 365 <span class="nt">-extensions</span> v3_req_client <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create kube-scheduler certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> kube-scheduler.key
<span class="nb">chmod </span>0600 kube-scheduler.key
openssl req <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-key</span> kube-scheduler.key <span class="se">\</span>
<span class="nt">-subj</span> <span class="s2">"/CN=system:kube-scheduler"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> ca.crt <span class="nt">-CAkey</span> ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> kube-scheduler.crt <span class="nt">-days</span> 365 <span class="nt">-extensions</span> v3_req_client <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create front proxy CA certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> front-proxy-ca.key
<span class="nb">chmod </span>0600 front-proxy-ca.key
openssl req <span class="nt">-x509</span> <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-nodes</span> <span class="nt">-key</span> front-proxy-ca.key <span class="nt">-days</span> 3650 <span class="se">\</span>
<span class="nt">-out</span> front-proxy-ca.crt <span class="nt">-subj</span> <span class="s2">"/CN=front-proxy-ca"</span> <span class="se">\</span>
<span class="nt">-extensions</span> v3_ca <span class="se">\</span>
<span class="nt">-config</span> ./openssl.cnf</code></pre></figure>
<p>Create front proxy client certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> front-proxy-client.key
<span class="nb">chmod </span>0600 front-proxy-client.key
openssl req <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-key</span> front-proxy-client.key <span class="se">\</span>
<span class="nt">-subj</span> <span class="s2">"/CN=front-proxy-client"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> front-proxy-ca.crt <span class="se">\</span>
<span class="nt">-CAkey</span> front-proxy-ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> front-proxy-client.crt <span class="nt">-days</span> 365 <span class="se">\</span>
<span class="nt">-extensions</span> v3_req_client <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create kube-proxy certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> kube-proxy.key
<span class="nb">chmod </span>0600 kube-proxy.key
openssl req <span class="nt">-new</span> <span class="nt">-key</span> kube-proxy.key <span class="se">\</span>
<span class="nt">-subj</span> <span class="s2">"/CN=kube-proxy/O=system:node-proxier"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> ca.crt <span class="nt">-CAkey</span> ca.key <span class="nt">-CAcreateserial</span> <span class="se">\</span>
<span class="nt">-out</span> kube-proxy.crt <span class="nt">-days</span> 365 <span class="nt">-extensions</span> v3_req_client <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create etcd CA certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> etcd-ca.key
<span class="nb">chmod </span>0600 etcd-ca.key
openssl req <span class="nt">-x509</span> <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-nodes</span> <span class="nt">-key</span> etcd-ca.key <span class="nt">-days</span> 3650 <span class="se">\</span>
<span class="nt">-out</span> etcd-ca.crt <span class="nt">-subj</span> <span class="s2">"/CN=etcd-ca"</span> <span class="nt">-extensions</span> v3_ca <span class="se">\</span>
<span class="nt">-config</span> ./openssl.cnf</code></pre></figure>
<p>Create etcd certificate</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> etcd.key
<span class="nb">chmod </span>0600 etcd.key
openssl req <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-key</span> etcd.key <span class="nt">-subj</span> <span class="s2">"/CN=etcd"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> etcd-ca.crt <span class="nt">-CAkey</span> etcd-ca.key <span class="se">\</span>
<span class="nt">-CAcreateserial</span> <span class="nt">-out</span> etcd.crt <span class="nt">-days</span> 365 <span class="se">\</span>
<span class="nt">-extensions</span> v3_req_etcd <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>Create etcd peer cert</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">openssl ecparam <span class="nt">-name</span> secp521r1 <span class="nt">-genkey</span> <span class="nt">-noout</span> <span class="nt">-out</span> etcd-peer.key
<span class="nb">chmod </span>0600 etcd-peer.key
openssl req <span class="nt">-new</span> <span class="nt">-sha256</span> <span class="nt">-key</span> etcd-peer.key <span class="nt">-subj</span> <span class="s2">"/CN=etcd-peer"</span> <span class="se">\</span>
| openssl x509 <span class="nt">-req</span> <span class="nt">-sha256</span> <span class="nt">-CA</span> etcd-ca.crt <span class="nt">-CAkey</span> etcd-ca.key <span class="se">\</span>
<span class="nt">-CAcreateserial</span> <span class="nt">-out</span> etcd-peer.crt <span class="nt">-days</span> 365 <span class="se">\</span>
<span class="nt">-extensions</span> v3_req_etcd <span class="se">\</span>
<span class="nt">-extfile</span> ./openssl.cnf</code></pre></figure>
<p>View created certificates</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in</span> <span class="k">*</span>crt
<span class="k">do
</span><span class="nb">echo</span> <span class="nv">$i</span>:
openssl x509 <span class="nt">-subject</span> <span class="nt">-issuer</span> <span class="nt">-noout</span> <span class="nt">-in</span> <span class="nv">$i</span>
<span class="nb">echo
</span><span class="k">done</span></code></pre></figure>
<p>Copy certificates to another controller</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ssh <span class="nt">-o</span> <span class="nv">StrictHostKeyChecking</span><span class="o">=</span>no root@k8s-controller-2 <span class="s2">"mkdir /etc/kubernetes"</span>
scp <span class="nt">-pr</span> <span class="nt">--</span> /etc/kubernetes/pki/ k8s-controller-2:/etc/kubernetes/
<span class="nb">cd</span> ~</code></pre></figure>
<h2 id="install-binaries">Install binaries</h2>
<p>Install these binaries in all controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">TAG</span><span class="o">=</span>v1.10.2
<span class="nv">URL</span><span class="o">=</span>https://storage.googleapis.com/kubernetes-release/release/<span class="nv">$TAG</span>/bin/linux/amd64
curl -# <span class="nt">-L</span> <span class="nt">-o</span> /usr/bin/hyperkube <span class="nv">$URL</span>/hyperkube
<span class="nb">chmod</span> +x /usr/bin/hyperkube
<span class="nb">cd</span> /usr/bin/
hyperkube <span class="nt">--make-symlinks</span>
<span class="nb">chmod</span> +x kube<span class="k">*</span>
<span class="nb">cd</span> ~</code></pre></figure>
<h2 id="generate-kubernetes-configs">Generate kubernetes configs</h2>
<h3 id="generate-kubeconfig-files-on-all-controller-nodes">Generate kubeconfig files on all controller nodes</h3>
<p>Create service account kubeconfig</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CONTROLLER1_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-1 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="o">=</span><span class="nv">$INTERNAL_IP</span>
<span class="nv">CLUSTER_NAME</span><span class="o">=</span><span class="s2">"default"</span>
<span class="nv">KCONFIG</span><span class="o">=</span>controller-manager.kubeconfig
<span class="nv">KUSER</span><span class="o">=</span><span class="s2">"system:kube-controller-manager"</span>
<span class="nv">KCERT</span><span class="o">=</span>sa
<span class="nb">cd</span> /etc/kubernetes/
kubectl config set-cluster <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--certificate-authority</span><span class="o">=</span>pki/ca.crt <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--server</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="k">}</span>:6443 <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-credentials <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--client-certificate</span><span class="o">=</span>pki/<span class="k">${</span><span class="nv">KCERT</span><span class="k">}</span>.crt <span class="se">\</span>
<span class="nt">--client-key</span><span class="o">=</span>pki/<span class="k">${</span><span class="nv">KCERT</span><span class="k">}</span>.key <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--cluster</span><span class="o">=</span><span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--user</span><span class="o">=</span><span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config use-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config view <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span></code></pre></figure>
<p>Create kube-scheduler kubeconfig</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CONTROLLER1_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-1 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="o">=</span><span class="nv">$INTERNAL_IP</span>
<span class="nv">CLUSTER_NAME</span><span class="o">=</span><span class="s2">"default"</span>
<span class="nv">KCONFIG</span><span class="o">=</span>scheduler.kubeconfig
<span class="nv">KUSER</span><span class="o">=</span><span class="s2">"system:kube-scheduler"</span>
<span class="nv">KCERT</span><span class="o">=</span>kube-scheduler
<span class="nb">cd</span> /etc/kubernetes/
kubectl config set-cluster <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--certificate-authority</span><span class="o">=</span>pki/ca.crt <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--server</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="k">}</span>:6443 <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-credentials <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--client-certificate</span><span class="o">=</span>pki/<span class="k">${</span><span class="nv">KCERT</span><span class="k">}</span>.crt <span class="se">\</span>
<span class="nt">--client-key</span><span class="o">=</span>pki/<span class="k">${</span><span class="nv">KCERT</span><span class="k">}</span>.key <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--cluster</span><span class="o">=</span><span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--user</span><span class="o">=</span><span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config use-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config view <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span></code></pre></figure>
<p>Create admin kubeconfig</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CONTROLLER1_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-1 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="o">=</span><span class="nv">$INTERNAL_IP</span>
<span class="nv">CLUSTER_NAME</span><span class="o">=</span><span class="s2">"default"</span>
<span class="nv">KCONFIG</span><span class="o">=</span>admin.kubeconfig
<span class="nv">KUSER</span><span class="o">=</span><span class="s2">"kubernetes-admin"</span>
<span class="nv">KCERT</span><span class="o">=</span>admin
<span class="nb">cd</span> /etc/kubernetes/
kubectl config set-cluster <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--certificate-authority</span><span class="o">=</span>pki/ca.crt <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--server</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="k">}</span>:6443 <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-credentials <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--client-certificate</span><span class="o">=</span>pki/<span class="k">${</span><span class="nv">KCERT</span><span class="k">}</span>.crt <span class="se">\</span>
<span class="nt">--client-key</span><span class="o">=</span>pki/<span class="k">${</span><span class="nv">KCERT</span><span class="k">}</span>.key <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--cluster</span><span class="o">=</span><span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--user</span><span class="o">=</span><span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config use-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config view <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span></code></pre></figure>
<h2 id="deploy-etcd">Deploy etcd</h2>
<blockquote>
<p>Do this in first conroller</p>
</blockquote>
<p>Install etcd binaries</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> ~
<span class="nv">TAG</span><span class="o">=</span>v3.3.4
<span class="nv">URL</span><span class="o">=</span>https://github.com/coreos/etcd/releases/download/<span class="nv">$TAG</span>
curl -# <span class="nt">-LO</span> <span class="nv">$URL</span>/etcd-<span class="nv">$TAG</span><span class="nt">-linux-amd64</span>.tar.gz
<span class="nb">tar </span>xvf etcd-<span class="nv">$TAG</span><span class="nt">-linux-amd64</span>.tar.gz
<span class="nb">chown</span> <span class="nt">-Rh</span> root:root etcd-<span class="nv">$TAG</span><span class="nt">-linux-amd64</span>/
find etcd-<span class="nv">$TAG</span><span class="nt">-linux-amd64</span>/ <span class="nt">-xdev</span> <span class="nt">-type</span> f <span class="nt">-exec</span> <span class="nb">chmod </span>0755 <span class="s1">'{}'</span> <span class="se">\;</span>
<span class="nb">cp </span>etcd-<span class="nv">$TAG</span><span class="nt">-linux-amd64</span>/etcd<span class="k">*</span> /usr/bin/
<span class="nb">mkdir</span> <span class="nt">-p</span> /var/lib/etcd</code></pre></figure>
<p>Create systemd for etcd</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CONTROLLER1_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-1 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">ETCD_CLUSTER_TOKEN</span><span class="o">=</span><span class="s2">"default-27a5f27fe2"</span> <span class="c"># this should be unique per cluster</span>
<span class="nv">ETCD_NAME</span><span class="o">=</span><span class="s1">'k8s-controller-1'</span>
<span class="nv">ETCD_CERT_FILE</span><span class="o">=</span>/etc/kubernetes/pki/etcd.crt
<span class="nv">ETCD_CERT_KEY_FILE</span><span class="o">=</span>/etc/kubernetes/pki/etcd.key
<span class="nv">ETCD_PEER_CERT_FILE</span><span class="o">=</span>/etc/kubernetes/pki/etcd-peer.crt
<span class="nv">ETCD_PEER_KEY_FILE</span><span class="o">=</span>/etc/kubernetes/pki/etcd-peer.key
<span class="nv">ETCD_CA_FILE</span><span class="o">=</span>/etc/kubernetes/pki/etcd-ca.crt
<span class="nv">ETCD_PEER_CA_FILE</span><span class="o">=</span>/etc/kubernetes/pki/etcd-ca.crt
<span class="nb">cat</span> <span class="o">></span> /etc/systemd/system/etcd.service <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[Unit]
Description=etcd
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
Befor=kube-apiserver.service
[Service]
ExecStart=/usr/bin/etcd </span><span class="se">\\</span><span class="sh">
--name </span><span class="k">${</span><span class="nv">ETCD_NAME</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--listen-client-urls https://</span><span class="k">${</span><span class="nv">INTERNAL_IP</span><span class="k">}</span><span class="sh">:2379,http://127.0.0.1:2379 </span><span class="se">\\</span><span class="sh">
--advertise-client-urls https://</span><span class="k">${</span><span class="nv">INTERNAL_IP</span><span class="k">}</span><span class="sh">:2379 </span><span class="se">\\</span><span class="sh">
--data-dir=/var/lib/etcd </span><span class="se">\\</span><span class="sh">
--cert-file=</span><span class="k">${</span><span class="nv">ETCD_CERT_FILE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--key-file=</span><span class="k">${</span><span class="nv">ETCD_CERT_KEY_FILE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--peer-cert-file=</span><span class="k">${</span><span class="nv">ETCD_PEER_CERT_FILE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--peer-key-file=</span><span class="k">${</span><span class="nv">ETCD_PEER_KEY_FILE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--trusted-ca-file=</span><span class="k">${</span><span class="nv">ETCD_CA_FILE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--peer-trusted-ca-file=</span><span class="k">${</span><span class="nv">ETCD_CA_FILE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--peer-client-cert-auth </span><span class="se">\\</span><span class="sh">
--client-cert-auth </span><span class="se">\\</span><span class="sh">
--initial-advertise-peer-urls https://</span><span class="k">${</span><span class="nv">INTERNAL_IP</span><span class="k">}</span><span class="sh">:2380 </span><span class="se">\\</span><span class="sh">
--listen-peer-urls https://</span><span class="k">${</span><span class="nv">INTERNAL_IP</span><span class="k">}</span><span class="sh">:2380 </span><span class="se">\\</span><span class="sh">
--initial-cluster-token </span><span class="k">${</span><span class="nv">ETCD_CLUSTER_TOKEN</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--initial-cluster k8s-controller-1=https://</span><span class="k">${</span><span class="nv">CONTROLLER1_IP</span><span class="k">}</span><span class="sh">:2380 </span><span class="se">\\</span><span class="sh">
--initial-cluster-state new
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
systemctl <span class="nb">enable </span>etcd
systemctl start etcd
systemctl status etcd <span class="nt">-l</span></code></pre></figure>
<p>Verify etcd is working
With etcdctl command</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">etcdctl <span class="se">\</span>
<span class="nt">--ca-file</span><span class="o">=</span>/etc/kubernetes/pki/etcd-ca.crt <span class="se">\</span>
<span class="nt">--cert-file</span><span class="o">=</span>/etc/kubernetes/pki/etcd.crt <span class="se">\</span>
<span class="nt">--key-file</span><span class="o">=</span>/etc/kubernetes/pki/etcd.key <span class="se">\</span>
cluster-health
etcdctl <span class="se">\</span>
<span class="nt">--ca-file</span><span class="o">=</span>/etc/kubernetes/pki/etcd-ca.crt <span class="se">\</span>
<span class="nt">--cert-file</span><span class="o">=</span>/etc/kubernetes/pki/etcd.crt <span class="se">\</span>
<span class="nt">--key-file</span><span class="o">=</span>/etc/kubernetes/pki/etcd.key <span class="se">\</span>
member list</code></pre></figure>
<p>Verify etcd is working with openssl</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"GET /health HTTP/1.1</span><span class="se">\n</span><span class="s2">Host: </span><span class="nv">$INTERNAL_IP</span><span class="se">\n</span><span class="s2">"</span> <span class="se">\</span>
| <span class="nb">timeout </span>2s openssl s_client <span class="nt">-CAfile</span> /etc/kubernetes/pki/etcd-ca.crt <span class="se">\</span>
<span class="nt">-cert</span> /etc/kubernetes/pki/etcd.crt <span class="se">\</span>
<span class="nt">-key</span> /etc/kubernetes/pki/etcd.key <span class="se">\</span>
<span class="nt">-connect</span> <span class="nv">$INTERNAL_IP</span>:2379 <span class="se">\</span>
<span class="nt">-ign_eof</span></code></pre></figure>
<h2 id="create-systemd-for-kubernetes-api-server">Create systemd for Kubernetes API Server</h2>
<p>Do this in all controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CONTROLLER1_IP</span><span class="o">=</span><span class="si">$(</span>getent ahostsv4 k8s-controller-1 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">SERVICE_CLUSTER_IP_RANGE</span><span class="o">=</span><span class="s2">"10.96.0.0/12"</span>
<span class="nb">cat</span> <span class="o">></span> /etc/systemd/system/kube-apiserver.service <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target etcd.service
[Service]
ExecStart=/usr/bin/hyperkube apiserver </span><span class="se">\\</span><span class="sh">
--apiserver-count=2 </span><span class="se">\\</span><span class="sh">
--allow-privileged=true </span><span class="se">\\</span><span class="sh">
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds </span><span class="se">\\</span><span class="sh">
--authorization-mode=RBAC </span><span class="se">\\</span><span class="sh">
--secure-port=6443 </span><span class="se">\\</span><span class="sh">
--bind-address=0.0.0.0 </span><span class="se">\\</span><span class="sh">
--advertise-address=</span><span class="k">${</span><span class="nv">INTERNAL_IP</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--insecure-port=0 </span><span class="se">\\</span><span class="sh">
--insecure-bind-address=127.0.0.1 </span><span class="se">\\</span><span class="sh">
--audit-log-maxage=30 </span><span class="se">\\</span><span class="sh">
--audit-log-maxbackup=3 </span><span class="se">\\</span><span class="sh">
--audit-log-maxsize=100 </span><span class="se">\\</span><span class="sh">
--audit-log-path=/var/log/kube-audit.log </span><span class="se">\\</span><span class="sh">
--client-ca-file=/etc/kubernetes/pki/ca.crt </span><span class="se">\\</span><span class="sh">
--etcd-cafile=/etc/kubernetes/pki/etcd-ca.crt </span><span class="se">\\</span><span class="sh">
--etcd-certfile=/etc/kubernetes/pki/etcd.crt </span><span class="se">\\</span><span class="sh">
--etcd-keyfile=/etc/kubernetes/pki/etcd.key </span><span class="se">\\</span><span class="sh">
--etcd-servers=https://</span><span class="k">${</span><span class="nv">CONTROLLER1_IP</span><span class="k">}</span><span class="sh">:2379 </span><span class="se">\\</span><span class="sh">
--service-account-key-file=/etc/kubernetes/pki/sa.pub </span><span class="se">\\</span><span class="sh">
--service-cluster-ip-range=</span><span class="k">${</span><span class="nv">SERVICE_CLUSTER_IP_RANGE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--service-node-port-range=30000-32767 </span><span class="se">\\</span><span class="sh">
--tls-cert-file=/etc/kubernetes/pki/kube-apiserver.crt </span><span class="se">\\</span><span class="sh">
--tls-private-key-file=/etc/kubernetes/pki/kube-apiserver.key </span><span class="se">\\</span><span class="sh">
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt </span><span class="se">\\</span><span class="sh">
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key </span><span class="se">\\</span><span class="sh">
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname </span><span class="se">\\</span><span class="sh">
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt </span><span class="se">\\</span><span class="sh">
--requestheader-username-headers=X-Remote-User </span><span class="se">\\</span><span class="sh">
--requestheader-group-headers=X-Remote-Group </span><span class="se">\\</span><span class="sh">
--requestheader-allowed-names=front-proxy-client </span><span class="se">\\</span><span class="sh">
--requestheader-extra-headers-prefix=X-Remote-Extra- </span><span class="se">\\</span><span class="sh">
--v=2
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
systemctl <span class="nb">enable </span>kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver <span class="nt">-l</span></code></pre></figure>
<h2 id="create-systemd-for-kubernetes-controller-manager">Create systemd for Kubernetes Controller Manager</h2>
<p>Do this in all controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CLUSTER_CIDR</span><span class="o">=</span><span class="s2">"192.168.0.0/16"</span>
<span class="nv">SERVICE_CLUSTER_IP_RANGE</span><span class="o">=</span><span class="s2">"10.96.0.0/12"</span>
<span class="nv">CLUSTER_NAME</span><span class="o">=</span><span class="s2">"default"</span>
<span class="nb">cat</span> <span class="o">></span> /etc/systemd/system/kube-controller-manager.service <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target kube-apiserver.service
[Service]
ExecStart=/usr/bin/hyperkube controller-manager </span><span class="se">\\</span><span class="sh">
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig </span><span class="se">\\</span><span class="sh">
--address=127.0.0.1 </span><span class="se">\\</span><span class="sh">
--leader-elect=true </span><span class="se">\\</span><span class="sh">
--controllers=*,bootstrapsigner,tokencleaner </span><span class="se">\\</span><span class="sh">
--service-account-private-key-file=/etc/kubernetes/pki/sa.key </span><span class="se">\\</span><span class="sh">
--insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers </span><span class="se">\\</span><span class="sh">
--cluster-cidr=</span><span class="k">${</span><span class="nv">CLUSTER_CIDR</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--allocate-node-cidrs=true </span><span class="se">\\</span><span class="sh">
--cluster-name=</span><span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--service-cluster-ip-range=</span><span class="k">${</span><span class="nv">SERVICE_CLUSTER_IP_RANGE</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt </span><span class="se">\\</span><span class="sh">
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key </span><span class="se">\\</span><span class="sh">
--root-ca-file=/etc/kubernetes/pki/ca.crt </span><span class="se">\\</span><span class="sh">
--use-service-account-credentials=true </span><span class="se">\\</span><span class="sh">
--v=2
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
systemctl <span class="nb">enable </span>kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manager <span class="nt">-l</span></code></pre></figure>
<h2 id="create-systemd-for-kubernetes-scheduler">Create systemd for Kubernetes Scheduler</h2>
<p>Do this in all controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cat</span> <span class="o">></span> /etc/systemd/system/kube-scheduler.service <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target kube-controller-manager.service
[Service]
ExecStart=/usr/bin/hyperkube scheduler </span><span class="se">\\</span><span class="sh">
--leader-elect=true </span><span class="se">\\</span><span class="sh">
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig </span><span class="se">\\</span><span class="sh">
--address=127.0.0.1 </span><span class="se">\\</span><span class="sh">
--v=2
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
systemctl <span class="nb">enable </span>kube-scheduler
systemctl start kube-scheduler
systemctl status kube-scheduler <span class="nt">-l</span></code></pre></figure>
<h2 id="verify-the-controllers">Verify the controllers</h2>
<p>Do this in all controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">KUBECONFIG</span><span class="o">=</span>/etc/kubernetes/admin.kubeconfig
kubectl version
kubectl get componentstatuses</code></pre></figure>
<h2 id="create-kubectl-bash-completion">Create kubectl bash completion</h2>
<p>Do this in all controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">echo</span> <span class="s1">'export KUBECONFIG=/etc/kubernetes/admin.kubeconfig'</span> <span class="o">>></span> .bashrc
<span class="nb">echo</span> <span class="s1">'source <(kubectl completion bash)'</span> <span class="o">>></span> .bashrc
<span class="nb">source</span> .bashrc</code></pre></figure>
<h2 id="generate-bootstrap-token">Generate bootstrap token</h2>
<p>Do this in one of controllers and save TOKEN_PUB, TOKEN_SECRET and BOOTSTRAP_TOKEN in secured places</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">TOKEN_PUB</span><span class="o">=</span><span class="si">$(</span>openssl rand <span class="nt">-hex</span> 3<span class="si">)</span>
<span class="nb">echo</span> <span class="nv">$TOKEN_PUB</span>
<span class="nv">TOKEN_SECRET</span><span class="o">=</span><span class="si">$(</span>openssl rand <span class="nt">-hex</span> 8<span class="si">)</span>
<span class="nb">echo</span> <span class="nv">$TOKEN_SECRET</span>
<span class="nv">BOOTSTRAP_TOKEN</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">TOKEN_PUB</span><span class="k">}</span><span class="s2">.</span><span class="k">${</span><span class="nv">TOKEN_SECRET</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="nv">$BOOTSTRAP_TOKEN</span>
kubectl <span class="nt">-n</span> kube-system create secret generic bootstrap-token-<span class="k">${</span><span class="nv">TOKEN_PUB</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--type</span> <span class="s1">'bootstrap.kubernetes.io/token'</span> <span class="se">\</span>
<span class="nt">--from-literal</span> <span class="nv">description</span><span class="o">=</span><span class="s2">"cluster bootstrap token"</span> <span class="se">\</span>
<span class="nt">--from-literal</span> token-id<span class="o">=</span><span class="k">${</span><span class="nv">TOKEN_PUB</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--from-literal</span> token-secret<span class="o">=</span><span class="k">${</span><span class="nv">TOKEN_SECRET</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--from-literal</span> usage-bootstrap-authentication<span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--from-literal</span> usage-bootstrap-signing<span class="o">=</span><span class="nb">true
</span>kubectl <span class="nt">-n</span> kube-system get secret/bootstrap-token-<span class="k">${</span><span class="nv">TOKEN_PUB</span><span class="k">}</span> <span class="nt">-o</span> yaml</code></pre></figure>
<h2 id="create-bootstrap-kubeconfig">Create bootstrap kubeconfig</h2>
<p>Do this in one of controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="o">=</span><span class="nv">$INTERNAL_IP</span>
<span class="nv">CLUSTER_NAME</span><span class="o">=</span><span class="s2">"default"</span>
<span class="nv">KCONFIG</span><span class="o">=</span><span class="s2">"bootstrap.kubeconfig"</span>
<span class="nv">KUSER</span><span class="o">=</span><span class="s2">"kubelet-bootstrap"</span>
<span class="nb">cd</span> /etc/kubernetes
kubectl config set-cluster <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--certificate-authority</span><span class="o">=</span>pki/ca.crt <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--server</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="k">}</span>:6443 <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--cluster</span><span class="o">=</span><span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--user</span><span class="o">=</span><span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config use-context <span class="k">${</span><span class="nv">KUSER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config view <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span></code></pre></figure>
<h2 id="expose-ca-and-bootstrap-kubeconfig-via-configmap">Expose CA and bootstrap kubeconfig via configmap</h2>
<p>Do this in one of controllers</p>
<blockquote>
<p>Make sure the bootstrap kubeconfig file does not contain the bootstrap token
before you expose it via the cluster-info configmap.</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">kubectl <span class="nt">-n</span> kube-public create configmap cluster-info <span class="se">\</span>
<span class="nt">--from-file</span> /etc/kubernetes/pki/ca.crt <span class="se">\</span>
<span class="nt">--from-file</span> /etc/kubernetes/bootstrap.kubeconfig</code></pre></figure>
<p>Allow anonymous user to acceess the cluster-info configmap.
Do this in one of controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">kubectl <span class="nt">-n</span> kube-public create role system:bootstrap-signer-clusterinfo <span class="se">\</span>
<span class="nt">--verb</span> get <span class="nt">--resource</span> configmaps
kubectl <span class="nt">-n</span> kube-public create rolebinding kubeadm:bootstrap-signer-clusterinfo <span class="se">\</span>
<span class="nt">--role</span> system:bootstrap-signer-clusterinfo <span class="nt">--user</span> system:anonymous</code></pre></figure>
<p>Allow a bootstrapping worker node join the cluster.
Do this in one of controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">kubectl create clusterrolebinding kubeadm:kubelet-bootstrap <span class="se">\</span>
<span class="nt">--clusterrole</span> system:node-bootstrapper <span class="nt">--group</span> system:bootstrappers</code></pre></figure>
<h2 id="install-docker">Install Docker</h2>
<p>Do this in all of servers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">apt <span class="nb">install </span>docker.io</code></pre></figure>
<p>Edit docker systemd file and check ExecStart to</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">/usr/bin/dockerd <span class="nt">--iptables</span><span class="o">=</span><span class="nb">false</span> <span class="nt">--storage-driver</span> overlay <span class="nt">-H</span> fd://</code></pre></figure>
<p>Then restart docker’s systemd</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">systemctl daemon-reload
systemctl restart docker.Service</code></pre></figure>
<h2 id="install-kubernetes-binaries-in-workers">Install Kubernetes binaries in workers</h2>
<p>Do this in all of workers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">TAG</span><span class="o">=</span>v1.10.2
<span class="nv">URL</span><span class="o">=</span>https://storage.googleapis.com/kubernetes-release/release/<span class="nv">$TAG</span>/bin/linux/amd64
curl -# <span class="nt">-L</span> <span class="nt">-o</span> /usr/bin/hyperkube <span class="nv">$URL</span>/hyperkube
<span class="nb">chmod</span> +x /usr/bin/hyperkube
<span class="nb">cd</span> /usr/bin/
hyperkube <span class="nt">--make-symlinks</span>
<span class="nb">chmod</span> +x kube<span class="k">*</span>
<span class="nb">cd</span> ~</code></pre></figure>
<h2 id="retrieve-ca-and-the-bootstrap-kubeconfig">Retrieve CA and the bootstrap kubeconfig</h2>
<p>Do this in all workers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">mkdir</span> <span class="nt">-p</span> /etc/kubernetes/pki
kubectl <span class="nt">-n</span> kube-public get cm/cluster-info <span class="se">\</span>
<span class="nt">--server</span> https://k8s-controller-1:6443 <span class="nt">--insecure-skip-tls-verify</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--username</span><span class="o">=</span>system:anonymous <span class="nt">--output</span><span class="o">=</span><span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.data.ca\.crt}'</span> <span class="se">\</span>
| <span class="nb">tee</span> /etc/kubernetes/pki/ca.crt
kubectl <span class="nt">-n</span> kube-public get cm/cluster-info <span class="se">\</span>
<span class="nt">--server</span> https://k8s-controller-1:6443 <span class="nt">--insecure-skip-tls-verify</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--username</span><span class="o">=</span>system:anonymous <span class="se">\</span>
<span class="nt">--output</span><span class="o">=</span><span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.data.bootstrap\.kubeconfig}'</span> <span class="se">\</span>
| <span class="nb">tee</span> /etc/kubernetes/bootstrap.kubeconfig</code></pre></figure>
<p>Now write previously generated BOOTSTRAP_TOKEN to the bootstrap kubeconfig</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">read</span> <span class="nt">-r</span> <span class="nt">-s</span> <span class="nt">-p</span> <span class="s2">"BOOTSTRAP_TOKEN: "</span> BOOTSTRAP_TOKEN
kubectl config set-credentials kubelet-bootstrap <span class="se">\</span>
<span class="nt">--token</span><span class="o">=</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span>/etc/kubernetes/bootstrap.kubeconfig</code></pre></figure>
<h2 id="install-cni-plugins">Install CNI plugins</h2>
<p>Do this in all servers</p>
<blockquote>
<p>Need to find latest version</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">mkdir</span> <span class="nt">-p</span> /etc/cni/net.d /opt/cni
<span class="nv">ARCH</span><span class="o">=</span>amd64
<span class="nv">CNI_RELEASE</span><span class="o">=</span>0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff
<span class="nv">URL</span><span class="o">=</span>https://storage.googleapis.com/kubernetes-release/network-plugins
curl <span class="nt">-sSL</span> <span class="nv">$URL</span>/cni-<span class="k">${</span><span class="nv">ARCH</span><span class="k">}</span>-<span class="k">${</span><span class="nv">CNI_RELEASE</span><span class="k">}</span>.tar.gz | <span class="nb">tar</span> <span class="nt">-xz</span> <span class="nt">-C</span> /opt/cni</code></pre></figure>
<h2 id="create-systemd-for-kubernetes-kubelet">Create systemd for Kubernetes Kubelet</h2>
<p>Do this all of server</p>
<blockquote>
<p>For master nodes do</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cp</span> <span class="nt">-rv</span> /etc/kubernetes/admin.kubeconfig /etc/kubernetes/kubelet.conf</code></pre></figure>
<blockquote>
<p>For worker nodes do</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in </span>k8s-worker-1 k8s-worker-2<span class="p">;</span> <span class="k">do
</span>scp <span class="nt">-p</span> <span class="nt">--</span> /etc/kubernetes/admin.kubeconfig <span class="nv">$i</span>:/etc/kubernetes/kubelet.conf
<span class="k">done</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">CLUSTER_DNS_IP</span><span class="o">=</span>10.96.0.10
<span class="nb">mkdir</span> <span class="nt">-p</span> /etc/kubernetes/manifests
<span class="nb">cat</span> <span class="o">></span> /etc/systemd/system/kubelet.service <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service kube-scheduler.service
Requires=docker.service
[Service]
ExecStart=/usr/bin/hyperkube kubelet </span><span class="se">\\</span><span class="sh">
--experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig </span><span class="se">\\</span><span class="sh">
--kubeconfig=/etc/kubernetes/kubelet.conf </span><span class="se">\\</span><span class="sh">
--pod-manifest-path=/etc/kubernetes/manifests </span><span class="se">\\</span><span class="sh">
--allow-privileged=true </span><span class="se">\\</span><span class="sh">
--network-plugin=cni </span><span class="se">\\</span><span class="sh">
--cni-conf-dir=/etc/cni/net.d </span><span class="se">\\</span><span class="sh">
--cni-bin-dir=/opt/cni/bin </span><span class="se">\\</span><span class="sh">
--cluster-dns=</span><span class="k">${</span><span class="nv">CLUSTER_DNS_IP</span><span class="k">}</span><span class="sh"> </span><span class="se">\\</span><span class="sh">
--cluster-domain=cluster.local </span><span class="se">\\</span><span class="sh">
--authorization-mode=Webhook </span><span class="se">\\</span><span class="sh">
--client-ca-file=/etc/kubernetes/pki/ca.crt </span><span class="se">\\</span><span class="sh">
--cgroup-driver=cgroupfs </span><span class="se">\\</span><span class="sh">
--cert-dir=/etc/kubernetes
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
systemctl <span class="nb">enable </span>kubelet
systemctl start kubelet
systemctl status kubelet <span class="nt">-l</span></code></pre></figure>
<blockquote>
<p>Make controller nodes unschedulable by any pods</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in </span>k8s-controller-1 k8s-controller-2
<span class="k">do
</span>kubectl label node <span class="nv">$i</span> node-role.kubernetes.io/master<span class="o">=</span>
kubectl taint nodes <span class="nv">$i</span> node-role.kubernetes.io/master<span class="o">=</span>:NoSchedule
<span class="k">done</span></code></pre></figure>
<h2 id="install-kube-proxy">Install kube-proxy</h2>
<p>Create a kube-proxy service account in one of controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">kubectl <span class="nt">-n</span> kube-system create serviceaccount kube-proxy</code></pre></figure>
<p>Create a kube-proxy kubeconfig</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">INTERNAL_IP</span><span class="o">=</span><span class="si">$(</span><span class="nb">hostname</span> <span class="nt">-I</span> | <span class="nb">awk</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
<span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="o">=</span><span class="nv">$INTERNAL_IP</span>
<span class="nb">export </span><span class="nv">KUBECONFIG</span><span class="o">=</span>/etc/kubernetes/admin.kubeconfig
<span class="nv">SECRET</span><span class="o">=</span><span class="si">$(</span>kubectl <span class="nt">-n</span> kube-system get sa/kube-proxy <span class="se">\</span>
<span class="nt">--output</span><span class="o">=</span><span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.secrets[0].name}'</span><span class="si">)</span>
<span class="nv">JWT_TOKEN</span><span class="o">=</span><span class="si">$(</span>kubectl <span class="nt">-n</span> kube-system get secret/<span class="nv">$SECRET</span> <span class="se">\</span>
<span class="nt">--output</span><span class="o">=</span><span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.data.token}'</span> | <span class="nb">base64</span> <span class="nt">-d</span><span class="si">)</span>
<span class="nv">CLUSTER_NAME</span><span class="o">=</span><span class="s2">"default"</span>
<span class="nv">KCONFIG</span><span class="o">=</span><span class="s2">"kube-proxy.kubeconfig"</span>
<span class="nb">cd</span> /etc/kubernetes
kubectl config set-cluster <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--certificate-authority</span><span class="o">=</span>pki/ca.crt <span class="se">\</span>
<span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--server</span><span class="o">=</span>https://<span class="k">${</span><span class="nv">KUBERNETES_PUBLIC_ADDRESS</span><span class="k">}</span>:6443 <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-context <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--cluster</span><span class="o">=</span><span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--user</span><span class="o">=</span>default <span class="se">\</span>
<span class="nt">--namespace</span><span class="o">=</span>default <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config set-credentials <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--token</span><span class="o">=</span><span class="k">${</span><span class="nv">JWT_TOKEN</span><span class="k">}</span> <span class="se">\</span>
<span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config use-context <span class="k">${</span><span class="nv">CLUSTER_NAME</span><span class="k">}</span> <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span>
kubectl config view <span class="nt">--kubeconfig</span><span class="o">=</span><span class="k">${</span><span class="nv">KCONFIG</span><span class="k">}</span></code></pre></figure>
<p>Bind a kube-proxy service account (from kube-system namespace) to a clusterrole system:node-proxier to allow RBAC</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">kubectl create clusterrolebinding kubeadm:node-proxier <span class="se">\</span>
<span class="nt">--clusterrole</span> system:node-proxier <span class="se">\</span>
<span class="nt">--serviceaccount</span> kube-system:kube-proxy</code></pre></figure>
<p>Copy kube-proxy.kubeconfig to workers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">for </span>i <span class="k">in </span>k8s-worker-1 k8s-worker-2
<span class="k">do
</span>scp <span class="nt">-p</span> <span class="nt">--</span> /etc/kubernetes/kube-proxy.kubeconfig <span class="nv">$i</span>:/etc/kubernetes/
<span class="k">done</span></code></pre></figure>
<p>Create systemd for kube-proxy</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">mkdir</span> /var/lib/kube-proxy
<span class="nb">cat</span> <span class="o">></span> /etc/systemd/system/kube-proxy.service <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target kubelet.service
[Service]
ExecStart=/usr/bin/hyperkube proxy </span><span class="se">\\</span><span class="sh">
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig </span><span class="se">\\</span><span class="sh">
--v=2
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
systemctl <span class="nb">enable </span>kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy <span class="nt">-l</span></code></pre></figure>
<h2 id="deploy-calico">Deploy Calico</h2>
<p>Do this in one of controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">kubectl apply <span class="nt">-f</span> <span class="se">\</span>
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply <span class="nt">-f</span> <span class="se">\</span>
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml</code></pre></figure>
<h2 id="deploy-kube-dns">Deploy kube-DNS</h2>
<p>Do This in one of controllers</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> /etc/kubernetes/manifests/
wget https://raw.githubusercontent.com/kelseyhightower/kubernetes-the-hard-way/master/deployments/kube-dns.yaml
<span class="nb">sed</span> <span class="nt">-i</span> <span class="s1">'s/clusterIP: 10.32.0.10/clusterIP: 10.96.0.10/g'</span> kube-dns.yaml
kubectl create <span class="nt">-f</span> kube-dns.yaml</code></pre></figure>
<blockquote>
<p>YOUR WELCOME :-)</p>
</blockquote>Hojjatblog@safeith.comNote: This document is deprecatedSSH over Tor hidden service2019-03-15T17:49:32+00:002019-03-15T17:49:32+00:00https://safeith.com/security/2019/03/15/ssh-over-tor-hidden-service<p>Sometimes you need to access your device with ssh connection, but you don’t have a static public IP on your machine, or you’re using a shared internet connection and want to give ssh access to somebody to connect to your device. What can you do? Maybe you know some ways, but I prefer to use <code class="language-plaintext highlighter-rouge">Tor hidden</code> service. In this small article, I will write how I do it on Archlinux.</p>
<p>In the first step, I install <code class="language-plaintext highlighter-rouge">ssh</code>, <code class="language-plaintext highlighter-rouge">tor</code>, and <code class="language-plaintext highlighter-rouge">torsocks</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>pacman <span class="nt">-S</span> ssh tor torsocks</code></pre></figure>
<p>In the second step, I configure tor to enable hidden service for ssh</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo echo</span> <span class="s1">'HiddenServiceDir /var/lib/tor/ssh_hidden_service/'</span> <span class="o">>></span> /etc/tor/torrc
<span class="nb">sudo echo</span> <span class="s1">'HiddenServicePort 22 127.0.0.1:22'</span> <span class="o">>></span> /etc/tor/torrc</code></pre></figure>
<p>In the third step, I start ssh and tor service</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>systemctl start sshd.service tor.service</code></pre></figure>
<p>In the fourth step, I check hostname of hidden service</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo cat</span> /var/lib/tor/ssh_hidden_service/hostname</code></pre></figure>
<p>In the fifth step, I check if it works ok</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">torify ssh hidden_hostname</code></pre></figure>Hojjatblog@safeith.comSometimes you need to access your device with ssh connection, but you don’t have a static public IP on your machine, or you’re using a shared internet connection and want to give ssh access to somebody to connect to your device. What can you do? Maybe you know some ways, but I prefer to use Tor hidden service. In this small article, I will write how I do it on Archlinux.Why?2019-03-15T10:55:00+00:002019-03-15T10:55:00+00:00https://safeith.com/public/2019/03/15/why<p><strong>Why I started this blog?</strong><br />
I started this blog because I want to write about new things I learn those and I’m going to keep those for others and myself.</p>
<p><strong>Why I use Jekyll?</strong><br />
Because it’s easy to manage, and another side it uses markdown, and I like markdown.</p>
<p><strong>Why I use GitHub?</strong><br />
Because it lets me track my changes quickly, and It has GitHub pages that allow me to serve my Jekyll over it.</p>Hojjatblog@safeith.comWhy I started this blog? I started this blog because I want to write about new things I learn those and I’m going to keep those for others and myself.