摘要：離線安裝包三步安裝，簡單到難以置信源碼分析說句實(shí)在話，的代碼寫的真心一般，質(zhì)量不是很高。然后給該租戶綁定角色。

k8s離線安裝包 三步安裝，簡單到難以置信

說句實(shí)在話，kubeadm的代碼寫的真心一般，質(zhì)量不是很高。

幾個(gè)關(guān)鍵點(diǎn)來先說一下kubeadm干的幾個(gè)核心的事：

kubeadm 生成證書在/etc/kubernetes/pki目錄下

kubeadm 生成static pod yaml配置，全部在/etc/kubernetes/manifasts下

kubeadm 生成kubelet配置，kubectl配置等 在/etc/kubernetes下

kubeadm 通過client go去啟動(dòng)dns

代碼入口 cmd/kubeadm/app/cmd/init.go 建議大家去看看cobra

找到Run函數(shù)來分析下主要流程：

如果證書不存在，就創(chuàng)建證書，所以如果我們有自己的證書可以把它放在/etc/kubernetes/pki下即可, 下文細(xì)看如果生成證書

    if res, _ := certsphase.UsingExternalCA(i.cfg); !res {
        if err := certsphase.CreatePKIAssets(i.cfg); err != nil {
            return err
        }

創(chuàng)建kubeconfig文件

        if err := kubeconfigphase.CreateInitKubeConfigFiles(kubeConfigDir, i.cfg); err != nil {
            return err
        }

創(chuàng)建manifest文件，etcd apiserver manager scheduler都在這里創(chuàng)建, 可以看到如果你的配置文件里已經(jīng)寫了etcd的地址了，就不創(chuàng)建了，這我們就可以自己裝etcd集群，而不用默認(rèn)單點(diǎn)的etcd，很有用

controlplanephase.CreateInitStaticPodManifestFiles(manifestDir, i.cfg); 
if len(i.cfg.Etcd.Endpoints) == 0 {
    if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(manifestDir, i.cfg); err != nil {
        return fmt.Errorf("error creating local etcd static pod manifest file: %v", err)
    }
}

等待APIserver和kubelet啟動(dòng)成功，這里就會(huì)遇到我們經(jīng)常遇到的鏡像拉不下來的錯(cuò)誤，其實(shí)有時(shí)kubelet因?yàn)閯e的原因也會(huì)報(bào)這個(gè)錯(cuò)，讓人誤以為是鏡像弄不下來

if err := waitForAPIAndKubelet(waiter); err != nil {
    ctx := map[string]string{
        "Error":                  fmt.Sprintf("%v", err),
        "APIServerImage":         images.GetCoreImage(kubeadmconstants.KubeAPIServer, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage),
        "ControllerManagerImage": images.GetCoreImage(kubeadmconstants.KubeControllerManager, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage),
        "SchedulerImage":         images.GetCoreImage(kubeadmconstants.KubeScheduler, i.cfg.GetControlPlaneImageRepository(), i.cfg.KubernetesVersion, i.cfg.UnifiedControlPlaneImage),
    }

    kubeletFailTempl.Execute(out, ctx)

    return fmt.Errorf("couldn"t initialize a Kubernetes cluster")
}

給master加標(biāo)簽，加污點(diǎn), 所以想要pod調(diào)度到master上可以把污點(diǎn)清除了

if err := markmasterphase.MarkMaster(client, i.cfg.NodeName); err != nil {
    return fmt.Errorf("error marking master: %v", err)
}

生成tocken

if err := nodebootstraptokenphase.UpdateOrCreateToken(client, i.cfg.Token, false, i.cfg.TokenTTL.Duration, kubeadmconstants.DefaultTokenUsages, []string{kubeadmconstants.NodeBootstrapTokenAuthGroup}, tokenDescription); err != nil {
    return fmt.Errorf("error updating or creating token: %v", err)
}

調(diào)用clientgo創(chuàng)建dns和kube-proxy

if err := dnsaddonphase.EnsureDNSAddon(i.cfg, client); err != nil {
    return fmt.Errorf("error ensuring dns addon: %v", err)
}

if err := proxyaddonphase.EnsureProxyAddon(i.cfg, client); err != nil {
    return fmt.Errorf("error ensuring proxy addon: %v", err)
}

筆者批判代碼無腦式的一個(gè)流程到底，要是筆者操刀定抽象成接口 RenderConf Save Run Clean等，DNS kube-porxy以及其它組件去實(shí)現(xiàn)，然后問題就是沒把dns和kubeproxy的配置渲染出來，可能是它們不是static pod的原因, 然后就是join時(shí)的bug下文提到

循環(huán)的調(diào)用了這一坨函數(shù)，我們只需要看其中一兩個(gè)即可，其它的都差不多

certActions := []func(cfg *kubeadmapi.MasterConfiguration) error{
    CreateCACertAndKeyfiles,
    CreateAPIServerCertAndKeyFiles,
    CreateAPIServerKubeletClientCertAndKeyFiles,
    CreateServiceAccountKeyAndPublicKeyFiles,
    CreateFrontProxyCACertAndKeyFiles,
    CreateFrontProxyClientCertAndKeyFiles,
}

根證書生成：

//返回了根證書的公鑰和私鑰
func NewCACertAndKey() (*x509.Certificate, *rsa.PrivateKey, error) {

    caCert, caKey, err := pkiutil.NewCertificateAuthority()
    if err != nil {
        return nil, nil, fmt.Errorf("failure while generating CA certificate and key: %v", err)
    }

    return caCert, caKey, nil
}

k8s.io/client-go/util/cert 這個(gè)庫里面有兩個(gè)函數(shù)，一個(gè)生成key的一個(gè)生成cert的：

key, err := certutil.NewPrivateKey()
config := certutil.Config{
    CommonName: "kubernetes",
}
cert, err := certutil.NewSelfSignedCACert(config, key)

config里面我們也可以填充一些別的證書信息：

type Config struct {
    CommonName   string
    Organization []string
    AltNames     AltNames
    Usages       []x509.ExtKeyUsage
}

私鑰就是封裝了rsa庫里面的函數(shù)：

    "crypto/rsa"
    "crypto/x509"
func NewPrivateKey() (*rsa.PrivateKey, error) {
    return rsa.GenerateKey(cryptorand.Reader, rsaKeySize)
}

自簽證書,所以根證書里只有CommonName信息，Organization相當(dāng)于沒設(shè)置：

func NewSelfSignedCACert(cfg Config, key *rsa.PrivateKey) (*x509.Certificate, error) {
    now := time.Now()
    tmpl := x509.Certificate{
        SerialNumber: new(big.Int).SetInt64(0),
        Subject: pkix.Name{
            CommonName:   cfg.CommonName,
            Organization: cfg.Organization,
        },
        NotBefore:             now.UTC(),
        NotAfter:              now.Add(duration365d * 10).UTC(),
        KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
        BasicConstraintsValid: true,
        IsCA: true,
    }

    certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
    if err != nil {
        return nil, err
    }
    return x509.ParseCertificate(certDERBytes)
}

生成好之后把之寫入文件：

 pkiutil.WriteCertAndKey(pkiDir, baseName, cert, key);
certutil.WriteCert(certificatePath, certutil.EncodeCertPEM(cert))

這里調(diào)用了pem庫進(jìn)行了編碼

encoding/pem

func EncodeCertPEM(cert *x509.Certificate) []byte {
    block := pem.Block{
        Type:  CertificateBlockType,
        Bytes: cert.Raw,
    }
    return pem.EncodeToMemory(&block)
}

然后我們看apiserver的證書生成：

caCert, caKey, err := loadCertificateAuthorithy(cfg.CertificatesDir, kubeadmconstants.CACertAndKeyBaseName)
//從根證書生成apiserver證書
apiCert, apiKey, err := NewAPIServerCertAndKey(cfg, caCert, caKey)

這時(shí)需要關(guān)注AltNames了比較重要，所有需要訪問master的地址域名都得加進(jìn)去，對(duì)應(yīng)配置文件中apiServerCertSANs字段,其它東西與根證書無差別

config := certutil.Config{
    CommonName: kubeadmconstants.APIServerCertCommonName,
    AltNames:   *altNames,
    Usages:     []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
}

可以看到創(chuàng)建了這些文件

return createKubeConfigFiles(
    outDir,
    cfg,
    kubeadmconstants.AdminKubeConfigFileName,
    kubeadmconstants.KubeletKubeConfigFileName,
    kubeadmconstants.ControllerManagerKubeConfigFileName,
    kubeadmconstants.SchedulerKubeConfigFileName,
)

k8s封裝了兩個(gè)渲染配置的函數(shù)：
區(qū)別是你的kubeconfig文件里會(huì)不會(huì)產(chǎn)生token，比如你進(jìn)入dashboard需要一個(gè)token，或者你調(diào)用api需要一個(gè)token那么請(qǐng)生成帶token的配置
生成的conf文件基本一直只是比如ClientName這些東西不同，所以加密后的證書也不同，ClientName會(huì)被加密到證書里，然后k8s取出來當(dāng)用戶使用

所以重點(diǎn)來了，我們做多租戶時(shí)也要這樣去生成。然后給該租戶綁定角色。

return kubeconfigutil.CreateWithToken(
    spec.APIServer,
    "kubernetes",
    spec.ClientName,
    certutil.EncodeCertPEM(spec.CACert),
    spec.TokenAuth.Token,
), nil

return kubeconfigutil.CreateWithCerts(
    spec.APIServer,
    "kubernetes",
    spec.ClientName,
    certutil.EncodeCertPEM(spec.CACert),
    certutil.EncodePrivateKeyPEM(clientKey),
    certutil.EncodeCertPEM(clientCert),
), nil

然后就是填充Config結(jié)構(gòu)體嘍, 最后寫到文件里，略

"k8s.io/client-go/tools/clientcmd/api
return &clientcmdapi.Config{
    Clusters: map[string]*clientcmdapi.Cluster{
        clusterName: {
            Server: serverURL,
            CertificateAuthorityData: caCert,
        },
    },
    Contexts: map[string]*clientcmdapi.Context{
        contextName: {
            Cluster:  clusterName,
            AuthInfo: userName,
        },
    },
    AuthInfos:      map[string]*clientcmdapi.AuthInfo{},
    CurrentContext: contextName,
}

這里返回了apiserver manager scheduler的pod結(jié)構(gòu)體,

specs := GetStaticPodSpecs(cfg, k8sVersion)
staticPodSpecs := map[string]v1.Pod{
    kubeadmconstants.KubeAPIServer: staticpodutil.ComponentPod(v1.Container{
        Name:          kubeadmconstants.KubeAPIServer,
        Image:         images.GetCoreImage(kubeadmconstants.KubeAPIServer, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage),
        Command:       getAPIServerCommand(cfg, k8sVersion),
        VolumeMounts:  staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeAPIServer)),
        LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeAPIServer, int(cfg.API.BindPort), "/healthz", v1.URISchemeHTTPS),
        Resources:     staticpodutil.ComponentResources("250m"),
        Env:           getProxyEnvVars(),
    }, mounts.GetVolumes(kubeadmconstants.KubeAPIServer)),
    kubeadmconstants.KubeControllerManager: staticpodutil.ComponentPod(v1.Container{
        Name:          kubeadmconstants.KubeControllerManager,
        Image:         images.GetCoreImage(kubeadmconstants.KubeControllerManager, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage),
        Command:       getControllerManagerCommand(cfg, k8sVersion),
        VolumeMounts:  staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeControllerManager)),
        LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeControllerManager, 10252, "/healthz", v1.URISchemeHTTP),
        Resources:     staticpodutil.ComponentResources("200m"),
        Env:           getProxyEnvVars(),
    }, mounts.GetVolumes(kubeadmconstants.KubeControllerManager)),
    kubeadmconstants.KubeScheduler: staticpodutil.ComponentPod(v1.Container{
        Name:          kubeadmconstants.KubeScheduler,
        Image:         images.GetCoreImage(kubeadmconstants.KubeScheduler, cfg.GetControlPlaneImageRepository(), cfg.KubernetesVersion, cfg.UnifiedControlPlaneImage),
        Command:       getSchedulerCommand(cfg),
        VolumeMounts:  staticpodutil.VolumeMountMapToSlice(mounts.GetVolumeMounts(kubeadmconstants.KubeScheduler)),
        LivenessProbe: staticpodutil.ComponentProbe(cfg, kubeadmconstants.KubeScheduler, 10251, "/healthz", v1.URISchemeHTTP),
        Resources:     staticpodutil.ComponentResources("100m"),
        Env:           getProxyEnvVars(),
    }, mounts.GetVolumes(kubeadmconstants.KubeScheduler)),
}

//獲取特定版本的鏡像
func GetCoreImage(image, repoPrefix, k8sVersion, overrideImage string) string {
    if overrideImage != "" {
        return overrideImage
    }
    kubernetesImageTag := kubeadmutil.KubernetesVersionToImageTag(k8sVersion)
    etcdImageTag := constants.DefaultEtcdVersion
    etcdImageVersion, err := constants.EtcdSupportedVersion(k8sVersion)
    if err == nil {
        etcdImageTag = etcdImageVersion.String()
    }
    return map[string]string{
        constants.Etcd:                  fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "etcd", runtime.GOARCH, etcdImageTag),
        constants.KubeAPIServer:         fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "kube-apiserver", runtime.GOARCH, kubernetesImageTag),
        constants.KubeControllerManager: fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "kube-controller-manager", runtime.GOARCH, kubernetesImageTag),
        constants.KubeScheduler:         fmt.Sprintf("%s/%s-%s:%s", repoPrefix, "kube-scheduler", runtime.GOARCH, kubernetesImageTag),
    }[image]
}
//然后就把這個(gè)pod寫到文件里了，比較簡單
 staticpodutil.WriteStaticPodToDisk(componentName, manifestDir, spec); 

創(chuàng)建etcd的一樣，不多廢話

這個(gè)錯(cuò)誤非常容易遇到，看到這個(gè)基本就是kubelet沒起來，我們需要檢查：selinux swap 和Cgroup driver是不是一致
setenforce 0 && swapoff -a && systemctl restart kubelet如果不行請(qǐng)保證 kubelet的Cgroup driver與docker一致，docker info|grep Cg

go func(errC chan error, waiter apiclient.Waiter) {
    // This goroutine can only make kubeadm init fail. If this check succeeds, it won"t do anything special
    if err := waiter.WaitForHealthyKubelet(40*time.Second, "http://localhost:10255/healthz"); err != nil {
        errC <- err
    }
}(errorChan, waiter)

go func(errC chan error, waiter apiclient.Waiter) {
    // This goroutine can only make kubeadm init fail. If this check succeeds, it won"t do anything special
    if err := waiter.WaitForHealthyKubelet(60*time.Second, "http://localhost:10255/healthz/syncloop"); err != nil {
        errC <- err
    }
}(errorChan, waiter)

我就是在此發(fā)現(xiàn)coreDNS的

if features.Enabled(cfg.FeatureGates, features.CoreDNS) {
    return coreDNSAddon(cfg, client, k8sVersion)
}
return kubeDNSAddon(cfg, client, k8sVersion)

然后coreDNS的yaml配置模板直接是寫在代碼里的：
/app/phases/addons/dns/manifests.go

    CoreDNSDeployment = `
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      serviceAccountName: coredns
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - key: {{ .MasterTaintKey }}
...

然后渲染模板，最后調(diào)用k8sapi創(chuàng)建,這種創(chuàng)建方式可以學(xué)習(xí)一下，雖然有點(diǎn)拙劣，這地方寫的遠(yuǎn)不如kubectl好

coreDNSConfigMap := &v1.ConfigMap{}
if err := kuberuntime.DecodeInto(legacyscheme.Codecs.UniversalDecoder(), configBytes, coreDNSConfigMap); err != nil {
    return fmt.Errorf("unable to decode CoreDNS configmap %v", err)
}

// Create the ConfigMap for CoreDNS or update it in case it already exists
if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil {
    return err
}

coreDNSClusterRoles := &rbac.ClusterRole{}
if err := kuberuntime.DecodeInto(legacyscheme.Codecs.UniversalDecoder(), []byte(CoreDNSClusterRole), coreDNSClusterRoles); err != nil {
    return fmt.Errorf("unable to decode CoreDNS clusterroles %v", err)
}
...

這里值得一提的是kubeproxy的configmap真應(yīng)該把a(bǔ)piserver地址傳入進(jìn)來，允許自定義，因?yàn)樽龈呖捎脮r(shí)需要指定虛擬ip，得修改，很麻煩
kubeproxy大差不差，不說了,想改的話改： app/phases/addons/proxy/manifests.go

kubeadm join比較簡單，一句話就可以說清楚，獲取cluster info, 創(chuàng)建kubeconfig，怎么創(chuàng)建的kubeinit里面已經(jīng)說了。帶上token讓kubeadm有權(quán)限
可以拉取

return https.RetrieveValidatedClusterInfo(cfg.DiscoveryFile)

cluster info內(nèi)容
type Cluster struct {
    // LocationOfOrigin indicates where this object came from.  It is used for round tripping config post-merge, but never serialized.
    LocationOfOrigin string
    // Server is the address of the kubernetes cluster (https://hostname:port).
    Server string `json:"server"`
    // InsecureSkipTLSVerify skips the validity check for the server"s certificate. This will make your HTTPS connections insecure.
    // +optional
    InsecureSkipTLSVerify bool `json:"insecure-skip-tls-verify,omitempty"`
    // CertificateAuthority is the path to a cert file for the certificate authority.
    // +optional
    CertificateAuthority string `json:"certificate-authority,omitempty"`
    // CertificateAuthorityData contains PEM-encoded certificate authority certificates. Overrides CertificateAuthority
    // +optional
    CertificateAuthorityData []byte `json:"certificate-authority-data,omitempty"`
    // Extensions holds additional information. This is useful for extenders so that reads and writes don"t clobber unknown fields
    // +optional
    Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

return kubeconfigutil.CreateWithToken(
    clusterinfo.Server,
    "kubernetes",
    TokenUser,
    clusterinfo.CertificateAuthorityData,
    cfg.TLSBootstrapToken,
), nil

CreateWithToken上文提到了不再贅述，這樣就能去生成kubelet配置文件了，然后把kubelet啟動(dòng)起來即可

kubeadm join的問題就是渲染配置時(shí)沒有使用命令行傳入的apiserver地址，而用clusterinfo里的地址，這不利于我們做高可用，可能我們傳入一個(gè)虛擬ip，但是配置里還是apiser的地址