Table of Contents

;-- mode: Org; coding: utf-8; fill-column: 100 --

1. Docker

1.1. terms

Image -docker run> Running Container -> Stopped Container -docker commit> New Image

parent image - image that your image is based on (FROM)

Состоит:

  • Docker daemon- manages Docker containers and handles container objects - Docker client program - provides a command-line interface
  • Container c т.з. ч. - стандартизированный метод упаковки приложений, его конфигурациии и всех зависимостей в единый объект.
  • Container c т.з. ОС - процесс извлеченный из tarball привязанный к namespaces и контролируемый через cgroups
  • Container - состоит из слоев - слепков состояний файловой системы из image + read-write layer on top. Images become containers when they run on Docker Engine. при перезапуске возвращается к image
    • UUID short and long
    • name - by user or random
  • Container Runtime - software that can run containers on a host operating system or each node in the cluster so that Pods can run there.
  • Image - read-only template used to build containers - to store and ship applications - series of read-only layers
  • Docker service - allows containers to be scaled across multiple Docker daemons - result is "swarm"
  • Docker registr - repository for Docker images - download ("pull") - upload ("push"). Registries can be public or private
  • Docker Hub - default registry where Docker looks for images https://hub.docker.com/
  • volumes -
  • Dockerfile - Docker can build images automatically by reading the instructions from a Dockerfile by "$docker build ."
  • task - container running in a service
  • replicas - containers or task in one service defined in docker-compose.yml

1.2. theory

1.2.1. network

Docker’s networking subsystem is pluggable, using drivers.

  • bridge: The default network driver.
  • host
  • overlay
  • ipvlan
  • macvlan
  • none
  • network plugins

all networs - user defined bridges

1.2.2. security

Attack surface of a container image is measured by the number of files in it, or how many megabytes of space it uses on disk.

attack surface:

  • quality of software and configuration(aka files
  • software in the direct execution path which is used in many different container images (C

libraries, web servers, encryption libraries, etc)

  • Standardizing on the exact same versions (Linux distribution and version) of this widely used software in the direct execution path (C libraries, web servers, encryption libraries, etc) reduces attack surface

1.2.3. distroless

a Linux distro is made up of two main components: a kernel, and a user space.

Even in a distroless set of container images, things like the Java virtual machine, Python, and Node.js are compiled against a C library which gives these user space programs access to low level functions in the Linux kernel (network sockets, storage volumes, files, etc).

Linux distros exist because we just want to write applications, we don’t want to patch and maintain widely used libraries and infrastructure like timezone data, C libraries, etc.

FROM scratch - official container https://docs.docker.com/build/building/base-images/

  • itself it is empty (in that it doesn’t contain any folders or files)
  • can conain executable that will be like a process on host machine: statically compiled and self-contained.
  • You also wouldn’t be able to login to the container either as there isn’t a shell unless you explicitly add one.

1.3. usage

explore:

  • docker images - list images and layers
    • docker images | grep superset | awk '{print $1}' | xargs docker image rm
  • docker ps - containers currently running
    • docker ps -a - was ran
  • docker logs image
  • docker pull busybox - download image
  • docker build -t fedora/jboss:1.0 . # –tag to give a name to the image
    • docker build image –build-args HTTPS_PROXY=172.0.0.1:8888
    • docker build -t aa -f Dockerfile.app .
  • docker run -it debian_my bash - create container based on image and attach to tty
  • docker exec -it debian_my bash - connect to running
    • docker exec -it debian_my bash -c "command ; command"
    • docker exec -w /workspace -it ps bash
  • docker rename ngin nginx
  • docker start/stop/kill/restart/rm nginx
  • docker events/top/stats/diff nginx
  • docker -it superset_db pg_dump -U superset superset > backups.sql - бэкап БД из контейнера в папку

networks

  • docker network create my-net
  • docker create –name my-nginx –network my-net –publish 8080:80 nginx:latest
  • docker networks ls # show networs
  • docker network inspect NAME
  • docker network create -d bridge MyBridgeNetwork
  • docker network rm
  • docker connect MyOverlayNerwork nginx - подключение работающего контейнера к сети
  • docker run -it -d –network=MyOverlayNetwork ngixn - подключение к сети при его запуске
  • docker network disconnect MyOverlayNerwork nginx

1.4. use cases:

1.4.1. containers

  • docker build -t container_name .
  • docker run -d CONTAINER_ID_REPOSITORY # execute and deattach
    • -p, –publish ip:[hostPort]:containerPort | [hostPort:]containerPort Publish a container's port, or range of ports, to the host.
      • docker run -p 80:5000 -d web_alone # 80 - host port, 5000 - container port
  • docker stop container
    • docker ps -q | xargs docker stop # stop all containers
  • docker pull [IMAGE] # from repository
  • docker import [URL/FILE] # Create an image from a tarball:
  • docker rm $(docker ps -a -q) - удалить все контейнеры
  • docker commit NAME REPOSITORY
  • docker logs CONTAINER_ID_REPOSITORY
  • docker inspect logs CONTAINER_ID_R | grep -E "LogPath" - где лог
  • docker logs containername >& ~/a
    • nano ~/a
  • docker exec -u root -it 1db047ccc674 bash - подключиться к уже запущенному

1.4.2. Exploring Docker container's file system

  • https://stackoverflow.com/questions/20813486/exploring-docker-containers-file-system
  • docker commit CONTAINER ID mysnapshot
  • docker run -it mysnapshot /bin/sh - запускает процесс в контейнере
  • docker exec -u root -it 1db047ccc674 bash - подключиться к уже запущенному
    • i - stdin
    • t - virtual tty
    • 1db047ccc674 - running container
  • export http_proxy='http://Pakhomov_KA@192.168.2.252:8080'
  • export https_proxy='http://Pakhomov_KA@192.168.2.252:8080'
  • apt-get update
  • apt-get install emacs-nox

1.4.3. Block internet

  1. 1

    service: one: networks:

    • internal
    • external

    two: networks:

    • internal

    networks: internal: internal: true external:

  2. 2

    COPY entrypoint.sh /entrypoint.sh

    ENTRYPOINT ["/entrypoint.sh", "–"]

    ACCEPT_CIDR=${ALLOWED_CIDR:-192.168.0.0/24}

    iptables -A INPUT -s $ACCEPT_CIDR -j ACCEPT iptables -A INPUT -j DROP iptables -A OUTPUT -d $ACCEPT_CIDR -j ACCEPT iptables -A OUTPUT -j DROP

    sudo -u app sh -c "$@"

1.4.4. Block access to LAN and out:

  • docker network create -o "com.docker.network.bridge.enable_ip_masquerade"="false" lan-restricted
  • Blocks
    • Local LAN
    • Internet
  • Does not block
    • Host running docker daemon (example access to 10.0.1.10:22)

1.4.5. Block access to other containers:

  • docker network create -o "com.docker.network.bridge.enable_icc"="false" icc-restricted
  • Blocks
    • Containers accessing other containers on the same icc-restricted network.
  • Does not block
    • Access to host running docker daemon
    • Local LAN
    • Internet

1.4.6. Block any external access:

networks: yournetwork: internal: true

1.4.7. backup

  • volumes: tar -cvzf a.tar.gz /var/lib/docker/volumes/<volume_name>
  • docker save -o ubuntu.tar ubuntu:lucid
  • docker save myimage:latest | gzip > myimage_latest.tar.gz

1.4.8. volume sizes

#!/bin/bash

for v in $(docker volume ls | grep -v DRIVER | awk '{print $2}') ; do
   p=$(docker volume inspect $v | jq '.[].Mountpoint' | tr -d '"')
   du -sh $p

1.4.9. monitor resource usage

  • docker stats –all –no-stream –no-trunc # memory, cpu
  • docker system df -v
  • docker status container_ID #to check single container resources

1.5. admin usage

Комманды

  • docker ps -a - все контейнеры
  • docker image ls - все images
  • docker load -i file.tar - Load saved image
  • docker pull debian:testing
  • docker inspect [CONTAINER ID] – "Config": { "Entrypoint": ["/pause"],
  • docker pull storm:TAG – download image
  • docker ps –no-trunc – full command
  • docker system prune –volumes - clean unused everything

docker run [OPTIONS] IMAGE [COMMAND] [ARG…]

Commit a container’s file changes or settings into a new image: docker commit CONTAINER ID mysnapshot

Build image and run:

  • docker build –tag=friendlyhello .
  • docker run -p 4000:80 friendlyhello -d - execute CMD in Dockerfile, background

1.6. CMD and ENTRYPOINT

docker run - require CMD or ENTRYPOINT specified in Dokerfile

both CMD and ENTRYPOINT have exec and shell form

  • ENTRYPOINT ["executable", "param1", "param2"] - exec form - recommended
  • ENTRYPOINT command param1 param2 - shell form - Ctrl + C will not stop ex. ping google.com

CMD can be used to set additional defaults that are more likely to be changed

  • ENTRYPOINT ["top", "-b"]
  • CMD ["-c"]

1.7. ARG and ENV

  • ARG is only available during the build of a Docker imag
  • ENV values are available to containers, but also RUN-style commands during the Docker build starting with the line where they are introduced. If you set an environment variable in an intermediate container using bash (RUN export VARI=5 && …) it will not persist in the next command.

docker build . –build-arg -t foo

FROM debian:bullseye-slim ARG WTF ENV WTF=$WTF CMD echo $WTF

1.8. links

1.9. install

https://docs-stage.docker.com/engine/install/ubuntu/ from deb https://download.docker.com/linux/ubuntu/dists/focal/stable/binary-amd64/Packages

from Dcoker Repository https://docs-stage.docker.com/engine/install/ubuntu/

Installing Docker CE using Docker Installation Script

Проверка:

  • docker run hello-world

Запускать без sudo https://docs-stage.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

1.9.1. Docker CE

https://kifarunix.com/install-docker-ce-on-ubuntu-20-04/

  • Docker CE (Community Edition) is the open-source, community supported version of Docker and is available for free.
  • Docker EE (Enterprise Edition) is a commercial/premium version of Docker CE and is support by Docker Inc.
    • docker or docker-engine or docker-io
    • $ sudo apt-get remove docker docker-engine docker.io containerd runc

1.10. proxy

  • ~/.docker/config.json
  • 192.168.2.252 это srv-proxy

{ "proxies": { "default": { "httpProxy": "http://192.168.2.252:8080", "httpsProxy": "http://192.168.2.252:8080" } } }

1.11. storage

by default a writable container layer.

  • volumes - on Docker var/lib/docker/volumes - best way - cannot be modified externaly - When you mount the volume into a container, this directory is what is mounted into the container
  • bind mounts - can be modified externaly

kebenrenets - The Docker image is at the root of the filesystem hierarchy, and any volumes are mounted at the specified paths within the image.

1.12. images

Хранятся в var/lib/docker

  • REPOSITORY - откуда был скачен или имя, <none> - локально

image is built up from a series of layers. Each layer except the very last one is read-only.

commands:

  • docker image ls - все images
  • docker rmi $(docker images -q) - удалить все images
  • docker run PARAMETERS repository_or_name - запускает image и делает его контенером, если его нет, то скачивает из публичного репозитория
    • -d - deattached in background
    • -p <host_port>:<container_port>

1.13. containers

Container has: read only layers: 1 writable layer - сохраняется только если завершается нормально, в случае сбоя теряется.

1.16. multi-stage builds

builder pattern - one Dockerfile to use for development, and a slimmed-down one to use for production, which only contained your application and exactly what was needed to run it.

  • FROM - starts new stage
  • FROM golang:1.16 AS builder; COPY –from=builder /go/a ./a # if stages will be reoreded
  • COPY –from=nginx:latest /etc/nginx/nginx.conf /nginx.conf # copy from separate image

1.17. clear

nnterms:

  • dangling - not associated with container

remove parts not connected to running contatinters:

  • docker system/image/container/network prune -a

remove images:

  • docker image prune # dangling
  • docker image prune -a # remove all images which are not used by containers
  • docker image prune -a –filter "until=24h" # removes older than 24 hours
  • docker rmi $(docker images -f "dangling=true" -q)
  • remove images by pattern
    • docker images -a | grep “some_pattern” | awk ‘{print $3}’ | xargs docker rmi

remove containers:

  • docker container prune # stopped containers
  • docker rm $(docker ps -q -f status=exited) # remove all the containers that are not running (exited).
  • docker rm $(docker ps -a -q) # wipe all containers
  • remove container by pattern
    • docker ps -a | grep “some_pattern” | awk ‘{print $3}’ | xargs docker rm

remove volumes (Volumes are never removed automatically):

  • docker volume prune # remove all volumes not used by at least one container
  • docker volume prune –filter "label!=keep" # only removes volumes which are not labelled with the keep label
  • docker volume ls -f name=syst
  • docker volume rm volume1_name volume2_name

remove networks:

  • docker network prune # networks which aren’t used by any containers

Prune everything:

  • docker system prune # shortcut that prunes images, containers, and networks
  • docker system prune –volumes # and volumes too

docker builder prune - удаление кэша сборки

1.18. tmpfs

  • sudo mount -t tmpfs tmpfs /var/lib/docker
  • systemctl restart docker

1.19. Dokerfile

  • COPY local.conf . # to current directory # preffered # stripped-down version of ADD
  • ADD . /code # Too Much Magic
    • ADD foo.tar.gz /tmp # autounpack
  • WORKDIR /code
  • ENV FLASK_APP app.py # set env variable inside of running container
  • RUN pip3 install –upgrade pip # run command to build image

1.20. apt Dokerfile

RUN apt-get update && apt-get install -y --no-install-recommends \
    cuda-libraries-11-8=${NV_CUDA_LIB_VERSION} \
    && rm -rf /var/lib/apt/lists/*

1.21. health check for container

can be baked into the image definition

  • HEALTHCHECK –interval=5s –timeout=3s CMD curl –fail http://localhost:8091/pools || exit 1
    • –interval=DURATION (default 30s)
    • –timeout=DURATION (default 30s)
    • –retries=N (default 3)

If health check is enabled, then the container can have three states:

  • starting
  • healthy
  • unhealthy

docker-compose.yml:

services:
        webapp:
                image: your-repo-name/webapp:1.0.1-nginx
                container_name: webapp
                network_mode: host
                healthcheck:
                        test: mysql airflowdb -u $MYSQL_USER --password=$MYSQL_PASSWORD -e 'SELECT 1;'
                        interval: 20s
                        timeout: 60s
                        start_period: 15s
                restart: always

2. docker-compose

multi-container Docker applications - for scaling - docker-compose

  • https://docs.docker.com/get-started/part3/
  • Docker Compouse - run commands on multiple containers at once
  • Docker Swarm - provides native clustering functionality for Docker containers. require docker-machine tool and VM

docker-compose.yml - несоклько инстанцев - балансировщик нагрузки работает внутри, но не снаружи.

service
conteiner in swarm - what ports it should use, how many replicas of the container should run

build 5 images-services from Docker Hub image "redis:alpine" and directories:

  • docker-compose build && docker-compose up -d
  • docker-compose up –force-recreate –build -d

2.1. TODO multi-stage builds

2.2. swarm

  • docker swarm init - to enable swarm mode and make your current machine a swarm manage
  • docker-machine - инструмент который работает на базе виртуальных машин. ХЗ как что
  • docekr swarm leave

2.3. docker-compose.yml

structure:

  1. docker-compose.yaml
version: "3.9"  # optional since v1.27.0
services:
  web:
    build: path_to_Dockerfile
    ports:
      - "8000:5000" # host port : container port
  1. Dockerfile

2.4. Environment variables

default environment variable file named .env (or –env-file command line option):

  • TAG=v1.5

var

  • ${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment.
  • ${VARIABLE?err} exits with an error message containing err if VARIABLE is unset in the environment.
web:
  environment:
    - DEBUG=1

2.5. Network

network per application

  • network name based on the “project name”, which is based on the name of the directory it lives in (You can override the project name with either the –project-name flag or the COMPOSE_PROJECT_NAME environment variable.)
    • myapp_default
  • container join the network under own name
  • standalone containers can connect default network (attachable=true)

ports: ["8001:5432"] - distinction between HOST_PORT and CONTAINER_PORT

  • service-to-service communication uses the CONTAINER_PORT
  • HOST_PORT is defined, the service is accessible outside the swarm as well.

2.5.1. ex

services:
  web:
    build: .
    ports:
      - "8000:8000"
  db:
    image: postgres
    ports:
      - "8001:5432"

2.6. errors

error: for webb No such image: sha256

  • docker-compose -f docker-compose-filename.yml down
  • docker-compose -f docker-compose-filename.yml up

Credentials store error: StoreError

OR remove "credsStore wincredis" from ~/.docker/config.json

infinity restarts

version: '3'
services:
  my-service:
    restart: on-failure:5

2.7. usage

  • docker-compose -f docker-compose.yml up
  • docker-compose up -d
  • docker-compose down
  • docker-compose down –volumes # and remove volumes
  • docker-compose up –no-deps –build <service_name> - rebuild
  • docker-compose build ServiceName+1

2.8. add folder (volumes)

services:

  • xxx_service: volumes: - ./local:/root/container:ro

The imperative way (Docker client):

  • docker volume create [OPTIONS] [VOLUME]

Types of volumes in Docker:

  • Docker host-mounted volumes
    • /host/path:/container/path
  • Docker named volumes
    • named_volume_name:/container/path
    • types:
      • internal - used only inside docker-compose (app)
      • external - can be used across docker installation - must be created by the user

docker external named volume:

  1. docker volume create –driver local –opt type=none –opt device=/var/opt/my_website/dist –opt o=bind web_data
volumes:
  web_data:
    external: true

2.9. Entrypoint and Cmd

Override Docker ENTRYPOINT and CMD.

  • entrypoint: /code/entrypoint.sh
  • entrypoint: ["php", "-d", "memory_limit=-1", "vendor/bin/phpunit"]

entrypoint: ["tail", "-f", "/dev/null"] # sleep

3. hub.docker.com

docker image history –no-trunc image_name > image_history

  • docker search
  • docker pull/push

repository

  • docker login
  • docker logout
  • docker push aa/nginx localhost:5000/myadmin/nginx
  • docker pull myregistry/nginx:master
  • docker pull eon01/nginx localhost:5000/myadmin/nginx
  • docker image history nginx
  • docker image history –no-trunc image_name > image_history
  • docker inspect nginx
  • docker rmi nginx
  • docker rmi $(docker images | grep "^<none>" | awk "{print $3}") - удаление всех образов без тега (none)
  • docker commit nginx - создание образа из контейнера
  • docker tag nginx my/nginx

4. kubernetes (K8S)

[k(j)uːbərˈnɛtɪs]

Процесс в K8S не может создавать под-процессы.

На мастере размещаются служемные поды

  • taind - ограничение

config maps - ?

bootstraping

Модули реализованы через API

  • in-tree
  • out of tree - addons - вне кодовой базы

4.1. theory

K8s is a platform for creating, deploying and managing distributed applications.

  • container-orchestration system for automating application deployment, scaling, and management.

container orchestration system for automating application deployment, scaling, and management. It aims to provide a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts".

It eliminates the need for orchestration - continuously drive the current state towards the provided desired state.

Master/slave (Primary/replica architecture) - model of asymmetric communication (data speed or quantity, when averaged over time, is different in one direction from the other).

4.1.1. Контейнер

  • Это стандартизированный метод упаковки приложения
  • процесс повязанный к namespaces и контролируемый через cgroups

4.1.2. main features

benefits of k8s:

  • Velocity
    • immutability - artifact created and don't change via user modifications
    • declarative configuration - desired state of the system
    • onlineself-healing systems
  • scaling
    • decoupling - each component is separated from other components by defined APIs and service load balancers.
  • abstracting your infrastructure
  • efficiency

o

  • service discovery and load balancing
  • storage orchistration (local storage, public cloud providers, etc)
  • automated rollouts and rollbacks (just tell the desired state) (развертывание и откат)
  • automatic bin packing (CPU and memory needs => a particular node)
  • Self-healing (liveness & readiness probes)
  • Secret and configuration management (deploy & update secrets and configuration without rebuilding container images)

probes

  • liveness - deadlock - application is running, but unable to make progress
  • readiness - when app is start to accept traffic. Pod is considered ready when all of its containers are ready.

4.1.3. Microservices

The ideal team size is the "two-pizza team" - 7-8 ppl.

The development of decoupled, service-oriented teams - each build single microservice.

The aggregation of all of these services provides the implementation of overall product's surface area.

4.1.4. Services & Endpoinds & probes

apiserver -> [NODE kube-proxy -> clusterIP(iptables)->] ->Backed Pod1, Backed Pod2, Backed Pod3

clusterIP - load balancer by iptables with (probability distribution)

endpoint - address:port

if CLUSTER-IP - is not None - it is balanced.

probes?

4.1.5. PV & Provisioners & PVC

PV - persistent volume - just a folder at some server.

containers don't work directly with PV, they use PVC (PV claims) - it is query for storage volume to get PV.

reclaim policy (RECLAIM) - "Delete" mean provisioner (local volume provisioner) may just remove all data from folder and make PV available again. "Retain" mean change state to … and don't remove data.

4.1.6. CRD - add own API to kubernetes

API - (kubectl get pods/alert/…)

External Controller for logs/alerts management.

4.2. history

  • 2003: Borg System - Google
  • 2013: Omega - Google
  • 2014: Kubernetes - Google - as an open source version of Borg
  • 2015: v1.0
  • 2016:
    • First release of Helm the package manager of Kubernetes.
    • Minikube a tool that makes it easy to run Kubernetes locally.
    • Kops official project for managing production-grade Kubernetes clusters.
    • kubeadm
    • supports OpenAPI - Specification defines a standard interface to RESTful APIs
  • 2017:
    • Google and IBM announce Istio
    • role-based access control (RBAC) authorizer
    • Introducing Kubeflow - Machine Learning Stack Built for Kubernetes.
    • Amazon EKS - to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes.
    • Azure AKS
    • the principal competitors:
      • VMware
      • Mesosphere, Inc.
      • Docker, Inc.
      • Microsoft Azure
      • AWS
  • the hard way guilde 4

4.3. objects, components, apis

4.3.1. Abstractions:

main (kind)

pod
group of containers with shared storage and network resources - ip adress, volumes - not durable by default. (has hidden default container)
  • "assigning a pod to a Node"
  • Pods are the smallest deployable units of computing.
service
provide load balancing, naming, discovery
namespace
just group resources
Ingress
objects provice an easy-to-use fronted that can combine multiple microservices into a single externalized API survace area.
configmap
store yaml config as a service.
container
process, unit of service, on the same physical or virtual machine in the cluster.

4.3.2. Kubernetes Objects or API object or API resources or resource type

Objects - persistent entities "record of intent". has spec and status.

  1. list of objects (kind)
    kubectl api-resources # to get all accessible API resources
    kubectl api-resources -o wide | grep tfjob # to get all supported operations for object
    
    • node - Master and Workers. Master - служебный, Worker - для установки
    • pods - one or more containers share the same resources and local network - used as the unit of replication
      • pods are scaled up and down as a unit, all containers in a pod must scale together
      • pods should remain as small as possible, , typically holding only a main process and its tightly-coupled helper containers (side-cars)
    • Ingress - exposes HTTP and HTTPS routes from outside the cluster to services

    Once you create the object, the Kubernetes system will constantly work to ensure that object exists

    • object spec - desired state
    • object status - actual state of the object

    Objects:

    • Node - phisical instance for kubernetes
      • Kubelet - process responsible for communication between the Kubernetes Master and the Node
      • Kube-Proxy - proxy-балансировщик, простейшее перенаправление потоков TCP и UDP (round robin) между набором бэкендов
      • A container runtime (like Docker, rkt)
    • Pod - has unique IP, one or more application containers(tightly coupled) (such as Docker or rkt), and some shared resources for those containers:
      • Volumes - Shared storage for containers
      • Networking, as a unique cluster IP address
      • configmap - kubectl -n aml get configmap provodki-app-cm-env -o yaml
    • service - абстракция которая определяет логический объединённый набор pod и политику доступа к ним.
    • Namespaces - разделить кластер K8S на несколько логических - с квотами не влияя на другие
  2. TODO kubectl get all:
    • pod
    • service
    • deployment.apps
    • replicaset.apps
    • statefulset.apps
    • job.batch

4.3.3. workload resources - high-level abstractions

configure controllers that make sure the right number of the right kind of pod are running, to match the state you specified.

Deployment
wrapper for ReplicaSet (rolling updates (versioning)). Pods are expected to be interchangeable.
ReplicaSet
pods scaling (deprecated) maintain a stable set of replica Pods. (not used not manage workloads
StatefulSet
like Deployment, but maintains a sticky identity for each of its Pods. provide: unique network identifiers, persistent storage, graceful deployment and scaling, automated rolling updates. Most used to be able to make a link between its Pods and their persistent storage. Когда пралажение хранит данные в volume.
DaemonSet
provide facilities that are local to a specific node. Одна реплика определенного пода присутствовала на каждой ноде. Для служебных задач каких-то.
Job/CronJob
task one-off/repeat task. Разовая нагрузка, к примеру, мы хотим при деплои прилажения сделать инициализацию базы данных.
  1. Deployment

    template and selector are the only required fields of the .spec.

    1. template

      Pod template

4.3.4. object spec

Required fields

apiVersion
v1 usualy
kind
What kind of object you want to create.
metadata
Data that helps uniquely identify the object, including a name string, UID, and optional namespace
spec
unuque for every object type. What state you desire for the object

Display an explanation of a specific field

$ kubectl explain [pods.spec.containers

Example:

apiVersion: apps/v1  # default  version of the Kubernetes API
kind: Deployment
metadata:               # uniquely identify the object - name, UID, namespace
  name: nginx-deployment
spec:                   # state you desire for the object
  replicas: 2   # tells deployment to run 2 pods matching the template

4.3.5. pod lifecycle -

Pod.status(PodStatus).phase(string) - is phase

  • Pending -
  • Running - at least one of its primary containers is running, or is in starting or restarting.
  • on of:
    • Succeeded - All containers is terminated in success
    • Failed - any container in the Pod terminated and at least one in failure. or node dies or is disconnected
    • Unknown - error in communicating with the node
  • Completed - status Succeeded is not a phase acutaly it is transitional state. scheduled, but not ready. Containers state is Terminated with Completed reason.

restartPolicy - applies to all containers:

  • Always(default) -
  • OnFailure -
  • Never -

Pod.status(PodStatus).conditions(PodCondition).type(string) - is Pod conditions:

  • PodScheduled - scheduled to a node
  • PodHasNetwork - Pod sandbox has been successfully created
  • ContainersReady - all containers are ready
  • Initialized - all init containers have completed successfully
  • Ready - Pod is able to serve requests

Notes:

  • Pods are only scheduled once in their lifetime - until it stops or is terminated.

4.3.6. container states

  • Waiting - something required
  • Running
  • Terminated

4.3.7. names and uids (all objects have)

  • metadata->Name - is unique for that type of resource.
  • metadata->UIDs - A Kubernetes systems-generated string to uniquely identify objects.

4.3.8. Labels (optional)

metadata->Labels - key/value pairs that are attached to objects.

can be used for:

  • to organize and to select subsets of objects

selects pods with labels:

k get pods -l environment=production,tier=frontend
k get pods -l 'environment in (production, qa)'
k get pods -l 'environment,environment notin (frontend)'

4.4. terms

workload
app running on k8s - one or several components, you run it inside a set of pods.
workload resources
Deployment, StatefulSet
container runtime
software that responsible for running containers.
control loop
is a non-terminating loop that regulates the state of a system. (used in controller)
control plane
set of components that run on mster nodes and control the overall state of the cluster.
bootstaping
is the process of setting up a new Kubernetes cluster from scratch.
provisioning machines
the process of creating and preparing virtual of phisical machines: install OS, configure network, etc.
container
executable image that contains software and all of ins dependencies
ConfigMaps
non-confidential data in key-value pairs. environment-specific configuration.
Secret
an object that contains a small amount of sensitive data such as a password. stored unencrypted in the API server's underlying data store (etcd). 1) as a files in a volume mounted 2) as a env variable 3) By the kubelet when pulling images
Pod template
specifications for creating Pods .spec.template included in workload resources such as Deployments, Jobs, and DaemonSets.
Services
type of object to group pods.
resources
types of objects, can by got by # kubectl api-resources. Defined and deployed with yaml of json files.
cluster
joined worker nodes with "control plane" which is nodes also.
  • Кубернетис кластер, это joined worker nodes with "control plane" which is nodes also. Результат выполнения kubeadm init и kubeadm join, в результате которой поднимаются все необходимые сервисы на нодах и из нод формирутся кластер. К которому можно подключиться через доступ к API server. Подробная статья. https://www.airplane.dev/blog/kubernetes-control-plane
resource
kind of API object
collection
A list of instances of a resource. There are dozens of collection types (such as PodList, ServiceList, and NodeList)
(no term)
reconsilation loop - user-declared desired state -> observed current state (action, observe) - у каждого вида рабочей нагрузки есть свой контроллер который наблюдает за текущим состоянием и желаемым. (kube-controller manager)

4.5. architecture

  • pods - ip adress, volumes - not durable by default. (has hidden default container). a set of running containers on your cluster
  • node - виртуальные машины в VMware. Can run(be assigned with) several pods.
  • controller - высшая обстрация над pods - CRI specification -
  • contrainer - run in pods - binary package with files
    • Docker image Format (often)

4.5.1. master node

  • API Server - communicate with kubelets of Nodes - управление, все к нему обращается.
  • Controller (Manager) - control loops that watch the state of your cluster. tries to move the current cluster state closer to the desired state.
  • Scheduler - планирование размещения pods and workloads на nodes. tracks resource allocation on each node.
  • etcd - ключ-значение хранилище - состояние кластера

4.5.2. worker node

  • kubelet - primary - управляет pod'ами,их контейнерами, образами, разделами, etc.
  • cAdvisor
  • Kube-Proxy - network between pods and final users
  • pods
  • Container runtime

4.5.3. Container Runtime Interface (CRI)

enables the kubelet to use a wide variety of container runtimes, without having a need to recompile the cluster components.

kubelet interacts with container runtimes via the Container Runtime Interface (CRI) or CRI API, which decouples the maintenance of core Kubernetes from the actual CRI implementation.

Implementations of CRI API (CRI plugins):

  • containerd https://github.com/containerd/containerd - can handle any container images which compliant with OCI Distribution Specification
  • CRI-O https://github.com/cri-o/cri-o - Open Container Initiative (OCI) for Linux containers (OS-level virtualization): LXD, Podman, CBL-Mariner.
  • Docker Engine - The software that hosts the containers of Dockers (by the Linux kernel and libvirt, LXC and systemd-nspawn)
  • Mirantis Container Runtime

4.5.4. istio

service mesh is a service-to-service communications between services. Is an abstraction layer over the underlying cluster management platform, such as Kubernete.

  • Service Mesh is the cloud native equivalent of TCP/IP, addressing application network communication, security and visibility issues.
  • istio is relying on Kubernetes but also scalable to virtual machine loads.
  • Istio’s core consists of a control plane and a data plane, with Envoy as the default data-plane agent.

Empower K8s: intelligent routing, automatic load balancing, fine-grained access control, traffic encryption, and policy enforcement. They also enable powerful observability capabilities, including request tracing, metric collection, and distributed logging

Istio for Kubernetes is built right into the app and keeps track of how the different parts of the app interact with each other.

https://github.com/istio/istio

4.5.5. user namespaces

Linux namespaces - feature of the Linux kernel. It makes: one set of processes sees one set of resources while another set of processes sees a different set of resources.

4.6. ЦФТ обучение (to delete)

  • \\srv-file\Share\DEV\Обучение\AML
  • k create deployment nginx –image=nginx - создать pod c одной репликой
  • k scale deployment nginx –replicas 3 - реплики 3

pod name - nginx-(HashOfConfiguration)-randomValue

Deployments

  • случайные имена
  • чтобы мы не делали сервис будет работать

StatefulSet DaemonSets Job - при инициализации приложения

4.7. master and worker node (rusnarbank)

linux

  • kubelet - node and master - взаимодействуют с API-сервером и подписываются на выставленные данных scheduler-oм
  • kube-proxy - отвечает за настройку iptables
  • Kubernetes CRI Container Runtime Interface
    • на dev - docker CRI shim
    • prod - CRI-O
  • Addons дополнительные компоненты worker - kubectl -n kube-system get pods
    • fluent-d - logging - Cluster-level Logging
    • weave addon - creates a virtual network that connects containers across multiple hosts and enables their automatic discovery
    • coredns addon
    • ambasador API gateway

master

  • kube-apiserver - API Server - все взаимодействие с k8s
  • kube-controller-manager - Controller manager - reconsilation loop
  • kube-scheduler - Scheduler - оценить на какой ноде должен быть запущен контейнер
  • etcd - distributed key-value store, у значений есть версия. Optimistic concurrency control

4.7.1. kubelet

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

сервис запускается в самой хостовой системе через systemd

мониторит pods

4.7.2. есть кластеры:

  1. AML dev
  2. AML prod
  3. norma2 test 2 masters
  4. norma2 prod 3 masters

Grafana, Prometeus - мониторинг кубернетиса

4.8. Ansible

Используется для развертывания подов на виртуалках. Kubernetes cluster on bare-metal or VMs.

Использует Vagrant?

Playbooks are Ansible’s configuration, deployment, and orchestration language. playbooks are your instruction manuals

4.9. from image to pod:

  • web server - gitlab registry
  • Scheduler - command to kubelet
  • kubelet use cri (Container Runtime Interface) to pull nodes

4.10. primary/replica architecture (Master/slave):

master - one or many nodes - control plane

  • etcd - distributed key-value data store stores the configuration (like Apache ZooKeeper)
  • API server - internal and external interface - REST JSON over HTTP
  • Scheduler - запускает pods на node based on resource availability - match resource "supply" to workload "demand"
  • Controller manager - communicating with the API server to create, update, and delete the resources it manages (pods, service endpoints, etc.).
    • Replication Controller - handles replication and scaling
    • Autosclaer - use metrics from Prometheus чтобы подстроить количество репли к количеству запросов

node or Worker -

  • Kubelet - starting, stopping, and maintaining application containers organized into pods
  • Kube-proxy (network proxy and a load balancer) - routing traffic to the appropriate container
  • Container runtime - must implement Container Runtime Interface (CRI) - Most container runtime environments use runc.
    • Docker
    • CRI-O
    • Containerd
    • Other CRI runtimes: frakti

runc provide OCI API. Kubelet communicate with the container runtime via the CRI’s gRPC interface.

4.11. CAP theorem

it is impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees:

  • Consistency: Every read receives the most recent write or an error
  • Availability: Every request receives a (non-error) response, without the guarantee that it contains the most recent write
  • Partition tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes

4.12. runC

4.13. ReplicationControllerо

homogeneous ˌhōməˈjēnēəs - однородная система heterogeneous ˌhɛt(ə)rə(ʊ)ˈdʒiːnɪəs - неоднородная система

4.14. app logs

3 variants:

  • letting applications directly output their traces in other systems
  • collector per pod
  • collector per node - this

Two architectures for log managements:

  1. "Elastic Stack" (formerly the "ELK stack") (products for Elastic Search):
    • Logstash - data collection and log-parsing engine
    • Kibana - visualisation platform
    • ?
    • ?
  2. Graylog - three-tier architecture and scalable storage (based on Elasticsearch - its backend)

Collecting logs per node - Logstash or (FileBeat, Fluentd, Fluent Bit…)

4.15. Helm - Пакетный менеджер

Think of it like apt/yum/homebrew for Kubernetes.

  • Container: Container refers to operating system level virtualization
  • Docker: Docker is a popular program to create and run containers
  • Kubernetes: Kubernetes is a popular container orchestration program.

Helm cnsist of:

  • Tiller Server: installed within a Kubernates cluster - interacts with the Kubernetes API server to install, upgrade, query and remove Kubernetes resources
  • Client: command-line interface - install, upgrade and rollback charts

Charts - Kubernetes resources package

  • collection of files
  • configuration

Release - running instance of a chart

4.15.1. .helm

  • Chart.yaml : This is the main file that contains the description of our chart
  • values.yaml : this is the file that contains the default values for our chart
  • templates/ : This is the directory where Kubernetes resources are defined as templates
  • .helmignore: patterns to ignore when packaging (similar in concept to .gitignore)

Template:

4.16. cluster management software

https://en.wikipedia.org/wiki/List_of_cluster_management_software

  • Kubernetes - google
  • Docker Swarm
  • CoreOS
  • Mesos – Apache

4.17. как могут поды иметь свои IP ведь они на одной ноде

оверлейная сеть

4.18. kubectl

4.18.1. theory

Config: uses $HOME/.kube/ or $KUBECONFIG or –kubeconfig or direct connect

to see config:

kubectl config view

direct connect:

kubectl get nodes \
    --server https://localhost:6443 \
    --user docker-for-desktop \
    --client-certificate my.cert \
    --client-key my.key \
    --insecure-skip-tls-verify

kubectl [command] [TYPE] [NAME] [flags]

TYPE
Specifies the resource type.
flags
Specifies optional flags. For example, you can use the -s or –server

4.18.2. kubectl get

  • all

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 307d

  • namespaces

4.18.3. Viewing and finding resources

# Get commands with basic output
kubectl get services                          # List all services in the namespace
kubectl get pods --all-namespaces             # List all pods in all namespaces
kubectl get pods -o wide                      # List all pods in the current namespace, with more details
kubectl get deployment my-dep                 # List a particular deployment
kubectl get pods                              # List all pods in the namespace
kubectl get pod my-pod -o yaml                # Get a pod's YAML

# Describe commands with verbose output
kubectl describe nodes my-node
kubectl describe pods my-pod

# List Services Sorted by Name
kubectl get services --sort-by=.metadata.name

# List pods Sorted by Restart Count
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'

# List PersistentVolumes sorted by capacity
kubectl get pv --sort-by=.spec.capacity.storage

# Get the version label of all pods with label app=cassandra
kubectl get pods --selector=app=cassandra -o \
  jsonpath='{.items[*].metadata.labels.version}'

# Retrieve the value of a key with dots, e.g. 'ca.crt'
kubectl get configmap myconfig \
  -o jsonpath='{.data.ca\.crt}'

# Retrieve a base64 encoded value with dashes instead of underscores.
kubectl get secret my-secret --template='{{index .data "key-name-with-dashes"}}'

# Get all worker nodes (use a selector to exclude results that have a label
# named 'node-role.kubernetes.io/control-plane')
kubectl get node --selector='!node-role.kubernetes.io/control-plane'

# Get all running pods in the namespace
kubectl get pods --field-selector=status.phase=Running

# Get ExternalIPs of all nodes
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'

# List Names of Pods that belong to Particular RC
# "jq" command useful for transformations that are too complex for jsonpath, it can be found at https://stedolan.github.io/jq/
sel=${$(kubectl get rc my-rc --output=json | jq -j '.spec.selector | to_entries | .[] | "\(.key)=\(.value),"')%?}
echo $(kubectl get pods --selector=$sel --output=jsonpath={.items..metadata.name})

# Show labels for all pods (or any other Kubernetes object that supports labelling)
kubectl get pods --show-labels

# Check which nodes are ready
JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}' \
 && kubectl get nodes -o jsonpath="$JSONPATH" | grep "Ready=True"

# Output decoded secrets without external tools
kubectl get secret my-secret -o go-template='{{range $k,$v := .data}}{{"### "}}{{$k}}{{"\n"}}{{$v|base64decode}}{{"\n\n"}}{{end}}'

# List all Secrets currently in use by a pod
kubectl get pods -o json | jq '.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name' | grep -v null | sort | uniq

# List all containerIDs of initContainer of all pods
# Helpful when cleaning up stopped containers, while avoiding removal of initContainers.
kubectl get pods --all-namespaces -o jsonpath='{range .items[*].status.initContainerStatuses[*]}{.containerID}{"\n"}{end}' | cut -d/ -f3

# List Events sorted by timestamp
kubectl get events --sort-by=.metadata.creationTimestamp

# List all warning events
kubectl events --types=Warning

# Compares the current state of the cluster against the state that the cluster would be in if the manifest was applied.
kubectl diff -f ./my-manifest.yaml

# Produce a period-delimited tree of all keys returned for nodes
# Helpful when locating a key within a complex nested JSON structure
kubectl get nodes -o json | jq -c 'paths|join(".")'

# Produce a period-delimited tree of all keys returned for pods, etc
kubectl get pods -o json | jq -c 'paths|join(".")'

# Produce ENV for all pods, assuming you have a default container for the pods, default namespace and the `env` command is supported.
# Helpful when running any supported command across all pods, not just `env`
for pod in $(kubectl get po --output=jsonpath={.items..metadata.name}); do echo $pod && kubectl exec -it $pod -- env; done

# Get a deployment's status subresource
kubectl get deployment nginx-deployment --subresource=status

4.18.4. request step by step

kubectl config view - cat  /etc/kubernetes/kubelet.conf

By default, the Kubernetes API server listens on port 6443 on the first non-localhost network interface, protected by TLS.

kube-apiserver is pods

4.19. kubeadm

kubeadm init and kubeadm join as best-practice "fast paths" for creating Kubernetes clusters.

  • kubeadm init to bootstrap a Kubernetes control-plane node
  • kubeadm join to bootstrap a Kubernetes worker node and join it to the cluster

4.20. well-known values and paths

  • /etc/kubernetes/manifests as the path where kubelet should look for static Pod manifests.

4.21. Installation steps

  1. Verify the MAC address and product_uuid are unique for every node
    1. ip link or ifconfig -a
    2. cat /sys/class/dmi/id/product_uuid
  2. check that required ports for master and worker nodes are open and don't blocked by firewall
  3. Installing a container runtime, most common:
    • containerd - began as part of Docker
    • CRI-O
    • Docker Engine
    • Mirantis Container Runtime
  4. Installing kubeadm, kubelet and kubectl
  5. Configuring a cgroup driver - isolates the resource usage (CPU, memory, disk I/O, etc.) of a collection of processes.
  6. Creating a cluster with kubeadm https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
    1. install to provisioning systems such as Ansible or Terraform or on your own.

4.22. .bashrc

apt-get install bash-completion

## Install
apt-get install bash-completion
## Bash
echo 'source <(kubectl completion bash)' >>~/.bashrc

echo 'alias k="kubectl"' >>~/.bashrc

4.23. common shorts

  • ns namespaces
  • componentstatuses cs
  • configmaps cm
  • endpoints ep
  • events ev
  • limitranges limits
  • namespaces ns
  • nodes no
  • persistentvolumeclaims pvc
  • persistentvolumes pv
  • pods po
  • replicationcontrollers rc
  • resourcequotas quota
  • serviceaccounts sa
  • services svc
  • customresourcedefinitions crd, crds
  • daemonsets ds
  • deployments deploy
  • replicasets rs
  • statefulsets sts
  • horizontalpodautoscalers hpa
  • cronjobs cj
  • certificiaterequests cr, crs
  • certificates cert, certs
  • certificatesigningrequests csr
  • ingresses ing
  • networkpolicies netpol
  • podsecuritypolicies psp
  • replicasets rs
  • scheduledscalers ss
  • priorityclasses pc
  • storageclasses sc

4.24. role-based access control (RBAC) policy

establish what resources a given role is able to access. Implement principle of least privilege.

kubectl auth can-i '*' '*' # allowed commands

4.25. system maintentance

kube-system - namespace for system services

k -n kube-system get all

4.26. Usage My

alias k=kubectl

4.26.1. explore

kubectl config view # cat /etc/kubernetes/kubelet.conf
kubectl cluster-info & kubectl cluster-info dump # more info
kubectl get nodes
kubectl describe node kube-worker-1
k logs -n kube-system kube-apiserver... # get pod description

# get containers in pod:
kubectl get pod --namespace=kube-system coredns-d46f87476-c5r4m  -o jsonpath='{.spec.containers[*].name}'

k get pods -n ml1 -o wide # get pods in namespace
k get -n ml1 pod paramserv-worker-0 -o yaml # get yaml of pod
k get ns # все неймспейсы
k -n aml get all #  все в неймспейсе
k -n aml exec -it front-77d466758c-ttpfj -- sh
k -n aml describe pods/podname # detailed information about a resource
k -n aml get configmap provodki-app-cm-env -o yaml # configmap for pod - Labels - ключ значение

# -- #
k get pod podname # Verify that the container is running
k exec --stdin --tty podname -- /bin/bash       # access to pod=podname
k exec -it podname -- /bin/bash         # access to pod=podname

4.26.2. manage

# -- test
k -n ml1 apply --dry-run='client' -f a.yaml: #  Must be "none", "server", or "client"

# -- control resources
k -n ml1 delete -f ./ML1/k8s/paramserv.yaml
# force delete pod in Terminated state:
kubectl delete pod paramserv-ps-0 -n ml1 --grace-period=0 --force

k -n ml1 apply -f ./ML1/k8s/paramserv.yaml # creating a pod
kubectl replace --force -f ./ML1/k8s/paramserv.yaml # delete + apply

kubectl scale deployment [deployment_name] --replicas=0 # change count of replicas

# disable node for scheduling
kubectl label node kube-exp-w3.k8s.sumus.work GPU-

# add label to pod
kubectl label pods foo unhealthy=true

4.27. USAGE (max prokopenko)

# Get-команды с основном выводом
kubectl get services # Вывести все сервисы в пространстве имён
kubectl get pods --all-namespaces # Вывести все поды во всех пространств имён
kubectl get pods -o wide # Вывести все поды в текущем пространстве имён с
подробностями
kubectl get deployment my-dep # Вывести определённое развёртывание
kubectl get pods # Вывести все поды в пространстве имён
kubectl get pod my-pod -o yaml # Получить информацию по поду в формате YAML
# Посмотреть дополнительные сведения команды с многословным выводом
kubectl describe nodes my-node
kubectl describe pods my-pod
# Вывести сервисы, отсортированные по имени
kubectl get services --sort-by=.metadata.name
# Вывести поды, отсортированные по количеству перезагрузок
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'
# Вывести постоянные тома (PersistentVolumes), отсортированные по емкости
kubectl get pv --sort-by=.spec.capacity.storage
# Получить метку версии всех подов с меткой app=cassandra
kubectl get pods --selector=app=cassandra -o \
jsonpath='{.items[*].metadata.labels.version}'
# Получить все рабочие узлы (с помощью селектора исключаем узлы с меткой 'node-role.kubernetes.io/master')
kubectl get node --selector='!node-role.kubernetes.io/master'
# Получить все запущенные поды в пространстве имён
kubectl get pods --field-selector=status.phase=Running
# Получить внешние IP-адреса (ExternalIP) всех узлов
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'
# Вывести имена подов, принадлежащие к определённому RC
# Использование команды "jq" помогает упросить поиск в jsonpath, подробнее смотрите на сайте https://stedolan.github.io/jq/
sel=${$(kubectl get rc my-rc --output=json | jq -j '.spec.selector | to_entries | .[] | "\(.key)=\(.value),"' )%? }
echo $(kubectl get pods --selector=$sel --output=jsonpath={.items..metadata.name})
# Показать метки всех подов (или любого другого объекта Kubernetes, которым можноприкреплять метки)
kubectl get pods --show-labels

# Получить готовые узлы
JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}
{@.type}={@.status};{end}{end}' \
&& kubectl get nodes -o jsonpath="$JSONPATH" | grep "Ready=True"
# Вывод декодированных секретов без внешних инструментов
kubectl get secret my-secret -o go-template='{{range $k,$v := .data}}{{"### "}}{{$k}}{{"\n"}}
{{$v|base64decode}}{{"\n\n"}}{{end}}'
# Вывести все секреты, используемые сейчас в поде.

kubectl get pods -o json | jq '.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name' | grep -
v null | sort | uniq
# Вывести все идентификаторы (containerID) контейнеров инициализации (initContainers) во
всех подах.

# Это полезно при очистке остановленных контейнеров, не удаляя при этом контейнеры
инициализации.

kubectl get pods --all-namespaces -o jsonpath='{range .items[*].status.initContainerStatuses[*]}
{.containerID}{"\n"}{end}' | cut -d/ -f3
# Вывести события, отсортированные по временной метке
kubectl get events --sort-by=.metadata.creationTimestamp
# Сравнить текущее состояние кластера с состоянием, в котором находился бы кластер в
случае применения манифеста.

kubectl diff -f ./my-manifest.yaml

4.28. usage

4.28.1. Обновление ресурсов

kubectl set image deployment/frontend www=image:v2 # Плавающее обновление контейнеров "www" развёртывания "frontend", обновление образа kubectl rollout history deployment/frontend # Проверить историю развёртывания, включая ревизии.

kubectl rollout undo deployment/frontend # Откатиться к предыдущему развёртыванию kubectl rollout undo deployment/frontend –to-revision=2 # Откатиться к определённой ревизии kubectl rollout status -w deployment/frontend # Отслеживать статус плавающего развёртывания "frontend" до его завершения kubectl rollout restart deployment/frontend # Перезапуск плавающего развёртывания "frontend"

kubectl get pod mypod -o yaml | sed 's/\(image: myimage\):.*$/\1:v4/' | kubectl replace -f - kubectl label pods my-pod new-label=awesome # Добавить метку kubectl annotate pods my-pod icon-url=http://goo.gl/XXBTWq # Добавить аннотацию kubectl autoscale deployment foo –min=2 –max=10 # Автоматически масштабировать развёртывание "foo" в диапазоне от 2 до 10 подов

4.28.2. удаление ресурсов

kubectl delete -f ./pod.json # Удалить под по типу и имени в pod.json kubectl delete pod,service baz foo # Удалить поды и сервисы с одноимёнными именам "baz" и "foo" kubectl delete pods,services -l name=myLabel # Удалить поды и сервисы с именем метки myLabel kubectl -n my-ns delete pod,svc –all # Удалить все поды и сервисы в пространстве имен my-ns

kubectl get pods -n mynamespace –no-headers=true | awk '/pattern1|pattern2/{print $1}' | xargs kubectl delete -n mynamespace pod

4.28.3. работа с подами, логами

kubectl logs my-pod # вывести логи пода (в stdout) kubectl logs -l name=myLabel # вывести логи пода с меткой myLabel (в stdout) kubectl logs my-pod –previous # вывести логи пода (в stdout) по предыдущему экземпляру контейнера kubectl logs my-pod -c my-container # вывести логи контейнера пода (в stdout, при работе с несколькими контейнерами) kubectl logs -l name=myLabel -c my-container # вывести логи пода с меткой myLabel (в stdout) kubectl logs my-pod -c my-container –previous # вывести логи контейнера пода (в stdout, при работе с несколькими контейнерами) по предыдущему экземпляру контейнера kubectl logs -f my-pod # вывести логи пода в режиме реального времени (в stdout) kubectl logs -f my-pod -c my-container # вывести логи контейнера пода в режиме реального времени (в stdout, при работе с несколькими контейнерами) kubectl logs -f -l name=myLabel –all-containers # вывести логи всех подов с меткой myLabel (в stdout) kubectl run -i –tty busybox –image=busybox – sh # запустить под как интерактивную оболочку kubectl run nginx –image=nginx –restart=Never -n mynamespace # Запустить под nginx в заданном пространстве имён kubectl run nginx –image=nginx –restart=Never # Запустить под nginx и записать его спецификацию в файл pod.yaml –dry-run -o yaml > pod.yaml kubectl attach my-pod -i # Прикрепить к запущенному контейнеру kubectl port-forward my-pod 5000:6000 # Переадресовать порт 5000 в локальной машине на порт 6000 в поде my-pod kubectl exec my-pod – ls / # Выполнить команду в существующем поде (в случае одного контейнера).

kubectl exec my-pod -c my-container – ls / # Выполнить команду в существующем поде (в случае нескольких контейнеров) kubectl top pod POD_NAME –containers # Показать метрики по заданному поду вместе с его контейнерами

4.28.4. Работа с кластером

kubectl config view # показать объединённые настройки kubeconfig

конфигурацию из этих файлов KUBECONFIG=~/.kube/config:~/.kube/kubconfig2 kubectl config view

kubectl config view -o jsonpath='{.users[?(@.name = "e2e")].user.password}' kubectl config view -o jsonpath'{.users[].name}' # показать первого пользователя kubectl config view -o jsonpath='{.users[*].name}' # получить список пользователей kubectl config get-contexts # показать список контекстов kubectl config current-context # показать текущий контекст (current-context) kubectl config use-context my-cluster-name # установить my-cluster-name как контекст по умолчанию

kubectl config set-credentials kubeuser/foo.kubernetes.com –username=kubeuser – password=kubepassword

kubectl config set-context –current –namespace=ggckad-s2

kubectl config set-context gce –user=cluster-admin –namespace=foo \ && kubectl config use-context gce kubectl config unset users.foo # удалить пользователя foo kubectl cordon my-node # Отметить узел my-node как неназначаемый kubectl drain my-node # Вытеснить узел my-node, чтобы подготовиться к эксплуатации kubectl uncordon my-node # Отметить узел my-node как назначаемый kubectl top node my-node # Показать метрики по заданному узлу kubectl cluster-info # Показать адреса главного узла и сервисов kubectl cluster-info dump # Вывести состояние текущего кластера в stdout kubectl cluster-info dump –output-directory=/path/to/cluster-state # Вывести состояние текущего кластера в /path/to/cluster-state

заменено указанным kubectl taint nodes foo dedicated=special-user:NoSchedule

4.29. What is Krew?

Krew is the plugin manager for kubectl command-line tool.

4.30. create namespace

k get ns --show-labels # with labels
---

apiVersion: v1
kind: Namespace
metadata:
  # name: mlsynth
  name: ml1
{'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'ml1'}}

4.31. create custom resource

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: dolphins.ship.io
spec:
group: ship.io
versions:
   - name: v1
     served: true
     storage: true
     schema:
       openAPIV3Schema:
         type: object
         properties:
           spec:
             type: object
             properties:
               name:
                 type: string
scope: Namespaced
names:
   plural: dolphins
   singular: dolphin
   kind: Dolphin
   shortNames:
   - dolphin
   categories:
   - all

4.32. one pod per node

template:
  metadata:
    labels:
      app: paramserv
  spec:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - paramserv
          topologyKey: kubernetes.io/hostname
{'template': {'metadata': {'labels': {'app': 'paramserv'}},
              'spec': {'affinity': {'podAntiAffinity': {'requiredDuringSchedulingIgnoredDuringExecution': [{'labelSelector': {'matchExpressions': [{'key': 'app',
                                                                                                                                                    'operator': 'In',
                                                                                                                                                    'values': ['paramserv']}]},
                                                                                                            'topologyKey': 'kubernetes.io/hostname'}]}}}}}

4.33. known errors and misleads in kubectl

k -n ml1 get all # will get pods and services

4.34. Troubleshooting

-v, –v Level number for the log level verbosity. Up to 8.

kubectl cluster-info - Display cluster info

responses:

4.35. DNS

4.35.1. theory

CoreDNS is a general-purpose authoritative DNS server that can serve as cluster DNS

4.35.2. Check if my DNS service is up:

$ kubectl get svc –namespace=kube-system NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 5d

Check if DNS endpoints are exposed:

$ kubectl get ep kube-dns –namespace=kube-system NAME ENDPOINTS AGE kube-dns 10.244.0.5:53,10.244.0.5:53 5d

Check the contents of /etc/resolv.conf in my container: $ kubectl exec -ti busybox – cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5

logs from the kube-dns container: $ kubectl logs –namespace=kube-system $(kubectl get pods –namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns

kubectl rollout restart -n kube-system deployment/coredns

4.35.3. edit configmap for CoreDNS

  • kubectl get -n kube-system configmaps coredns -o yaml > core_dns.yaml
  • kubectl replace -n kube-system -f core_dns.yaml

or

  • kubectl -n kube-system edit configmap coredns

4.36. TODO network & security

  • cilium - networking, observability, and security solution with an eBPF-based dataplane.
  • Calico - full-blown Kubernetes “networking solution” with functionality including network policy controller, kube-proxy replacement and network traffic observability.

5. REST API Documentation UIs

In 2015, the Swagger standard changed its name to OpenAPI. Your API documentation will be displayed through the Swagger UI.

  • DapperDox -
  • ReDoc
  • RAML 2 HTML -

5.2. Swagger - развязность

OpenAPI Specification (formerly Swagger Specification) is an API description format for REST APIs.

YAML, being a superset of JSON

Consist off

  • Swagger Editor – browser-based editor where you can write OpenAPI specs.
  • Swagger UI – renders OpenAPI specs as interactive API documentation.
  • Swagger Codegen – generates server stubs and client libraries from an OpenAPI spec.

5.2.1. swagger offline

Create spec.js file containg Swagger JSON https://stackoverflow.com/questions/30400477/how-to-open-local-files-in-swagger-ui

  <script src='spec.js' type="text/javascript"></script>
  <script type="text/javascript">

      var url = window.location.search.match(/url=([^&]+)/);
      if (url && url.length > 1) {
        url = decodeURIComponent(url[1]);
      } else {
        url = "http://petstore.swagger.io/v2/swagger.json";
      }

  window.onload = function() {
    // Build a system
  const editor = SwaggerEditorBundle({
     // url: url,
      spec: spec,

Hide editor

 /**

    Styling for printing out of the editor

 */
.Pane1,
.topbar {
display: none;
}
.SplitPane {
position: relative !important;
display: block !important;
}
.Pane2 {
overflow-y: auto;
width: 100% !important;
}

5.2.2. Specification 2.0

https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md

  • Field Names are case sensitive:
    • Fixed fields, which have a declared name
    • Patterned fields, which declare a regex pattern for the field name
  • # Comment
  • list - [] or - with \n
  1. Data Types
    integer integer int32 signed 32 bits  
    long integer int64 signed 64 bits  
    float number float   10.1
    double number double    
    string string     "blabla"
    byte string byte base64 encoded characters  
    binary string binary any sequence of octets  
    boolean boolean     true false
    date string date As defined by full-date - RFC3339  
    dateTime string date-time As defined by date-time - RFC3339  
    password string password Used to hint UIs the input needs to be obscured.  

    "file" - used by the Parameter Object and the Response Object to set the parameter type or the response as being a file

    formats - to more finely define the data (accompanied by property)

    • "email", "uuid", etc.,
  2. Hierarchy

    Swagger Object - root. Required subobjects:

    • swagger: "2.0"
    • info
      • version: string - application API version
    • paths - The available paths and operations for the API.

    not required:

    • definitions Definitions Object - data types produced and consumed by operations - used by schema: \n $ref: "#/definitions/Order"
    1. Paths Object

      /{path} Path Item Object

      • get
      • put
      • post
      • delete
      • options
      • head
      • patch
      • parameters - общие для всех операций на этом Path. Могут быть overriden на уровне операции

      Operation Object

      • tags [] - for logical grouping
      • summary - what the operation does
      • description - GFM syntax can be used for rich text representation
      • externalDocs
      • operationId: "addPet" - MUST be unique among all operations described in the API - recommended to follow common programming naming conventions
      • consumes: - for all parameters
        • "application/json"
        • "application/xml"
      • produces: - for all responses
        • "application/xml"
        • "application/json"
      • parameters - [Parameter Object]
      • responses - Responses Object REQUIRED list of sub
      • schemes ["http", "https"]
      • deprecated boolean
      • security - [Security Requirement Object]

      Parameter Object parameter types:

      • /pet/{petId}: - from path petId
      • query (/items?id=###)
      • header - Custom headers that are expected as part of the request.
      • body - payload of an HTTP - can only be one body parameter
      • form - payload of an HTTP request when either application/x-www-form-urlencoded, multipart/form-data or both are used as the content type - only parameter type that can be used to send files - cannot be declared together with a body parameter

      fields:

      • name string - case sensitive REQUIRED
      • in string - parameter type "query", "header", "path", "formData" or "body" REQUIRED
      • description - string
      • required boolean - for path must be true
      • schema - Schema Object - for body only

      not for body:

      • type - REQUIRED - "string", "number", "integer", "boolean", "array" (or "file" for "formData")
      • format - string see Data Type Formats
      • allowEmptyValue boolean - Default false
      • items ItemsObject - Required if type is "array"
      • default any - MUST conform to type

      Responses Object

      • status code
        • description string REQUIRED
        • schema
        • headers
        • examples
    2. Schema
      definitions:
        Id:
          type: "object"
          properties:
            id:
              type: "integer"
              format: "int64"
          xml:
            name: "Id"
        Exception:
          type: "object"
          properties:
            status:
              type: "string"
              example: "exception"
            description:
              type: "string"
          xml:
            name: "Exception"
      

Created: 2024-03-03 Sun 09:49

Validate