内部负载均衡和 VIP

创建一个mynet的overlay网络，创建一个service

  1. vagrant@swarm-manager:~$ docker network ls
  2. NETWORK ID     NAME              DRIVER    SCOPE
  3. afc8f54c1d07   bridge            bridge    local
  4. 128fd1cb0fae   docker_gwbridge   bridge    local
  5. 0ea68b0d28b9   host              host      local
  6. 14fy2l7a4mci   ingress           overlay   swarm
  7. lpirdge00y3j   mynet             overlay   swarm
  8. a8edf1804fb6   none              null      local
  9. vagrant@swarm-manager:~$ docker service create --name web --network mynet --replicas 2 containous/whoami
  10. jozc1x1c1zpyjl9b5j5abzm0g
  11. overall progress: 2 out of 2 tasks
  12. 1/2: running   [==================================================>]
  13. 2/2: running   [==================================================>]
  14. verify: Service converged
  15. vagrant@swarm-manager:~$ docker service ls
  16. ID             NAME      MODE         REPLICAS   IMAGE                      PORTS
  17. jozc1x1c1zpy   web       replicated   2/2        containous/whoami:latest
  18. vagrant@swarm-manager:~$ docker service ps web
  19. ID             NAME      IMAGE                      NODE            DESIRED STATE   CURRENT STATE            ERROR     PORTS
  20. pwi87g86kbxd   web.1     containous/whoami:latest   swarm-worker1   Running         Running 47 seconds ago
  21. xbri2akxy2e8   web.2     containous/whoami:latest   swarm-worker2   Running         Running 44 seconds ago
  22. vagrant@swarm-manager:~$

创建一个client

  1. vagrant@swarm-manager:~$ docker service create --name client --network mynet xiaopeng163/net-box:latest ping 8.8.8.8
  2. skbcdfvgidwafbm4nciq82env
  3. overall progress: 1 out of 1 tasks
  4. 1/1: running   [==================================================>]
  5. verify: Service converged
  6. vagrant@swarm-manager:~$ docker service ls
  7. ID             NAME      MODE         REPLICAS   IMAGE                        PORTS
  8. skbcdfvgidwa   client    replicated   1/1        xiaopeng163/net-box:latest
  9. jozc1x1c1zpy   web       replicated   2/2        containous/whoami:latest
  10. vagrant@swarm-manager:~$ docker service ps client
  11. ID             NAME       IMAGE                        NODE            DESIRED STATE   CURRENT STATE            ERROR     PORTS
  12. sg9b3dqrgru4   client.1   xiaopeng163/net-box:latest   swarm-manager   Running         Running 28 seconds ago
  13. vagrant@swarm-manager:~$

尝试进入client这个容器，去ping web这个service name， 获取到的IP 10.0.1.30，称之为VIP（虚拟IP）

  1. vagrant@swarm-manager:~$ docker container ls
  2. CONTAINER ID   IMAGE                        COMMAND          CREATED          STATUS          PORTS     NAMES
  3. 36dce35d56e8   xiaopeng163/net-box:latest   "ping 8.8.8.8"   19 minutes ago   Up 19 minutes             client.1.sg9b3dqrgru4f14k2tpxzg2ei
  4. vagrant@swarm-manager:~$ docker container exec -it 36dc sh
  5. /omd # curl web
  6. Hostname: 6039865a1e5d
  7. IP: 127.0.0.1
  8. IP: 10.0.1.32
  9. IP: 172.18.0.3
  10. RemoteAddr: 10.0.1.37:40972
  11. GET / HTTP/1.1
  12. Host: web
  13. User-Agent: curl/7.69.1
  14. Accept: */*
  16. /omd # curl web
  17. Hostname: c3b3e99b9bb1
  18. IP: 127.0.0.1
  19. IP: 10.0.1.31
  20. IP: 172.18.0.3
  21. RemoteAddr: 10.0.1.37:40974
  22. GET / HTTP/1.1
  23. Host: web
  24. User-Agent: curl/7.69.1
  25. Accept: */*
  27. /omd # curl web
  28. Hostname: 6039865a1e5d
  29. IP: 127.0.0.1
  30. IP: 10.0.1.32
  31. IP: 172.18.0.3
  32. RemoteAddr: 10.0.1.37:40976
  33. GET / HTTP/1.1
  34. Host: web
  35. User-Agent: curl/7.69.1
  36. Accept: */*
  38. /omd #
  39. /omd # ping web -c 2
  40. PING web (10.0.1.30): 56 data bytes
  41. 64 bytes from 10.0.1.30: seq=0 ttl=64 time=0.044 ms
  42. 64 bytes from 10.0.1.30: seq=1 ttl=64 time=0.071 ms
  44. --- web ping statistics ---
  45. 2 packets transmitted, 2 packets received, 0% packet loss
  46. round-trip min/avg/max = 0.044/0.057/0.071 ms
  47. /omd #

这个虚拟IP在一个特殊的网络命令空间里，这个空间连接在我们的mynet这个overlay的网络上

通过 docker network inspect mynet 可以看到这个命名空间，叫lb-mynet

  1. "Containers": {
  2. "36dce35d56e87d43d08c5b9a94678fe789659cb3b1a5c9ddccd7de4b26e8d588": {
  3.     "Name": "client.1.sg9b3dqrgru4f14k2tpxzg2ei",
  4.     "EndpointID": "e8972d0091afaaa091886799aca164b742ca93408377d9ee599bdf91188416c1",
  5.     "MacAddress": "02:42:0a:00:01:24",
  6.     "IPv4Address": "10.0.1.36/24",
  7.     "IPv6Address": ""
  8. },
  9. "lb-mynet": {
  10.     "Name": "mynet-endpoint",
  11.     "EndpointID": "e299d083b25a1942f6e0f7989436c3c3e8d79c7395a80dd50b7709825022bfac",
  12.     "MacAddress": "02:42:0a:00:01:25",
  13.     "IPv4Address": "10.0.1.37/24",
  14.     "IPv6Address": ""
  15. }

通过下面的命令，找到这个命名空间的名字

  1. vagrant@swarm-manager:~$ sudo ls /var/run/docker/netns/
  2. 1-14fy2l7a4m  1-lpirdge00y  dfb766d83076  ingress_sbox  lb_lpirdge00
  3. vagrant@swarm-manager:~$

名字叫 lb_lpirdge00

通过nsenter进入到这个命名空间的sh里， 可以看到刚才的VIP地址10.0.1.30

  1. vagrant@swarm-manager:~$ sudo nsenter --net=/var/run/docker/netns/lb_lpirdge00 sh
  2. #
  3. # ip a
  4. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
  5.     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  6.     inet 127.0.0.1/8 scope host lo
  7.     valid_lft forever preferred_lft forever
  8. 50: eth0@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
  9.     link/ether 02:42:0a:00:01:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0
  10.     inet 10.0.1.37/24 brd 10.0.1.255 scope global eth0
  11.     valid_lft forever preferred_lft forever
  12.     inet 10.0.1.30/32 scope global eth0
  13.     valid_lft forever preferred_lft forever
  14.     inet 10.0.1.35/32 scope global eth0
  15.     valid_lft forever preferred_lft forever
  16. #

和ingress网络一样，可以查看iptables，ipvs的负载均衡， 基本就可以理解负载均衡是怎么一回事了。 Mark=0x106, 也就是262（十进制），会轮询把请求发给10.0.1.31 和 10.0.1.32

  1. # iptables -nvL -t mangle
  2. Chain PREROUTING (policy ACCEPT 128 packets, 11198 bytes)
  3. pkts bytes target     prot opt in     out     source               destination
  5. Chain INPUT (policy ACCEPT 92 packets, 6743 bytes)
  6. pkts bytes target     prot opt in     out     source               destination
  7. 72  4995 MARK       all  --  *      *       0.0.0.0/0            10.0.1.30            MARK set 0x106
  8.     0     0 MARK       all  --  *      *       0.0.0.0/0            10.0.1.35            MARK set 0x107
  10. Chain FORWARD (policy ACCEPT 36 packets, 4455 bytes)
  11. pkts bytes target     prot opt in     out     source               destination
  13. Chain OUTPUT (policy ACCEPT 101 packets, 7535 bytes)
  14. pkts bytes target     prot opt in     out     source               destination
  16. Chain POSTROUTING (policy ACCEPT 128 packets, 11198 bytes)
  17. pkts bytes target     prot opt in     out     source               destination
  18. # ipvsadm
  19. IP Virtual Server version 1.2.1 (size=4096)
  20. Prot LocalAddress:Port Scheduler Flags
  21. -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  22. FWM  262 rr
  23. -> 10.0.1.31:0                  Masq    1      0          0
  24. -> 10.0.1.32:0                  Masq    1      0          0
  25. FWM  263 rr
  26. -> 10.0.1.36:0                  Masq    1      0          0
  27. #

这个流量会走我们的mynet这个overlay网络。