Istio：熔断（针对DestinationRule）

而在Istio中，这个熔断指的是，当业务流量非常大的时候，为了保护系统整体可用性，Istio会根据连接池的熔断配置对一部分请求直接拒绝访问（即，返回503状态码）

1、连接池配置

下面，我们来模拟一下这个场景，将连接forecast的连接池配置为最多接收3个并发连接，如果超过3个，那么多余的就会触发熔断机制，即返回503错误码。

(1)首先部署一个用来测试的客户端pod

[root@master01 ~]# cd /root/cloud-native-istio/11_traffic-management/11.10
[root@master01 11.10]# kubectl apply -f fortio-deploy.yaml -n weather

(2)查看pod

[root@master01 11.10]# kubectl get po -n weather |grep fortio
fortio-deploy-956754996-pj2zw        2/2     Running   0          4m41s

(3)为了更好演示效果，将forecast-v2扩展到5个pod

[root@master01 11.10]# kubectl -n weather edit deploy forecast-v2
更改replicas: 5
...
...
spec:
  progressDeadlineSeconds: 600
  replicas: 5
  revisionHistoryLimit: 10
...
...

查看pod

[root@master01 11.10]# kubectl get po -n weather |grep forecast-v2
forecast-v2-895b74c49-dff8l          2/2     Running           0          68m
forecast-v2-895b74c49-fn7qk          2/2     Running           0          68m
forecast-v2-895b74c49-jld6f          2/2     Running           0          68m
forecast-v2-895b74c49-qdt9n          2/2     Running           0          2d2h
forecast-v2-895b74c49-zm97h          2/2     Running           0          79s

(4)配置熔断之前，我们先做一些准备工作：

需要将前面设置过的recommendation-dr和recommendation-route删除掉，否则会影响到实验效果。

[root@master01 11.10]# kubectl delete vs recommendation-route -n weather
[root@master01 11.10]# kubectl delete dr recommendation-dr -n weather

测试没有熔断策略时的效果，发起10个请求，其中-c指定并发数，-qps定义每秒查询数，如果为0表示不限制，-n指定请求总数：

[root@master01 11.10]# kubectl -n weather exec -it `kubectl get po -n weather |grep fortio|awk '{print $1}'` -- /usr/bin/fortio load -c 10 -qps 0 -n 10  http://forecast:3002/weather?locate=hangzhou

观察到后面的总结信息,10个请求全部都是200状态码。

...
...
Sockets used: 10 (for perfect keepalive, would be 10)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
10.0.80.175:3002: 10
Code 200 : 10 (100.0 %)
...
...

(5)配置熔断策略

[root@master01 11.10]# kubectl apply -f circuit-breaking.yaml -n weather

(6)查看策略

[root@master01 11.10]# kubectl get dr forecast-dr -n weather -o yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"forecast-dr","namespace":"weather"},"spec":{"host":"forecast","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"}],"trafficPolicy":{"connectionPool":{"http":{"http1MaxPendingRequests":5,"maxRequestsPerConnection":1},"tcp":{"maxConnections":3}},"outlierDetection":{"baseEjectionTime":"2m","consecutive5xxErrors":2,"interval":"10s","maxEjectionPercent":40}}}}
  creationTimestamp: "2023-11-09T01:59:24Z"
  generation: 3
  name: forecast-dr
  namespace: weather
  resourceVersion: "1217033"
  uid: 5ff2bc37-e4fd-404f-b76b-36be05401267
spec:
  host: forecast
  subsets:
  - labels:
      version: v1
    name: v1
  - labels:
      version: v2
    name: v2
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 5
        maxRequestsPerConnection: 1
      tcp:
        maxConnections: 3
    outlierDetection:
      baseEjectionTime: 2m
      consecutive5xxErrors: 2
      interval: 10s
      maxEjectionPercent: 40

说明：

maxConnections: 3 同一时间最多只能有3个tcp连接，超过就要拒绝。

http1MaxPendingRequests: 5 定义http请求pending状态的最大数值为5个。服务器的承载能力有限，如果并发量很大，那么就会排队，这个参数用来设置当队列已经满了，然后最多有几个请求被挂起，这个被挂起其实就是在队列里排队了，超过这个数值，直接拒绝。

maxRequestsPerConnection: 设置每个连接最大的http请求数量为1个，就是说一个tcp连接最多只能有一个http请求，第二个就不让连了。

(7)测试（并发数为10，总共进行100个请求）

[root@master01 11.10]# kubectl -n weather exec -it `kubectl get po -n weather |grep fortio|awk '{print $1}'` -- /usr/bin/fortio load -c 10 -qps 0 -n 100  http://forecast:3002/weather?locate=hangzhou

注意看，后面的统计数据。可以测试多次，大家会发现每次测试结果并不一样。

...
...
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
10.0.80.175:3002: 28
Code 200 : 81 (81.0 %)
Code 503 : 19 (19.0 %)
...
...

2、异常点检测

这个策略是这样的：如果请求的目标服务一旦返回了50x的错误码，那么Istio就会将该服务隔离，即不再将请求转发到该服务上。

下面我们针对从forecast-v2到recommendation的请求来做此策略。

(1)准备工作（部署forecast和recommendation相关资源）

[root@master01 ~]# cd /root/cloud-native-istio/11_traffic-management/11.11
[root@master01 11.11]# sed -i 's/replicas: 1/replicas: 6/' recommendation-all.yaml
[root@master01 11.11]# kubectl apply -f forecast-v2-deployment.yaml -n weather
[root@master01 11.11]# kubectl apply -f recommendation-all.yaml -n weather

(2)配置策略

[root@master01 11.11]# kubectl apply -f circuit-breaking.yaml -n weather

(3)查看策略

[root@master01 11.11]# kubectl get dr recommendation-dr -n weather -o yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"recommendation-dr","namespace":"weather"},"spec":{"host":"recommendation","subsets":[{"labels":{"version":"v1"},"name":"v1"}],"trafficPolicy":{"connectionPool":{"http":{"http1MaxPendingRequests":5,"maxRequestsPerConnection":1},"tcp":{"maxConnections":3}},"outlierDetection":{"baseEjectionTime":"2m","consecutive5xxErrors":2,"interval":"10s","maxEjectionPercent":50}}}}
  creationTimestamp: "2023-11-11T04:41:57Z"
  generation: 2
  name: recommendation-dr
  namespace: weather
  resourceVersion: "1218643"
  uid: 4edf354b-bc2e-488b-a427-d703f161f348
spec:
  host: recommendation
  subsets:
  - labels:
      version: v1
    name: v1
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 5
        maxRequestsPerConnection: 1
      tcp:
        maxConnections: 3
    outlierDetection:
      baseEjectionTime: 2m
      consecutive5xxErrors: 2
      interval: 10s
      maxEjectionPercent: 50

outlierDetection这部分配置就是异常点检测的配置。

这段配置的含义是：每10秒扫描一次recommendation服务查看访问其状态码，如果连续两次都是5xx的状态码，则要剔除50%的异常实例（比如一共有6个pod，出现异常的实例有2个，那么将会剔除1个pod）

评论区