大规模场景下对Istio的性能优化

简介

当前istio下发xDS使用的是全量下发策略，也就是网格里的所有sidecar(envoy)，内存里都会有整个网格内所有的服务发现数据。这样的结果是，每个sidecar内存都会随着网格规模增长而增长。

Aeraki-mesh

aeraki-mesh项目下有一个子项目专门用来处理istio配置分发性能问题，我们找到该项目：
https://github.com/aeraki-mesh/lazyxds

从该项目的部署yaml中，我们知道它会在网格中增加两个组件：

• egress：充当类似网格模型中默认网关角色

• controller：用来分析并补全服务间的依赖关系

Egress

对应的配置文件为：lazyxds-egress.yaml
下面来一一查看该组件的组成部分

组件配置

apiVersion: apps/v1
kind: Deployment
metadata:
name: istio-egressgateway-lazyxds
namespace: istio-system
labels:
app: istio-egressgateway-lazyxds
istio: egressgateway
spec:
replicas: 1
selector:
matchLabels:
app: istio-egressgateway-lazyxds
istio: egressgateway
template:
metadata:
annotations:
sidecar.istio.io/discoveryAddress: istiod.istio-system.svc:15012
sidecar.istio.io/inject: "false"
labels:
app: istio-egressgateway-lazyxds
istio: egressgateway
spec:
containers:
- args:
......
image: docker.io/istio/proxyv2:1.10.0
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
......
volumeMounts:
- mountPath: /etc/istio/custom-bootstrap
name: custom-bootstrap-vo服务器托管网lume
......
volumes:
- configMap:
defaultMode: 420
name: lazyxds-als-bootstrap
name: custom-bootstrap-volume

由于配置太多，这里只挑选主要的部分，从上面可以看出，其实是启动一个istio proxy，该proxy的启动配置文件是使用的configmap挂载出来的。

启动配置

apiVersion: v1
kind: ConfigMap
metadata:
name: lazyxds-als-bootstrap
namespace: istio-system
data:
custom_bootstrap.json: |
{
"static_resources": {
"clusters": [{
"na服务器托管网me": "lazyxds-accesslog-service",
"type": "STRICT_DNS",
"connect_timeout": "1s",
"http2_protocol_options": {},
"dns_lookup_family": "V4_ONLY",
"load_assignment": {
"cluster_name": "lazyxds-accesslog-service",
"endpoints": [{
"lb_endpoints": [{
"endpoint": {
"address": {
"socket_address": {
"address": "lazyxds.istio-system",
"port_value": 8080
}
}
}
}]
}]
},
"respect_dns_ttl": true
}]
}
}

从上面配置可以知道：

• 定义了proxy组件代理的集群，该集群为"lazyxds-accesslog-service"

• 该集群对应的后端服务地址为"lazyxds.istio-system"，端口为8080
  这个后端就是lazyxds controller，后面细说

EnvoyFilter

从yaml文件我们看到，还定义了一个envoyfilter来修改proxy代理的流量配置

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: lazyxds-egress-als
namespace: istio-system
spec:
workloadSelector:
labels:
app: istio-egressgateway-lazyxds
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
access_log:
......
- name: envoy.access_loggers.http_grpc
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig
common_config:
log_name: http_envoy_accesslog
transport_api_version: "V3"
grpc_service:
envoy_grpc:
cluster_name: lazyxds-accesslog-service

从这个配置文件，可以看出在启动envoy时，会向其注入一个accesslog service，也就是envoy的日志收集器，而这个service就是lazyxds-accesslog-service

Controller

具体的lazy xds实现就是通过这个controller实现的

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: lazyxds
name: lazyxds
namespace: istio-system
spec:
replicas: 1
selector:
matchLabels:
app: lazyxds
template:
metadata:
labels:
app: lazyxds
spec:
serviceAccountName: lazyxds
containers:
- image: aeraki/lazyxds:latest
imagePullPolicy: Always
name: app
ports:
- containerPort: 8080
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app: lazyxds
name: lazyxds
namespace: istio-system
spec:
ports:
- name: grpc-als
port: 8080
protocol: TCP
selector:
app: lazyxds
type: ClusterIP

从配置可以看到，在egress环节我们知道了proxy的代理的后端地址为lazyxds.istio-system，刚好对应这里的controller。

并且我们还知道，envoy的访问日志最终会发送给这个controller来处理，而这就是实现增量下发envoy配置的关键之处，也就是解决istio性能的解决之法。

增量下发

Accesslog接口

要接受envoy的访问日志，必须实现envoy定义的接口：

typeAccessLogServiceServer interface{
// Envoy will connect and send StreamAccessLogsMessage messages forever. It does not expect any
// response to be sent as nothing would be done in the case of failure. The server should
// disconnect if it expects Envoy to reconnect. In the future we may decide to add a different
// API for "critical" access logs in which Envoy will buffer access logs for some period of time
// until it gets an ACK so it could then retry. This API is designed for high throughput with the
// expectation that it might be lossy.
StreamAccessLogs(AccessLogService_StreamAccessLogsServer) error
}

日志解析

lazyxds实现如下：

func(server *Server)StreamAccessLogs(logStream als.AccessLogService_StreamAccessLogsServer)error{
for{
data, err := logStream.Recv()
iferr != nil{
returnerr
}

httpLog := data.GetHttpLogs()
ifhttpLog != nil{
for_, entry := rangehttpLog.LogEntry {
server.log.V(4).Info("http log entry", "entry", entry)
fromIP := getDownstreamIP(entry)
iffromIP == ""{
continue
}

upstreamCluster := entry.CommonProperties.UpstreamCluster
svcID := utils.UpstreamCluster2ServiceID(upstreamCluster)

toIP := getUpstreamIP(entry)

iferr := server.handler.HandleAccess(fromIP, svcID, toIP); err != nil{
server.log.Error(err, "handle access error")
}
}
}
}
}

上面主要的逻辑就是解析envoy的访问日志，然后进行处理：

• lazy xds Controller 会对接收到的日志进行访问关系分析，然后把新的依赖关系表达到 sidecar CRD 中。

• 同时 Controller 还会更新 Egress 的规则：删除、更新或创建。

Slime

网易Slime方案与腾讯云Aeraki方案的思路一致
文档：https://cloudnative.to/blog/netease-slime/
github：https://github.com/slime-io/slime/tree/master/staging/src/slime.io/slime/modules/lazyload

https://cloud.tencent.com/developer/article/1922778

https://www.zhaohuabing.com/post/2018-09-25-istio-traffic-management-impl-intro/