java性能調(diào)優(yōu)記錄（限流）

keithyau 發(fā)布于2019-08-16 14:31 / 2695人閱讀

摘要：在中，一個(gè)線程可以處理多個(gè)，但是一個(gè)只能綁定到一個(gè)，這是基于線程安全和同步考慮而設(shè)計(jì)的。線程阻塞再次進(jìn)行壓力測(cè)試，結(jié)果如下最終結(jié)果沒有任何提升，利用率依然不超過，也還是在單個(gè)利用率最高不超過，說明這次的瓶頸不是。但是其中出現(xiàn)了軟中斷。

1. 問題

spring-cloud-gateway 網(wǎng)關(guān)新增了一個(gè)限流功能，使用的是模塊自帶的限流過濾器 RequestRateLimiterGatewayFilterFactory，基于令牌桶算法，通過 redis 實(shí)現(xiàn)。

其原理是 redis 中針對(duì)每個(gè)限流要素（比如針對(duì)接口限流），保存 2 個(gè) key：tokenKey（令牌數(shù)量），timeKey（調(diào)用時(shí)間）。每次接口調(diào)用時(shí)，更新 tokenKey 的值為：原先的值 + （當(dāng)前時(shí)間 - 原先時(shí)間）* 加入令牌的速度，如果新的 tokenKey 的值大于 1，那么允許調(diào)用，否則不允許；同時(shí)更新 redis 中 tokenKey，timeKey 的值。整個(gè)過程通過 lua 腳本實(shí)現(xiàn)。

在加入限流功能之前，500 客戶端并發(fā)訪問，tps 為 6800 req/s，50% 時(shí)延為 70ms；加入限流功能之后，tps 為 2300 req/s，50% 時(shí)延為 205ms，同時(shí)，原先 cpu 占用率幾乎 600%（6 核）變成不到 400%（cpu 跑不滿了）。

2. 排查和解決過程

2.1 單個(gè) CPU 跑滿

查看單個(gè)線程的 cpu 占用：

[root@auth-service imf2]# top -Hp 29360
top - 15:16:27 up 102 days, 18:04,  1 user,  load average: 1.61, 0.72, 0.34
Threads: 122 total,   9 running, 113 sleeping,   0 stopped,   0 zombie
%Cpu(s): 42.0 us,  7.0 sy,  0.0 ni, 49.0 id,  0.0 wa,  0.0 hi,  2.0 si,  0.0 st
KiB Mem :  7678384 total,   126844 free,  3426148 used,  4125392 buff/cache
KiB Swap:  6291452 total,  2212552 free,  4078900 used.  3347956 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
29415 root      20   0 6964708   1.1g  14216 R 97.9 15.1   3:01.65 java
29392 root      20   0 6964708   1.1g  14216 R 27.0 15.1   0:45.42 java
29391 root      20   0 6964708   1.1g  14216 R 24.8 15.1   0:43.95 java
29387 root      20   0 6964708   1.1g  14216 R 23.8 15.1   0:46.38 java
29388 root      20   0 6964708   1.1g  14216 R 23.4 15.1   0:48.21 java
29390 root      20   0 6964708   1.1g  14216 R 23.0 15.1   0:45.93 java
29389 root      20   0 6964708   1.1g  14216 R 22.3 15.1   0:44.36 java

線程 29415 幾乎跑滿了 cpu，查看是什么線程：

[root@auth-service imf2]# printf "%x
" 29415
72e7
[root@auth-service imf2]# jstack 29360 | grep 72e7
"lettuce-nioEventLoop-4-1" #40 daemon prio=5 os_prio=0 tid=0x00007f604cc92000 nid=0x72e7 runnable [0x00007f606ce90000]

果然是操作 redis 的線程，和預(yù)期一致。

查看 redis：cpu 占用率不超過 15%，沒有 10ms 以上的慢查詢。應(yīng)該不會(huì)是 redis 的問題。

查看線程棧信息：

通過以下腳本每秒記錄一次 jstack：

[root@eureka2 jstack]# cat jstack.sh
#!/bin/sh
i=0
while [ $i -lt 30 ]; do
/bin/sleep 1
i=`expr $i + 1`
jstack 29360 > "$i".txt
done

查看 lettuce 線程主要執(zhí)行哪些函數(shù)：

"lettuce-nioEventLoop-4-1" #36 daemon prio=5 os_prio=0 tid=0x00007f1eb07ab800 nid=0x4476 runnable [0x00007f1eec8fb000]
   java.lang.Thread.State: RUNNABLE
        at sun.misc.URLClassPath$Loader.findResource(URLClassPath.java:715)
        at sun.misc.URLClassPath.findResource(URLClassPath.java:215)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:569)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:567)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findResource(URLClassLoader.java:566)
        at org.springframework.boot.loader.LaunchedURLClassLoader.findResource(LaunchedURLClassLoader.java:57)
        at java.lang.ClassLoader.getResource(ClassLoader.java:1096)
        at org.springframework.core.io.ClassPathResource.resolveURL(ClassPathResource.java:155)
        at org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:193)
        at org.springframework.core.io.AbstractFileResolvingResource.lastModified(AbstractFileResolvingResource.java:220)
        at org.springframework.scripting.support.ResourceScriptSource.retrieveLastModifiedTime(ResourceScriptSource.java:119)
        at org.springframework.scripting.support.ResourceScriptSource.isModified(ResourceScriptSource.java:109)
        - locked <0x000000008c074d00> (a java.lang.Object)
        at org.springframework.data.redis.core.script.DefaultRedisScript.getSha1(DefaultRedisScript.java:89)
        - locked <0x000000008c074c10> (a java.lang.Object)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor.eval(DefaultReactiveScriptExecutor.java:113)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor.lambda$execute$0(DefaultReactiveScriptExecutor.java:105)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor$$Lambda$1268/1889039573.doInRedis(Unknown Source)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor.lambda$execute$6(DefaultReactiveScriptExecutor.java:167)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor$$Lambda$1269/1954779522.get(Unknown Source)
        at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46)

可知該線程主要在執(zhí)行 ReactiveRedisTemplate 類的 execute(RedisScript script, List keys, List args) 方法，即運(yùn)行 lua 腳本。

猜想：既然是因?yàn)?lettuce-nioEventLoop 線程跑滿了 cpu，那么通過創(chuàng)建多個(gè) lettuce-nioEventLoop 線程，以充分利用多核的特點(diǎn)，是否可以解決呢？

以下為源碼分析階段：

// 1. RedisConnectionFactory bean 的創(chuàng)建依賴 ClientResources
@Bean
@ConditionalOnMissingBean(RedisConnectionFactory.class)
public LettuceConnectionFactory redisConnectionFactory(
    ClientResources clientResources) throws UnknownHostException {
    LettuceClientConfiguration clientConfig = getLettuceClientConfiguration(
        clientResources, this.properties.getLettuce().getPool());
    return createLettuceConnectionFactory(clientConfig);
}

// 2. ClientResources bean 的創(chuàng)建如下
@Bean(destroyMethod = "shutdown")
@ConditionalOnMissingBean(ClientResources.class)
public DefaultClientResources lettuceClientResources() {
    return DefaultClientResources.create();
}

public static DefaultClientResources create() {
    return builder().build();
}

// 3. 創(chuàng)建 EventLoopGroupProvider 對(duì)象
protected DefaultClientResources(Builder builder) {
    this.builder = builder;

    // 默認(rèn)為 null，執(zhí)行這塊代碼
    if (builder.eventLoopGroupProvider == null) {
        // 設(shè)置處理 redis 連接的線程數(shù)：默認(rèn)為
        //    Math.max(1,
        //        SystemPropertyUtil.getInt("io.netty.eventLoopThreads",
        //                Math.max(MIN_IO_THREADS, Runtime.getRuntime().availableProcessors())));
        // 針對(duì)多核處理器，該值一般等于 cpu 的核的數(shù)量
        int ioThreadPoolSize = builder.ioThreadPoolSize;

        if (ioThreadPoolSize < MIN_IO_THREADS) {
            logger.info("ioThreadPoolSize is less than {} ({}), setting to: {}", MIN_IO_THREADS, ioThreadPoolSize,
                    MIN_IO_THREADS);
            ioThreadPoolSize = MIN_IO_THREADS;
        }

        this.sharedEventLoopGroupProvider = false;
        // 創(chuàng)建 EventLoopGroupProvider 對(duì)象
        this.eventLoopGroupProvider = new DefaultEventLoopGroupProvider(ioThreadPoolSize);

    } else {
        this.sharedEventLoopGroupProvider = true;
        this.eventLoopGroupProvider = builder.eventLoopGroupProvider;
    }

    // 以下代碼省略 ...
}

// 4. 通過 EventLoopGroupProvider 創(chuàng)建 EventExecutorGroup 對(duì)象
public static  EventExecutorGroup createEventLoopGroup(Class type, int numberOfThreads) {

    if (DefaultEventExecutorGroup.class.equals(type)) {
        return new DefaultEventExecutorGroup(numberOfThreads, new DefaultThreadFactory("lettuce-eventExecutorLoop", true));
    }

    // 我們采用的是 Nio 模式，會(huì)執(zhí)行這個(gè)分支
    if (NioEventLoopGroup.class.equals(type)) {
        return new NioEventLoopGroup(numberOfThreads, new DefaultThreadFactory("lettuce-nioEventLoop", true));
    }

    if (EpollProvider.isAvailable() && EpollProvider.isEventLoopGroup(type)) {
        return EpollProvider.newEventLoopGroup(numberOfThreads, new DefaultThreadFactory("lettuce-epollEventLoop", true));
    }

    if (KqueueProvider.isAvailable() && KqueueProvider.isEventLoopGroup(type)) {
        return KqueueProvider.newEventLoopGroup(numberOfThreads, new DefaultThreadFactory("lettuce-kqueueEventLoop", true));
    }

    throw new IllegalArgumentException(String.format("Type %s not supported", type.getName()));
}

// 5. NioEventLoopGroup 繼承了 MultithreadEventLoopGroup；
//    創(chuàng)建了多個(gè) NioEventLoop；
//      每個(gè) NioEventLoop 都是單線程；
//      每個(gè) NioEventLoop 都可以處理多個(gè)連接。
public class NioEventLoopGroup extends MultithreadEventLoopGroup { ... }
public abstract class MultithreadEventLoopGroup extends MultithreadEventExecutorGroup implements EventLoopGroup { ... }
public final class NioEventLoop extends SingleThreadEventLoop { ... }

以上分析可知，默認(rèn)創(chuàng)建的 RedisConnectionFactory bean 其實(shí)是支持多線程的，但通過 jstack 等方式查看 lettuce-nioEventLoop 線程卻只有一個(gè)。

[root@ ~]# ss | grep 6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:36184                ::ffff:10.201.0.30:6379

查看 redis 連接，發(fā)現(xiàn)只有一個(gè)。在 Netty 中，一個(gè) EventLoop 線程可以處理多個(gè) Channel，但是一個(gè) Channel 只能綁定到一個(gè) EventLoop，這是基于線程安全和同步考慮而設(shè)計(jì)的。這解釋了為什么只有一個(gè) lettuce-nioEventLoop。

下面繼續(xù)分析為什么會(huì)只有一個(gè)連接呢？繼續(xù)源碼分析：

// 1. 創(chuàng)建 RedisConnectionFactory bean
@Bean
@ConditionalOnMissingBean(RedisConnectionFactory.class)
public LettuceConnectionFactory redisConnectionFactory(
    ClientResources clientResources) throws UnknownHostException {
    LettuceClientConfiguration clientConfig = getLettuceClientConfiguration(
        clientResources, this.properties.getLettuce().getPool());
    return createLettuceConnectionFactory(clientConfig);
}

// 2. 查看 createLettuceConnectionFactory(clientConfig) 方法
private LettuceConnectionFactory createLettuceConnectionFactory(
    LettuceClientConfiguration clientConfiguration) {
    if (getSentinelConfig() != null) {
        return new LettuceConnectionFactory(getSentinelConfig(), clientConfiguration);
    }
    if (getClusterConfiguration() != null) {
        return new LettuceConnectionFactory(getClusterConfiguration(),
                                            clientConfiguration);
    }
    // 沒有哨兵模式，沒有集群，執(zhí)行這塊代碼
    return new LettuceConnectionFactory(getStandaloneConfig(), clientConfiguration);
}

// 3. 獲取 redis 連接
private boolean shareNativeConnection = true;
public LettuceReactiveRedisConnection getReactiveConnection() {
    // 默認(rèn)為 true
    return getShareNativeConnection()
        ? new LettuceReactiveRedisConnection(getSharedReactiveConnection(), reactiveConnectionProvider)
        : new LettuceReactiveRedisConnection(reactiveConnectionProvider);
}

LettuceReactiveRedisConnection(StatefulConnection sharedConnection, LettuceConnectionProvider connectionProvider) {
    Assert.notNull(sharedConnection, "Shared StatefulConnection must not be null!");
    Assert.notNull(connectionProvider, "LettuceConnectionProvider must not be null!");
    this.dedicatedConnection = new AsyncConnect(connectionProvider, StatefulConnection.class);
    this.pubSubConnection = new AsyncConnect(connectionProvider, StatefulRedisPubSubConnection.class);
    // 包裝 sharedConnection
    this.sharedConnection = Mono.just(sharedConnection);
}

protected Mono> getConnection() {
    // 直接返回 sharedConnection
    if (sharedConnection != null) {
        return sharedConnection;
    }

    return getDedicatedConnection();
}

// 4. shareNativeConnection 是怎么來的
protected StatefulConnection getSharedReactiveConnection() {
    return shareNativeConnection ? getOrCreateSharedReactiveConnection().getConnection() : null;
}

private SharedConnection getOrCreateSharedReactiveConnection() {
    synchronized (this.connectionMonitor) {
        if (this.reactiveConnection == null) {
            this.reactiveConnection = new SharedConnection<>(reactiveConnectionProvider, true);
        }
        return this.reactiveConnection;
    }
}

StatefulConnection getConnection() {
    synchronized (this.connectionMonitor) {
        // 第一次通過 getNativeConnection() 獲取連接；之后直接返回該連接
        if (this.connection == null) {
            this.connection = getNativeConnection();
        }
        if (getValidateConnection()) {
            validateConnection();
        }
        return this.connection;
    }
}

分析以上源碼，關(guān)鍵就在于 shareNativeConnection 默認(rèn)為 true，導(dǎo)致只有一個(gè)連接。

更改 shareNativeConnection 的值為 true，并開啟 lettuce 連接池，最大連接數(shù)設(shè)置為 6；再次測(cè)試，

[root@eureka2 jstack]# ss | grep 6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:48937                ::ffff:10.201.0.30:6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:35842                ::ffff:10.201.0.30:6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:48932                ::ffff:10.201.0.30:6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:48930                ::ffff:10.201.0.30:6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:48936                ::ffff:10.201.0.30:6379
tcp    ESTAB      0      0      ::ffff:10.201.0.27:48934                ::ffff:10.201.0.30:6379

[root@eureka2 jstack]# jstack 23080 | grep lettuce-epollEventLoop
"lettuce-epollEventLoop-4-6" #69 daemon prio=5 os_prio=0 tid=0x00007fcfa4012000 nid=0x5af2 runnable [0x00007fcfa81ef000]
"lettuce-epollEventLoop-4-5" #67 daemon prio=5 os_prio=0 tid=0x00007fcf94003800 nid=0x5af0 runnable [0x00007fcfa83f1000]
"lettuce-epollEventLoop-4-4" #60 daemon prio=5 os_prio=0 tid=0x00007fcfa0003000 nid=0x5ae9 runnable [0x00007fcfa8af8000]
"lettuce-epollEventLoop-4-3" #59 daemon prio=5 os_prio=0 tid=0x00007fcfb00b8000 nid=0x5ae8 runnable [0x00007fcfa8bf9000]
"lettuce-epollEventLoop-4-2" #58 daemon prio=5 os_prio=0 tid=0x00007fcf6c00f000 nid=0x5ae7 runnable [0x00007fcfa8cfa000]
"lettuce-epollEventLoop-4-1" #43 daemon prio=5 os_prio=0 tid=0x00007fcfac248800 nid=0x5a64 runnable [0x00007fd00c2b9000]

可以看到已經(jīng)建立了 6 個(gè) redis 連接，并且創(chuàng)建了 6 個(gè) eventLoop 線程。

2.2 線程阻塞

再次進(jìn)行壓力測(cè)試，結(jié)果如下：

[root@hystrix-dashboard wrk]# wrk -t 10 -c 500 -d 30s --latency -T 3s -s post-test.lua "http://10.201.0.27:8888/api/v1/json"
Running 30s test @ http://10.201.0.27:8888/api/v1/json
  10 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   215.83ms  104.38ms   1.00s    75.76%
    Req/Sec   234.56     49.87   434.00     71.45%
  Latency Distribution
     50%  210.63ms
     75%  281.30ms
     90%  336.78ms
     99%  519.51ms
  69527 requests in 30.04s, 22.43MB read
Requests/sec:   2314.14
Transfer/sec:    764.53KB
[root@eureka2 jstack]# top -Hp 23080
top - 10:08:10 up 162 days, 12:31,  2 users,  load average: 2.92, 1.19, 0.53
Threads: 563 total,   9 running, 554 sleeping,   0 stopped,   0 zombie
%Cpu(s): 50.5 us, 10.2 sy,  0.0 ni, 36.2 id,  0.1 wa,  0.0 hi,  2.9 si,  0.0 st
KiB Mem :  7677696 total,   215924 free,  3308248 used,  4153524 buff/cache
KiB Swap:  6291452 total,  6291452 free,        0 used.  3468352 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
23280 root      20   0 7418804   1.3g   7404 R 42.7 17.8   0:54.75 java
23272 root      20   0 7418804   1.3g   7404 S 31.1 17.8   0:44.63 java
23273 root      20   0 7418804   1.3g   7404 S 31.1 17.8   0:44.45 java
23271 root      20   0 7418804   1.3g   7404 R 30.8 17.8   0:44.63 java
23282 root      20   0 7418804   1.3g   7404 S 30.5 17.8   0:44.96 java
23119 root      20   0 7418804   1.3g   7404 R 24.8 17.8   1:27.30 java
23133 root      20   0 7418804   1.3g   7404 R 23.8 17.8   1:29.55 java
23123 root      20   0 7418804   1.3g   7404 S 23.5 17.8   1:28.98 java
23138 root      20   0 7418804   1.3g   7404 S 23.5 17.8   1:44.19 java
23124 root      20   0 7418804   1.3g   7404 R 22.8 17.8   1:32.21 java
23139 root      20   0 7418804   1.3g   7404 R 22.5 17.8   1:29.49 java

最終結(jié)果沒有任何提升，cpu 利用率依然不超過 400%，tps 也還是在 2300 request/s；單個(gè) cpu 利用率最高不超過 50%，說明這次的瓶頸不是 cpu。

通過 jstack 查看線程狀態(tài)，

"lettuce-epollEventLoop-4-3" #59 daemon prio=5 os_prio=0 tid=0x00007fcfb00b8000 nid=0x5ae8 waiting for monitor entry [0x00007fcfa8bf8000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.springframework.data.redis.core.script.DefaultRedisScript.getSha1(DefaultRedisScript.java:88)
        - waiting to lock <0x000000008c1da690> (a java.lang.Object)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor.eval(DefaultReactiveScriptExecutor.java:113)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor.lambda$execute$0(DefaultReactiveScriptExecutor.java:105)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor$$Lambda$1317/1912229933.doInRedis(Unknown Source)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor.lambda$execute$6(DefaultReactiveScriptExecutor.java:167)
        at org.springframework.data.redis.core.script.DefaultReactiveScriptExecutor$$Lambda$1318/1719274268.get(Unknown Source)
        at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46)
        at reactor.core.publisher.FluxDoFinally.subscribe(FluxDoFinally.java:73)
        at reactor.core.publisher.FluxOnErrorResume.subscribe(FluxOnErrorResume.java:47)
        at reactor.core.publisher.MonoReduceSeed.subscribe(MonoReduceSeed.java:65)
        at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
        at reactor.core.publisher.Mono.subscribe(Mono.java:3608)
        at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:169)
        at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:53)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476)
        at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476)
        at reactor.core.publisher.MonoProcessor.subscribe(MonoProcessor.java:457)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476)
        at reactor.core.publisher.MonoHasElement$HasElementSubscriber.onNext(MonoHasElement.java:74)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476)
        at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389)
        at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:123)
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
        at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
        at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238)
        at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92)
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
        at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
        at io.lettuce.core.RedisPublisher$RedisSubscription.onNext(RedisPublisher.java:270)
        at io.lettuce.core.RedisPublisher$SubscriptionCommand.complete(RedisPublisher.java:754)
        at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:59)
        at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:646)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:604)
        at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:556)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

有 4 個(gè) lettuce-epollEventLoop 線程都處于 BLOCKED 狀態(tài)，繼續(xù)查看源碼：

public class DefaultRedisScript implements RedisScript, InitializingBean {
    private @Nullable ScriptSource scriptSource;
    private @Nullable String sha1;
    private @Nullable Class resultType;

    public String getSha1() {
        // 1. 線程需要先獲取 shaModifiedMonitor 鎖
        synchronized (shaModifiedMonitor) {
            // 第一次調(diào)用時(shí)或者 lua 腳本文件被修改時(shí)，需要重新計(jì)算 sha1 的值
            // 否則直接返回sha1
            if (sha1 == null || scriptSource.isModified()) {
                this.sha1 = DigestUtils.sha1DigestAsHex(getScriptAsString());
            }
            return sha1;
        }
    }

    public String getScriptAsString() {
        try {
            return scriptSource.getScriptAsString();
        } catch (IOException e) {
            throw new ScriptingException("Error reading script text", e);
        }
    }
}

public class ResourceScriptSource implements ScriptSource {
    // 只有第一次調(diào)用或者 lua 腳本文件被修改時(shí)，才會(huì)執(zhí)行這個(gè)方法
    @Override
    public String getScriptAsString() throws IOException {
        synchronized (this.lastModifiedMonitor) {
            this.lastModified = retrieveLastModifiedTime();
        }
        Reader reader = this.resource.getReader();
        return FileCopyUtils.copyToString(reader);
    }

    @Override
    public boolean isModified() {
        // 2. 每次都需要判斷 lua 腳本是否被修改
        // 線程需要再獲取 lastModifiedMonitor 鎖
        synchronized (this.lastModifiedMonitor) {
            return (this.lastModified < 0 || retrieveLastModifiedTime() > this.lastModified);
        }
    }
}

對(duì)于限流操作，重要性并沒有那么高，而且計(jì)算接口調(diào)用次數(shù)的 lua 腳本，一般也不會(huì)經(jīng)常改動(dòng)，所以沒必要獲取 sha1 的值的時(shí)候都查看下腳本是否有改動(dòng)；如果偶爾改動(dòng)的話，可以通過新增一個(gè)刷新接口，在改動(dòng)腳本文件后通過手動(dòng)刷新接口來改變 sha1 的值。

所以這里，可以把同步操作去掉；我改成了這樣：

public class CustomRedisScript extends DefaultRedisScript {
    private @Nullable String sha1;

    CustomRedisScript(ScriptSource scriptSource, Class resultType) {
        setScriptSource(scriptSource);
        setResultType(resultType);
        this.sha1 = DigestUtils.sha1DigestAsHex(getScriptAsString());
    }

    @Override
    public String getSha1() {
        return sha1;
    }
}

2.3 cpu 出現(xiàn)大量軟中斷

繼續(xù)測(cè)試，結(jié)果如下：

[root@hystrix-dashboard wrk]# wrk -t 10 -c 500 -d 30s -T 3s -s post-test.lua --latency "http://10.201.0.27:8888/api/v1/json"
Running 30s test @ http://10.201.0.27:8888/api/v1/json
  10 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   155.60ms  110.40ms   1.07s    67.68%
    Req/Sec   342.90     64.88   570.00     70.35%
  Latency Distribution
     50%  139.14ms
     75%  211.03ms
     90%  299.74ms
     99%  507.03ms
  102462 requests in 30.02s, 33.15MB read
Requests/sec:   3413.13
Transfer/sec:      1.10MB

cpu 利用率 500% 左右，tps 達(dá)到了 3400 req/s，性能大幅度提升。查看 cpu 狀態(tài)：

[root@eureka2 imf2]# top -Hp 19021
top - 16:24:09 up 163 days, 18:47,  2 users,  load average: 3.03, 1.08, 0.47
Threads: 857 total,   7 running, 850 sleeping,   0 stopped,   0 zombie
%Cpu0  : 60.2 us, 10.0 sy,  0.0 ni,  4.3 id,  0.0 wa,  0.0 hi, 25.4 si,  0.0 st
%Cpu1  : 64.6 us, 16.3 sy,  0.0 ni, 19.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  : 65.7 us, 15.8 sy,  0.0 ni, 18.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  : 54.5 us, 15.8 sy,  0.0 ni, 29.5 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  : 55.0 us, 17.8 sy,  0.0 ni, 27.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  : 53.2 us, 16.4 sy,  0.0 ni, 30.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7677696 total,   174164 free,  3061892 used,  4441640 buff/cache
KiB Swap:  6291452 total,  6291452 free,        0 used.  3687692 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
19075 root      20   0 7722156   1.2g  14488 S 41.4 15.9   0:55.71 java
19363 root      20   0 7722156   1.2g  14488 R 40.1 15.9   0:41.33 java
19071 root      20   0 7722156   1.2g  14488 R 37.1 15.9   0:56.38 java
19060 root      20   0 7722156   1.2g  14488 S 35.4 15.9   0:52.74 java
19073 root      20   0 7722156   1.2g  14488 R 35.1 15.9   0:55.83 java

cpu0 利用率達(dá)到了 95.7%，幾乎跑滿。但是其中出現(xiàn)了 si（軟中斷）: 25.4%。

查看軟中斷類型：

[root@eureka2 imf2]# watch -d -n 1 "cat /proc/softirqs"
                    CPU0       CPU1      CPU2       CPU3    CPU4       CPU5
          HI:          0          0          0          0          0          0
       TIMER: 1629142082  990710808  852299786  606344269  586896512  566624764
      NET_TX:     291570     833710      9616       5295    5358    2012064
      NET_RX: 2563401537   32502894   31370533    6886869    6530120    6490002
       BLOCK:       18130       1681   41404591    8751054    8695636    8763338
BLOCK_IOPOLL:          0          0          0          0          0          0
     TASKLET:   39225643          0          0        817      17304    2516988
       SCHED:  782335782  442142733  378856479  248794679  238417109  259695794
     HRTIMER:          0          0          0          0          0          0
         RCU:  690827224  504025610  464412234  246695846  254062933  248859132

其中 NET_RX，CPU0 的中斷次數(shù)遠(yuǎn)遠(yuǎn)大于其他 CPU，初步判斷是網(wǎng)卡問題。

我這邊網(wǎng)卡是 ens32，查看網(wǎng)卡的中斷號(hào)：

[root@eureka2 imf2]# cat /proc/interrupts | grep ens
 18: 2524017495          0          0          0          0          7   IO-APIC-fasteoi   ens32
[root@eureka2 imf2]# cat /proc/irq/18/smp_affinity
01
[root@eureka2 imf2]# cat /proc/irq/18/smp_affinity_list
0

網(wǎng)卡的中斷配置到了 CPU0。（01：表示 cpu0，02：cpu1，04：cpu2，08：cpu3，10：cpu4，20：cpu5）

smp_affinity：16 進(jìn)制；smp_affinity_list：配置到了哪些 cpu。

查看網(wǎng)卡隊(duì)列模式：

[root@eureka2 ~]# lspci -vvv
02:00.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)
    Subsystem: VMware PRO/1000 MT Single Port Adapter
    Physical Slot: 32
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- SERR- 
由于是單隊(duì)列模式，所以通過修改 /proc/irq/18/smp_affinity 的值不能生效。
可以通過 RPS/RFS 在軟件層面模擬多隊(duì)列網(wǎng)卡功能。
[root@eureka2 ~]# echo 3e > /sys/class/net/ens32/queues/rx-0/rps_cpus
[root@eureka2 rx-0]# sysctl net.core.rps_sock_flow_entries=32768
[root@eureka2 rx-0]# echo 32768 > /sys/class/net/ens32/queues/rx-0/rps_flow_cnt
/sys/class/net/ens32/queues/rx-0/rps_cpus: 1e，設(shè)置模擬網(wǎng)卡中斷分配到 cpu1-5 上。
繼續(xù)測(cè)試，
[root@hystrix-dashboard wrk]# wrk -t 10 -c 500 -d 30s -T 3s -s post-test.lua --latency "http://10.201.0.27:8888/api/v1/json"
Running 30s test @ http://10.201.0.27:8888/api/v1/json
  10 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   146.75ms  108.45ms   1.01s    65.53%
    Req/Sec   367.80     64.55   575.00     67.93%
  Latency Distribution
     50%  130.93ms
     75%  200.72ms
     90%  290.32ms
     99%  493.84ms
  109922 requests in 30.02s, 35.56MB read
Requests/sec:   3661.21
Transfer/sec:      1.18MB
[root@eureka2 rx-0]# top -Hp 19021
top - 09:39:49 up 164 days, 12:03,  1 user,  load average: 2.76, 2.02, 1.22
Threads: 559 total,   9 running, 550 sleeping,   0 stopped,   0 zombie
%Cpu0  : 55.1 us, 13.0 sy,  0.0 ni, 17.5 id,  0.0 wa,  0.0 hi, 14.4 si,  0.0 st
%Cpu1  : 60.1 us, 14.0 sy,  0.0 ni, 22.5 id,  0.0 wa,  0.0 hi,  3.4 si,  0.0 st
%Cpu2  : 59.5 us, 14.3 sy,  0.0 ni, 22.4 id,  0.0 wa,  0.0 hi,  3.7 si,  0.0 st
%Cpu3  : 58.6 us, 15.2 sy,  0.0 ni, 22.2 id,  0.0 wa,  0.0 hi,  4.0 si,  0.0 st
%Cpu4  : 59.1 us, 14.8 sy,  0.0 ni, 22.7 id,  0.0 wa,  0.0 hi,  3.4 si,  0.0 st
%Cpu5  : 57.7 us, 16.2 sy,  0.0 ni, 23.0 id,  0.0 wa,  0.0 hi,  3.1 si,  0.0 st
KiB Mem :  7677696 total,   373940 free,  3217180 used,  4086576 buff/cache
KiB Swap:  6291452 total,  6291452 free,        0 used.  3533812 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
19060 root      20   0 7415812   1.2g  13384 S 40.7 16.7   3:23.05 java
19073 root      20   0 7415812   1.2g  13384 R 40.1 16.7   3:20.56 java
19365 root      20   0 7415812   1.2g  13384 R 40.1 16.7   2:36.65 java
可以看到軟中斷也分配到了 cpu1-5 上；至于為什么還是 cpu0 上軟中斷比例最高，猜測(cè)是因?yàn)檫€有一些其他中斷并且默認(rèn)配置在 cpu0 上？
同時(shí)，tps 也從 3400 -> 3600，提升不大。
2.4 增加 redis 連接
經(jīng)過以上修改，cup 利用率還是不超過 500%，說明在某些地方還是存在瓶頸。
嘗試修改了下 lettuce 連接池，
spring:
  redis:
    database: x
    host: x.x.x.x
    port: 6379
    lettuce:
      pool:
        max-active: 18
        min-idle: 1
        max-idle: 18
主要是把 max-active 參數(shù) 6 增大到了 18，繼續(xù)測(cè)試：
[root@hystrix-dashboard wrk]# wrk -t 10 -c 500 -d 120s -T 3s -s post-test.lua --latency "http://10.201.0.27:8888/api/v1/json"
Running 2m test @ http://10.201.0.27:8888/api/v1/json
  10 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   117.66ms   96.72ms   1.34s    86.48%
    Req/Sec   485.42     90.41   790.00     70.80%
  Latency Distribution
     50%   90.04ms
     75%  156.01ms
     90%  243.63ms
     99%  464.04ms
  578298 requests in 2.00m, 187.01MB read
Requests/sec:   4815.57
Transfer/sec:      1.56MB
6 核 cpu 幾乎跑滿，同時(shí) tps 也從 3600 -> 4800，提升明顯！
這說明之前的瓶頸出在 redis 連接上，那么如何判斷 tcp 連接是瓶頸呢？（嘗試通過 ss、netstat 等命令查看 tcp 發(fā)送緩沖區(qū)、接收緩沖區(qū)、半連接隊(duì)列、全連接隊(duì)列等，未發(fā)現(xiàn)問題。先放著，以后在研究）

云服務(wù)器 GPU云服務(wù)器 java性能調(diào)優(yōu) 性能調(diào)優(yōu) webrtc性能調(diào)優(yōu) tomcat性能調(diào)優(yōu)

文章版權(quán)歸作者所有，未經(jīng)允許請(qǐng)勿轉(zhuǎn)載,若此文章存在違規(guī)行為，您可以聯(lián)系管理員刪除。

轉(zhuǎn)載請(qǐng)注明本文地址：http://systransis.cn/yun/73051.html

發(fā)表評(píng)論

登陸后可評(píng)論

0條評(píng)論

keithyau

男|高級(jí)講師

我要關(guān)注我要私信

TA的文章

《十萬字Java入門練習(xí)100例》1-10例——紙上得來終覺淺,絕知此事要躬行

閱讀 2470·2021-10-09 09:44
如何通過ip連接主機(jī)-如何利用ip地址連接打印機(jī)？

閱讀 3842·2021-09-22 15:43
GigsGigsCloud：美國VPS限時(shí)2折，洛杉磯CN2+聯(lián)通9929線路，首月$6/起

閱讀 2955·2021-09-02 09:47
Vultr機(jī)房測(cè)評(píng) - Vultr韓國首爾Seoul機(jī)房綜合速度和線路去程回程測(cè)試

閱讀 2596·2021-08-12 13:29
微信小程序 + 騰訊地圖SDK 實(shí)現(xiàn)路線規(guī)劃

閱讀 3902·2019-08-30 15:43
個(gè)人Web前端開發(fā)切圖PS設(shè)置

閱讀 1704·2019-08-30 13:06
BFC-塊格式化上下文（block formatting context）

閱讀 2212·2019-08-29 16:07
css預(yù)編譯--sass基礎(chǔ)篇

閱讀 2775·2019-08-29 15:23

成人国产在线小视频_日韩寡妇人妻调教在线播放_色成人www永久在线观看_2018国产精品久久_亚洲欧美高清在线30p_亚洲少妇综合一区_黄色在线播放国产_亚洲另类技巧小说校园_国产主播xx日韩_a级毛片在线免费

資訊專欄INFORMATION COLUMN

上云采購季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺(tái)、長期優(yōu)惠，快來選購！

java性能調(diào)優(yōu)記錄（限流）

2.1 單個(gè) CPU 跑滿

2.2 線程阻塞

2.3 cpu 出現(xiàn)大量軟中斷

2.4 增加 redis 連接

相關(guān)文章

Java學(xué)習(xí)路線總結(jié)，搬磚工逆襲Java架構(gòu)師（全網(wǎng)最強(qiáng)）

Javag工程師成神之路（2019正式版）

【推薦】最新200篇：技術(shù)文章整理

【推薦】最新200篇：技術(shù)文章整理

發(fā)表評(píng)論

0條評(píng)論

keithyau

男|高級(jí)講師

TA的文章

《十萬字Java入門練習(xí)100例》1-10例——紙上得來終覺淺,絕知此事要躬行

如何通過ip連接主機(jī)-如何利用ip地址連接打印機(jī)？

GigsGigsCloud：美國VPS限時(shí)2折，洛杉磯CN2+聯(lián)通9929線路，首月$6/起

Vultr機(jī)房測(cè)評(píng) - Vultr韓國首爾Seoul機(jī)房綜合速度和線路去程回程測(cè)試

微信小程序 + 騰訊地圖SDK 實(shí)現(xiàn)路線規(guī)劃

個(gè)人Web前端開發(fā)切圖PS設(shè)置

BFC-塊格式化上下文（block formatting context）

css預(yù)編譯--sass基礎(chǔ)篇

最新活動(dòng)

資訊專欄INFORMATION COLUMN

上云采購季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺(tái)、長期優(yōu)惠，快來選購！

java性能調(diào)優(yōu)記錄（限流）

2.1 單個(gè) CPU 跑滿

2.2 線程阻塞

2.3 cpu 出現(xiàn)大量軟中斷

2.4 增加 redis 連接

相關(guān)文章

發(fā)表評(píng)論

0條評(píng)論

男|高級(jí)講師

TA的文章

最新活動(dòng)

上云采購季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺(tái)、長期優(yōu)惠，快來選購！