Spring Cloud Config Server遷移節(jié)點或容器化帶來的問題

AZmake 發(fā)布于2019-08-15 17:53 / 2218人閱讀

如果您跟我一樣，目前正在使用Spring Cloud Config做為配置中心的話，本篇將來要描述的問題，強烈推薦了解和關(guān)注！因為這個問題目前存在于所有的版本中，還沒有完全修復(fù)。

原文鏈接：http://blog.didispace.com/Spr...

問題現(xiàn)象

為了說明下面的內(nèi)容，我們可以先嘗試重現(xiàn)一下問題：在一個測試環(huán)境中，將Spring Cloud Config的配置中心遷移到另外一個節(jié)點上，即配置中心的IP地址發(fā)生了變化。在完成遷移之后，我們會發(fā)現(xiàn)該環(huán)境下各個微服務(wù)應(yīng)用的健康狀態(tài)會變得時好時壞，并且在日志中會出現(xiàn)類似下面的報錯：

2018-05-13 17:01:28,569 WARN  [http-nio-9920-exec-1] org.springframework.cloud.config.client.ConfigServerHealthIndicator - Health check failed
java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
    at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:132)
    at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$FastClassBySpringCGLIB$$fa44b2a.invoke()
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
    at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)
    at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$EnhancerBySpringCGLIB$$3a43a1f4.locate()
    at org.springframework.cloud.config.client.ConfigServerHealthIndicator.getPropertySource(ConfigServerHealthIndicator.java:54)
    at org.springframework.cloud.config.client.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:35)
    at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
    at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85)
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:177)
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:166)
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:143)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at com.yonghui.feign.filter.RequestOriginFilter.doFilter(RequestOriginFilter.java:41)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at com.yonghui.rpc.feature.web.boot.RpcHolder4BootFilter.doFilterInternal(RpcHolder4BootFilter.java:29)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at com.yonghui.rpc.feature.web.boot.FeatureSupport4BootFilter.doFilterInternal(FeatureSupport4BootFilter.java:24)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://192.168.5.103:9010/config-server/test": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:539)
    at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:172)
    at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:93)
    ... 95 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.http.HttpClient.(HttpClient.java:242)
    at sun.net.www.http.HttpClient.New(HttpClient.java:339)
    at sun.net.www.http.HttpClient.New(HttpClient.java:357)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
    at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
    ... 99 more

可以看到類似上面的健康檢查失敗錯誤，但是并不是一直這樣，這個環(huán)境下的微服務(wù)會出現(xiàn)時好時壞的情況，那么為什么會出現(xiàn)這種現(xiàn)象呢？

原因分析

從錯誤日志中我們可以發(fā)現(xiàn)一個非常關(guān)鍵的信息： I/O error on GET request for "http://192.168.5.103:9010/config-server/test"。

報錯說明了微服務(wù)檢查配置中心獲取配置的連接是否暢通的時候出現(xiàn)了連接不上的情況，但是這個鏈接信息其實并不是當(dāng)前配置中心的地址，而是我們遷移之前的配置中心的地址。

從健康檢查的實現(xiàn)源碼ConfigServerHealthIndicator中為入口去分析和調(diào)試，我們可以解答上面現(xiàn)象的兩個疑問：

    @Override
    protected void doHealthCheck(Builder builder) throws Exception {
        PropertySource propertySource = getPropertySource();
        builder.up();
        if (propertySource instanceof CompositePropertySource) {
            List sources = new ArrayList<>();
            for (PropertySource ps : ((CompositePropertySource) propertySource).getPropertySources()) {
                sources.add(ps.getName());
            }
            builder.withDetail("propertySources", sources);
        } else if (propertySource!=null) {
            builder.withDetail("propertySources", propertySource.toString());
        } else {
            builder.unknown().withDetail("error", "no property sources located");
        }
    }

    private PropertySource getPropertySource() {
        long accessTime = System.currentTimeMillis();
        if (isCacheStale(accessTime)) {
            this.lastAccess = accessTime;
            this.cached = locator.locate(this.environment);
        }
        return this.cached;
    }

為什么會健康檢查訪問的還是老的配置中心地址？

真正導(dǎo)致健康檢查失敗的語句是getPropertySource中的 this.cached = locator.locate(this.environment);而這里的具體實現(xiàn)在org.springframework.cloud.config.client.ConfigServicePropertySourceLocator類中，具體實現(xiàn)如下：

@Override
@Retryable(interceptor = "configServerRetryInterceptor")
public org.springframework.core.env.PropertySource locate(
            org.springframework.core.env.Environment environment) {
    ConfigClientProperties properties = this.defaultProperties.override(environment);
    CompositePropertySource composite = new CompositePropertySource("configService");
    RestTemplate restTemplate = this.restTemplate == null ? getSecureRestTemplate(properties)
                : this.restTemplate;
    Exception error = null;
    String errorBody = null;
    logger.info("Fetching config from server at: " + properties.getRawUri());
        ...
}

可以看到，真正去訪問的地址是直接從properties.getRawUri()獲取的，它已經(jīng)是一個固化的值，而不是通過服務(wù)發(fā)現(xiàn)機制來動態(tài)獲取的。這就導(dǎo)致了當(dāng)我們把配置中心做了遷移，或者直接部署在容器中出現(xiàn)重啟的時候，IP發(fā)生變化，而所有的微服務(wù)還以為訪問的是原來的配置中心地址，就會出現(xiàn)健康檢查失敗的問題，導(dǎo)致服務(wù)不可用的現(xiàn)象。

為什么健康檢查時好時壞？

上面的問題會導(dǎo)致健康檢查失敗，但是這個服務(wù)并不是一直都不好，而是間斷性的出現(xiàn)不健康。這主要還是健康檢查時間中的機制導(dǎo)致，這里可以具體看ConfigServerHealthIndicator的getPropertySource函數(shù)，該方法執(zhí)行的時候中間并不是每一次檢查都會去訪問配置中心（執(zhí)行locator.locate(this.environment)方法），因此客戶端的健康檢查并不會每次都健康檢查失敗，從而出現(xiàn)了微服務(wù)健康檢查時好時壞的情況。

如何解決

該問題目前也在官方的issue中被提出，還處于open狀態(tài)，具體可見：https://github.com/spring-clo...

由于該問題目前并沒有得到解決，雖然提交了一個PR，但是還有待完善以及提供一些測試，原本想完全處理好之后再寫一篇文章，但是發(fā)現(xiàn)最近不少問過類似問題，所以索性先寫一篇文章，提醒一下用戶以及給一些相關(guān)的建議。

當(dāng)前版本上不太容易通過擴展的方式去解決的這個問題，所以大家可以變通的去避免這個問題：

部署在虛擬機上而不是容器上，避免IP的變動

可以考慮關(guān)閉微服務(wù)隊config客戶端的健康檢查，增加參數(shù)management.health.config.enabled=false；但是這個操作有一個弊端，雖然遷移不會引發(fā)服務(wù)時好時壞的問題了，但是如果有動態(tài)配置刷新需求，如果遷移了配置中心，刷新配置操作也是會失敗的。

補充一個網(wǎng)友給的思路，通過內(nèi)部域名來綁定配置中心，而不是ip地址

GPU云服務(wù)器云服務(wù)器大數(shù)據(jù)帶來的問題租用idc機房解決的問題和帶來的效益 Spring Cloud spring-cloud

文章版權(quán)歸作者所有，未經(jīng)允許請勿轉(zhuǎn)載,若此文章存在違規(guī)行為，您可以聯(lián)系管理員刪除。

轉(zhuǎn)載請注明本文地址：http://systransis.cn/yun/69498.html

發(fā)表評論

登陸后可評論

0條評論

AZmake

男|高級講師

我要關(guān)注我要私信

TA的文章

Spark綜合學(xué)習(xí)筆記（三）搜狗搜索日志分析

閱讀 3325·2021-11-18 10:02
為什么這么多應(yīng)屆生要進入互聯(lián)網(wǎng)行業(yè)？

閱讀 3473·2021-10-11 10:58
長知識了！這8個很少用但卻很實用的 Python 庫真棒！

閱讀 3399·2021-09-24 09:47
云主機怎么登陸-怎么登錄云主機？

閱讀 1180·2021-09-22 15:21
寶塔面板建站網(wǎng)站未備案域名無法打開網(wǎng)頁怎么解決?

閱讀 4037·2021-09-10 11:10
??擼完這個springboot項目，我對boot輕車熟路！【源碼+視頻都開源】【強烈建議收藏】??

閱讀 3303·2021-09-03 10:28
初探keyframes-animation

閱讀 1775·2019-08-30 15:45
div垂直居中知幾種？

閱讀 2173·2019-08-30 14:22

成人国产在线小视频_日韩寡妇人妻调教在线播放_色成人www永久在线观看_2018国产精品久久_亚洲欧美高清在线30p_亚洲少妇综合一区_黄色在线播放国产_亚洲另类技巧小说校园_国产主播xx日韩_a级毛片在线免费

資訊專欄INFORMATION COLUMN

上云采購季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺、長期優(yōu)惠，快來選購！

Spring Cloud Config Server遷移節(jié)點或容器化帶來的問題

相關(guān)文章

**服務(wù)遷移之路 | Spring Cloud向Service Mesh轉(zhuǎn)變**

**部署微服務(wù)：Spring Cloud vs. Kubernetes**

SpringCloud打造微服務(wù)平臺--概覽

發(fā)表評論

0條評論

AZmake

男|高級講師

TA的文章

Spark綜合學(xué)習(xí)筆記（三）搜狗搜索日志分析

為什么這么多應(yīng)屆生要進入互聯(lián)網(wǎng)行業(yè)？

長知識了！這8個很少用但卻很實用的 Python 庫真棒！

云主機怎么登陸-怎么登錄云主機？

寶塔面板建站網(wǎng)站未備案域名無法打開網(wǎng)頁怎么解決?

??擼完這個springboot項目，我對boot輕車熟路！【源碼+視頻都開源】【強烈建議收藏】??

初探keyframes-animation

div垂直居中知幾種？

最新活動

資訊專欄INFORMATION COLUMN

上云采購季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺、長期優(yōu)惠，快來選購！

Spring Cloud Config Server遷移節(jié)點或容器化帶來的問題

相關(guān)文章

發(fā)表評論

0條評論

男|高級講師

TA的文章

最新活動

上云采購季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺、長期優(yōu)惠，快來選購！