摘要:從單體系統(tǒng)到微服務(wù)的正確打開方式原文標(biāo)題原文鏈接注每一段譯文后跟作者原文,原文中可能包含著作者所提到的內(nèi)容的跳轉(zhuǎn)超鏈接。從一整塊單體系統(tǒng)遷移到微服務(wù)生態(tài)系統(tǒng)簡直是一段史詩般的旅程。
從單體系統(tǒng)到微服務(wù)的正確打開方式
原文標(biāo)題:How to break a Monolith into Microservices解耦目標(biāo)和解耦時機
原文鏈接:https://martinfowler.com/arti...
注:每一段譯文后跟作者原文,原文中可能包含著作者所提到的內(nèi)容的跳轉(zhuǎn)超鏈接。
隨著整個系統(tǒng)變得過于龐大而難以應(yīng)對,許多企業(yè)開始傾向于將其分解開來,轉(zhuǎn)變?yōu)槲⒎?wù)架構(gòu)。這是一個很有價值的旅程,但也并不是一件很容易完成的事。我們了解到,為了實現(xiàn)這個目標(biāo),需要從一個簡單的服務(wù)開始,然后根據(jù)各部分的垂直業(yè)務(wù)功能,不斷抽出對于系統(tǒng)業(yè)務(wù)來說至關(guān)重要并需要經(jīng)常變更的各種服務(wù)。在最初階段,這些服務(wù)應(yīng)該會很大,并且最好不依賴于尚未分離的大型單體系統(tǒng)本體。我們應(yīng)該確保每一步遷移都是對整個架構(gòu)的原子性演進(jìn)。
As monolithic systems become too large to deal with, many enterprises are drawn to breaking them down into the microservices architectural style. It is a worthwhile journey, but not an easy one. We"ve learned that to do this well, we need to start with a simple service, but then draw out services that are based on vertical capabilities that are important to the business and subject to frequent change. These services should be large at first and preferably not dependent upon the remaining monolith. We should ensure that each step of migration represents an atomic improvement to the overall architecture.
從一整塊單體系統(tǒng)遷移到微服務(wù)生態(tài)系統(tǒng)簡直是一段史詩般的旅程。走上這條征途的人們期望能擴張業(yè)務(wù)規(guī)模,加快改變步伐,并且避免高成本的變革。他們希望擴張團(tuán)隊的數(shù)量,并且使各個團(tuán)隊能夠并行且獨立地產(chǎn)出價值。他們希望快速對其核心業(yè)務(wù)功能點進(jìn)行嘗試,并更快地產(chǎn)生價值。 他們也希望擺脫與修改和維護(hù)現(xiàn)有大型單體系統(tǒng)相關(guān)的高成本。
Migrating a monolithic system to an ecosystem of microservices is an epic journey. The ones who embark on this journey have aspirations such as increasing the scale of operation, accelerating the pace of change and escaping the high cost of change. They want to grow their number of teams while enabling them to deliver value in parallel and independently of each other. They want to rapidly experiment with their business"s core capabilities and deliver value faster. They also want to escape the high cost associated with making changes to their existing monolithic systems.
在向微服務(wù)架構(gòu)遷移的過程中,決定在什么時候以什么樣的方式進(jìn)行增量遷移是一個架構(gòu)上的挑戰(zhàn)。 在這篇文章中,我將分享一些技巧,可以指導(dǎo)交付團(tuán)隊(包括開發(fā)人員、架構(gòu)師和技術(shù)經(jīng)理 )做出這些決定。
Deciding what capability to decouple when and how to migrate incrementally are some of the architectural challenges of decomposing a monolith to an ecosystem of microservices. In this write-up, I share a few techniques that can guide the delivery teams - developers, architects, technical managers - to make these decomposition decisions along the journey.
為了闡述本文所關(guān)注的技術(shù),我將以一個三層架構(gòu)的在線零售系統(tǒng)為例,這個系統(tǒng)的UI層、業(yè)務(wù)邏輯層和數(shù)據(jù)層緊緊地耦合在了一起。我選擇它的原因,是因為這個架構(gòu)具有很多企業(yè)正在運營的大型單體系統(tǒng)的代表性,并且其所采用的技術(shù)棧足夠現(xiàn)代,可以說明我們應(yīng)該對其解耦,而不是重寫或者直接替換。
To clarify the techniques I use a multitier online retail application. This application tightly couples user facing, business logic and data layer. The reason I have chosen this example is that its architecture has the characteristics of monolithic applications that many businesses run and its technology stack is modern enough to justify decomposition instead of a complete rewrite and replacement.
微服務(wù)生態(tài)系統(tǒng)的目標(biāo)在我們開始之前,每個人都對微服務(wù)生態(tài)系統(tǒng)有一個共同的理解是很重要的。所謂微服務(wù)生態(tài)系統(tǒng),是指封裝了各個業(yè)務(wù)功能的服務(wù)平臺。而所謂業(yè)務(wù)功能,是指企業(yè)在特定領(lǐng)域為實現(xiàn)特定目標(biāo)和責(zé)任所做的事情。每個微服務(wù)都將公開一個API,開發(fā)人員可以以"自助"的方式發(fā)現(xiàn)和使用這些API。微服務(wù)具有獨立的生命周期,開發(fā)人員們可以獨立構(gòu)建、測試和發(fā)布各個微服務(wù)。在微服務(wù)生態(tài)系統(tǒng)的組織結(jié)構(gòu)內(nèi),各個團(tuán)隊都是獨立并長期負(fù)責(zé)一個或多個服務(wù)的。與普遍認(rèn)知的微服務(wù)的“微”字可能不同的是,各個服務(wù)的規(guī)??赡芤蚪M織的運營成熟度而異。正如Martin Fowler所說,“微服務(wù)是一種標(biāo)簽,而不是描述”。
Before embarking, it is critical that everyone has a common understanding of a microservices ecosystem. Microservices ecosystem is a platform of services each encapsulating a business capability. A business capability represents what a business does in a particular domain to fulfill its objectives and responsibilities. Each microservice expose an API that developers can discover and use in a self-serve manner. Microservices have independent lifecycle. Developers can build, test and release each microservice independently. The microservices ecosystem enforces an organizational structure of autonomous long standing teams, each responsible for one or multiple services. Contrary to general perception and ‘micro’ in microservices, the size of each service matters least and may vary depending on the operational maturity of the organization. As Martin Fowler puts it, "microservices is a label and not the description".
圖1:服務(wù)封裝了業(yè)務(wù)功能,并通過自助的服務(wù)API公開數(shù)據(jù)和功能
Figure 1: Services encapsulate business capabilities, expose data and functionality through self-serve APIs
旅途指南在深入研究之前,重要的是要知道將現(xiàn)有系統(tǒng)分解為微服務(wù)可能會帶來很高的總體成本,并且可能需要經(jīng)過很多次迭代才能實現(xiàn)。 開發(fā)人員和架構(gòu)師必須仔細(xì)評估是否應(yīng)該對當(dāng)前系統(tǒng)進(jìn)行分解,以及判斷對于當(dāng)前系統(tǒng)來說微服務(wù)本身是否是正確的發(fā)展方向。在想清楚這個問題之后,咱們開始吧。
Before diving into the guide, it is important to know that there is a high overall cost associated with decomposing an existing system to microservices and it may take many iterations. It is necessary for developers and architects to closely evaluate whether the decomposition of an existing monolith is the right path, and whether the microservices itself is the right destination. Having cleared that out, let’s go through the guide.
從簡單而基礎(chǔ)的功能點開始熱身吧只要開始了微服務(wù),就至少需要有最低水平的運維。這需要部署按需訪問的環(huán)境、需要構(gòu)建新的持續(xù)集成系統(tǒng)以實現(xiàn)服務(wù)的獨立構(gòu)建、測試和部署,以及需要一個安全的、可調(diào)試的、可監(jiān)控的分布式體系結(jié)構(gòu)。無論我們正在構(gòu)建新的服務(wù),還是在對現(xiàn)行系統(tǒng)進(jìn)行解耦,對于運維能力都是有要求的。有關(guān)這一點的更多信息,請參閱Martin Fowler關(guān)于微服務(wù)的先決條件的文章。好消息是,自從Martin撰文以來,微服務(wù)架構(gòu)的運維技術(shù)有了迅速發(fā)展,其中包括創(chuàng)建被稱為“服務(wù)網(wǎng)絡(luò)”(Service Mesh)的專用基礎(chǔ)設(shè)施層,可以運行快速、可靠和安全的微服務(wù)網(wǎng)絡(luò),可以創(chuàng)建提供更高級別的基礎(chǔ)設(shè)施抽象的“容器編排系統(tǒng)"、以及諸如GoCD等在容器中構(gòu)建、測試和部署微服務(wù)的持續(xù)集成系統(tǒng)的發(fā)展。
Starting down a microservices path requires a minimum level of operational readiness. It requires on demand access to deployment environment, building new kinds of continuous delivery pipelines to independently build, test, and deploy executable services, and the ability to secure, debug and monitor a distributed architecture. Operational readiness maturity is required whether we are building greenfield services or decomposing an existing system. For more on this operational readiness see Martin Fowler’s article on Microservices prerequisites. The good news is that since Martin’s article, the technology to operate a microservices architecture has evolved rapidly. This includes creation of Service Mesh, a dedicated infrastructure layer to run fast, reliable and secure network of microservices, container orchestration systems to provide a higher level of deployment infrastructure abstraction, and evolution of continuous delivery systems such as GoCD to build, test and deploy microservices as containers.
我的建議是,開發(fā)人員和運維團(tuán)隊首先可以從微服務(wù)的底層基礎(chǔ)設(shè)施,諸如持續(xù)集成系統(tǒng)和API管理系統(tǒng)開始構(gòu)建。從這些與老系統(tǒng)分離的功能點開始構(gòu)建微服務(wù),不需要對當(dāng)前正在運行的面向用戶的業(yè)務(wù)進(jìn)行更改,甚至可能都不需要進(jìn)行數(shù)據(jù)存儲。對于交付團(tuán)隊來說,此時需要不斷優(yōu)化的是對他們的交付方法進(jìn)行驗證、對團(tuán)隊成員能力進(jìn)行提升、構(gòu)建出最基本的可以獨立部署的安全服務(wù)的基礎(chǔ)設(shè)施,從而可以對外公開服務(wù)的API。例如對于在線零售系統(tǒng)來說,首先可以從老系統(tǒng)中分離出來的是用戶認(rèn)證鑒權(quán)服務(wù),以及對于新的客戶程序來說可以提供更好的直觀展現(xiàn)的“客戶信息”服務(wù)。
My suggestion is for developers and operation teams to build out the underlying infrastructure, continuous delivery pipelines and the API management system with the first and second service that they decompose or build new. Start with capabilities that are fairly decoupled from the monolith, they don’t require changes to many client facing applications that are currently using the monolith and possibly don’t need a data store. What the delivery teams are optimizing for at the point is validating their delivery approaches, upskilling the team members, and building out minimum infrastructure needed to deliver independently deployable secure services that expose self-serve APIs. As an example, for an online retail application, the first service can be the ‘end user authentication’ service that the monolith could call to authenticate the end users, and the second service could be the ‘customer profile’ service, a facade service providing a better view of the customers for new client applications.
首先,我建議從簡單的邊緣服務(wù)開始解耦。在此之后,再采用不同的方法深入單體系統(tǒng)解耦其余功能。我建議先做邊緣服務(wù)開始的原因,是因為在剛開始時,交付團(tuán)隊的最大風(fēng)險是無法正確地掌握微服務(wù)的運維。因此,為了掌握必須的“運維先決條件”,從邊緣服務(wù)開始練手是一個很好的選擇。而 一旦這個問題得到解決,剩下的問題就可以迎刃而解。
First I recommended decoupling simple edge services. Next we take a different approach decoupling capabilities deeply embedded in the monolithic system. I advise doing edge services first because at the beginning of the journey, the delivery teams" biggest risk is failing to operate the microservices properly. So it’s good to use the edge services to practice the operational prerequisites they need. Once they have addressed that, they can then address the key problem of splitting the monolith.
圖2:從簡單功能開始鍛煉運維能力
Figure 2: Warming up with a simple capability that has a small radius of change to build our operational readiness
減少對老系統(tǒng)的依賴將新的微服務(wù)系統(tǒng)與老系統(tǒng)之間的依賴關(guān)系最小化是一個基本原則。微服務(wù)的一個主要優(yōu)點是具有快速和獨立的發(fā)布周期。一旦與老系統(tǒng)之間有任何依賴,如數(shù)據(jù)、邏輯或API,都將導(dǎo)致微服務(wù)系統(tǒng)與老系統(tǒng)的發(fā)布周期相耦合,從而使得微服務(wù)的發(fā)布優(yōu)勢不復(fù)存在。通常,擺脫老系統(tǒng)的主要動機就是其成本之高昂,以及深鎖于其中的業(yè)務(wù)功能的更新之緩慢。所以,我們希望逐步通過減少對老系統(tǒng)的依賴關(guān)系的方式,將核心業(yè)務(wù)解耦。如果團(tuán)隊能夠遵循這些原則,將業(yè)務(wù)功能寫進(jìn)自己的服務(wù)中,依賴關(guān)系就能從對單體系統(tǒng)的依賴轉(zhuǎn)變?yōu)閷ξ⒎?wù)的依賴。這是一個理想的依賴方向,因為它將不會拖慢新服務(wù)的更新速度。
As a founding principle the delivery teams need to minimize the dependencies of newly formed microservices to the monolith. A major benefit of microservices is to have a fast and independent release cycle. Having dependencies to the monolith - data, logic, APIs - couples the service to the monolith"s release cycle, prohibiting this benefit. Often the main motivation for moving away from the monolith is the high cost and slow pace of change of the capabilities locked in it, so we want to progressively move in a direction that decouples these core capabilities by removing dependencies to the monolith. If the teams follow this guideline as they build out capabilities into their own services, what they find is instead, dependencies in the reverse direction, from the monolith to the services. This is a desired dependency direction as it does not slow down the pace of change for new services.
在在線零售系統(tǒng)中,購買和促銷是核心功能。購買功能將在結(jié)算過程中使用促銷功能,為顧客提供他們可以獲得的最佳促銷方案,并給他們所購買的商品。如果我們需要決定在這兩種功能之中哪一個先解耦,我的建議是先解耦促銷,再解耦購買。因為通過這個順序,我們可以減少對老系統(tǒng)的依賴。在這個順序中,購買功能將會先被鎖定在整體結(jié)構(gòu)中,并依賴于新的促銷微服務(wù)。
Consider in a retail online system, where ‘buy’ and ‘promotions’ are core capabilities. ‘buy’ uses ‘promotions’ during the checkout process to offer the customers the best promotions that they qualify for, given the items they are buying. If we need to decide which of these two capabilities to decouple next, I suggest to start with decoupling ‘promotions’ first and then "buy". Because in this order we reduce the dependencies back to the monolith. In this order ‘buy’ first remains locked in the monolith with a dependency out to the new ‘promotions’ microservice.
下一條準(zhǔn)則提供了另一些方法來決定服務(wù)的解耦順序。這意味著,可能并不是總能找到一個可以徹底避免對老系統(tǒng)的依賴的方案。如果新服務(wù)最終依然還是調(diào)用到了老系統(tǒng),我建議在老系統(tǒng)中開發(fā)新的API,并通過新服務(wù)的防腐層來訪問這些API,以確保老系統(tǒng)中的概念不會直接暴露出來。就算老系統(tǒng)的內(nèi)部可能實現(xiàn)并不是這樣的,也應(yīng)該致力于定義一個能夠反應(yīng)領(lǐng)域的明確概念和其結(jié)構(gòu)的API。在這種不幸的情況下,交付團(tuán)隊可能需要直面困難,承擔(dān)修改老系統(tǒng)的成本,進(jìn)行測試和發(fā)布與老系統(tǒng)耦合在一起的新服務(wù)。
Next guidelines offer other ways for deciding the order in which developers decouple services. This means that they may not be always able to avoid dependencies back to the monolith. In cases where a new service ends up with a call back to the monolith, I suggest to expose a new API from the monolith, and access the API through an anti-corruption layer in the new service to make sure that the monolith concepts do not leak out. Strive to define the API reflecting the well defined domain concepts and structures, even though the monolith’s internal implementation might be otherwise. In this unfortunate case the delivery teams will be bearing the cost and difficulty of changing the monolith, testing and releasing the new services coupled with the monolith release.
圖3:解耦沒有依賴關(guān)系的服務(wù),并盡量減少對老系統(tǒng)的修改
Figure 3: Decouple the service that doesn’t require a dependency back to the monolith first and minimize changes to the monolith
盡早解耦粘性功能首先我們假設(shè),交付團(tuán)隊樂意于構(gòu)建微服務(wù),并準(zhǔn)備解決遇到的棘手的問題。然而,交付團(tuán)隊很可能會發(fā)現(xiàn)自己缺乏將服務(wù)解耦到對老系統(tǒng)不再有依賴關(guān)系的能力。造成這種情況的根本原因,往往是因為在整體架構(gòu)中的某個功能模塊的設(shè)計上存在問題,沒有被很好地定義為領(lǐng)域概念,導(dǎo)致系統(tǒng)中大量的功能都依賴于它。為了能夠?qū)⒔怦钸M(jìn)行下去,開發(fā)人員需要識別出這些“粘性功能(Sticky Capabilities)”,將其解構(gòu)為擁有良好定義的領(lǐng)域概念,并將這些領(lǐng)域概念轉(zhuǎn)化為多帶帶的服務(wù)。
I am assuming that at this point the delivery teams are comfortable with building microservices and ready to attack the sticky problems. However they may find themselves limited with the capabilities that they can decouple next without a dependency back to the monolith. The root cause of this, is often a capability within the monolith that is leaky, not well defined as a domain concept, with many of the monolith capabilities depending on it. In order to be able to progress, the developers need to identify the sticky capability, deconstruct it into well defined domain concepts and then reify those domain concepts into separate services.
比如在一個網(wǎng)站的單體系統(tǒng)中,會話(Session)是最常見的耦合因素之一。在在線零售系統(tǒng)的示例中,Session中通常存放著許多東西,從跨越了多個域邊界的用戶偏好(比如物流和支付的偏好設(shè)置)到用戶的意圖和用戶交互(比如最近訪問的頁面,瀏覽過的產(chǎn)品和愿望清單)。如果我們不進(jìn)行解耦、重構(gòu)當(dāng)前的會話概念,我們將很難解耦剩下的更多功能,因為它們通過四處彌漫著的Session與老系統(tǒng)緊緊地耦合在了一起。 同時,我也不鼓勵在整體框架之外另外創(chuàng)建一個“會話”服務(wù),因為它只會導(dǎo)致類似的強耦合。相比之下,目前這種耦合關(guān)系僅存在于整體的業(yè)務(wù)流程之中,更糟糕的是,讓它散布到業(yè)務(wù)流程之外乃至整個網(wǎng)絡(luò)中。
For example in a web based monolith, the notion of ‘(web) session’ is one of those most common coupling factors. In the online retail example, the session is often a bucket for many attributes ranging from user preferences across different domain boundaries such as shipping and payment preferences, to user intentions and interactions such as recently visited pages, clicked products, and wish list. Unless we tackle decoupling, deconstructing and reifying the current notion of ‘session’, we will struggle to decouple many of the future capabilities as they will be entangled with the monolith through the leaky session concepts. I also discourage creating a ‘session’ service outside of the monolith, as it will just result in a similar tight coupling that currently exist within the monolith process, only worse, out of process and across the network.
開發(fā)人員可以逐步地從粘性功能中將微服務(wù)提取出來,即一次只提供一個服務(wù)。例如,首先重構(gòu)“愿望清單”并將其提取到新服務(wù)中,然后將“默認(rèn)付款方式”重構(gòu)為另一個微服務(wù),并以此類推。
Developers can incrementally extract microservices from the sticky capability, one service at time. As an example, refactor "customer wish list" first and extract that into a new service, then refactor "customer payment preferences" into another microservice and repeat.
圖4:找到最多的耦合概念并將其解耦重構(gòu)為具體的領(lǐng)域服務(wù)
Figure 4: Identify the most coupling concept and decouple, deconstruct and reify into concrete domain services
使用依賴性和結(jié)構(gòu)化代碼分析工具(如Structure101)來確定整體結(jié)構(gòu)中最具耦合性和約束性的業(yè)務(wù)功能。垂直解耦并且提前發(fā)布數(shù)據(jù)Use dependency and structural code analysis tools such as Structure101 to identify the most coupling and constraining factor capabilities in the monolith.
解耦業(yè)務(wù)功能的主要驅(qū)動力就是讓它們可以獨立發(fā)布。它作為第一原則,指導(dǎo)開發(fā)人員如何進(jìn)行解耦的每一個決定。單體系統(tǒng)通常由多個緊密集成的層結(jié)構(gòu)甚至是多個子系統(tǒng)組成,而這些系統(tǒng)需要一起發(fā)布,并且具有很脆弱的相互依賴性。例如,一個在線零售系統(tǒng)可能由若干個的直接面向客戶的前端應(yīng)用程序,和一個實現(xiàn)了許多業(yè)務(wù)功能的集中式數(shù)據(jù)存儲的后端系統(tǒng)組成。
The main driver for decoupling capabilities out of a monolith is to be able to release them independently. This first principle should guide every decision that developers make around how to perform the decoupling. A monolithic system often is composed of tightly integrated layers or even multiple systems that need to be released together and have brittle interdependencies. For example, in an online retail system, the monolith composed of one or multiple customer facing online shopping applications, a back-end system implementing many of the business capabilities with a centrally integrated data store to hold state.
大多數(shù)的所謂解耦嘗試,只是試圖將一些面向用戶的組件分離出來,并通過外觀模式為前端提供易于使用的API,而數(shù)據(jù)依然被深鎖于單體系統(tǒng)之中。這種方法雖然能快速取得一些成效,比如使得系統(tǒng)可以更頻繁地更改UI,但是一旦涉及到系統(tǒng)的核心功能,更新速度便立刻受限于整個系統(tǒng)中更新最慢的部分——單體系統(tǒng)本身及它的數(shù)據(jù)存儲。簡而言之,沒有對數(shù)據(jù)本身進(jìn)行分離的架構(gòu)并不是微服務(wù)。把所有數(shù)據(jù)都存儲在一起的思想本身就與微服務(wù)的“分布式數(shù)據(jù)管理”思想相悖。
Most decoupling attempts start with extracting the user facing components and a few facade services to provide developer friendly APIs for the modern UIs, while the data remains locked in one schema and storage system. Though this approach gives some quick wins such as changing the UI more frequently, when it comes to core capabilities the delivery teams can only move as fast as the slowest part, the monolith and its monolithic data store. Simply put, without decoupling the data, the architecture is not microservices. Keeping all the data in the same data store is counter to the Decentralized Data Management characteristic of microservices.
解決方案:垂直地分離各個業(yè)務(wù)功能,連業(yè)務(wù)邏輯帶數(shù)據(jù)一起進(jìn)行解耦,并且各個前端應(yīng)用的調(diào)用重定向到新的API上。
The strategy is to move out capabilities vertically, decouple the core capability with its data and redirect all front-end applications to the new APIs.
對于這種數(shù)據(jù)加服務(wù)一起解耦的方法而言,其解耦過程中的主要障礙就是那些需要對集中存儲的共享數(shù)據(jù)進(jìn)行并發(fā)讀寫的各個應(yīng)用程序。針對這種情況,交付團(tuán)隊需要提供合適的數(shù)據(jù)遷移策略,具體取決于他們能否在同時對所有數(shù)據(jù)的讀寫做重定向和遷移。Stripe所寫的“數(shù)據(jù)遷移的四階段策略”適用于許多需要進(jìn)行數(shù)據(jù)增量遷移的環(huán)境,并且可以保證所有正在進(jìn)行遷移的系統(tǒng)可以不中斷地運作。
Having multiple applications writing and reading to and from the centrally shared data is the main blocker to decoupling the data along with the service. The delivery teams need to incorporate a data migration strategy that suits their environment depending on whether they are able to redirect and migrate all the data readers/writers at the same time or not. Stripe’s four phase data migration strategy is one that applies to many environments that require to incrementally migrate the applications that integrate through the database, while all the systems under change need to run continuously.
圖5:將數(shù)據(jù)和服務(wù)遷移到微服務(wù)中,并將所有調(diào)用方調(diào)整和重定向至新API
Figure 5: Decouple capability with its data to a microservice exposing a new interface, modify and redirect consumers to the new API
避免只解耦調(diào)用接口或后端服務(wù)而不解耦數(shù)據(jù)的反模式解耦重要而易變的業(yè)務(wù)功能Avoid the anti pattern of only decoupling facades, only decoupling the backend service and never decoupling data.
從單體系統(tǒng)中將業(yè)務(wù)功能解耦出來是一件很難的事情。我聽說,Neal Ford將其比作“一臺小心翼翼的器官手術(shù)”。從在線零售系統(tǒng)中提取業(yè)務(wù)功能時,需要仔細(xì)地將業(yè)務(wù)邏輯、數(shù)據(jù)、UI組件提取出來并將它們重定向到新服務(wù)。因為其所需的工作量不是一點兩點,所以開發(fā)者們需要不斷地評估進(jìn)行解耦的成本與其能夠帶來的好處,比如,是想要追求更快的開發(fā)速度,還是要追求規(guī)模上的成長。舉個例子,如果交付團(tuán)隊的目標(biāo)是想在修改單體系統(tǒng)中的現(xiàn)有業(yè)務(wù)功能時耗時更少,那么他們必須找到修改最耗時的業(yè)務(wù)功能并將其分離。把代碼中不斷發(fā)生變化的部分多帶帶分離出來,可以從開發(fā)人員那里得到很多的愛,還可以讓他們可以最快速地產(chǎn)出價值。交付團(tuán)隊還可以通過對代碼的提交記錄做分析,找到那些變化得最多的部分,并將其與產(chǎn)品路線圖、產(chǎn)品組合一起分析,以確定在將來最期望的功能,并將其分離出來。他們需要與業(yè)務(wù)經(jīng)理、產(chǎn)品經(jīng)理一起交談,以了解對他們而言真正重要的差異化能力。
Decoupling capabilities from the monolith is hard. I’ve heard Neal Ford use the analogy of a careful organ surgery. In the online retail application, extracting a capability involves carefully extracting the capability’s data, logic, user facing components and redirecting them to the new service. Because this is a non-trivial amount of work, the developers need to continuously evaluate the cost of decoupling against the benefits that they get, e.g. going faster or growing in scale. For example, if the delivery teams" objective is to accelerate the modifications to existing capabilities locked in a monolith, then they must identify the capability that is being modified the most to take out. Decouple parts of the code that are continuously undergoing change and getting a lot of love from the developers and are constraining them most to deliver value fast. The delivery teams can analyse the code commit patterns to find out what has historically changed most, and overlay that with the product roadmap and portfolio to understand the most desired capabilities that will be getting attention in near future. They need to talk to the business and product managers to understand the differentiating capabilities that really matter to them.
例如,在一個在線零售系統(tǒng)中,“客戶個性化”是一項為了提供最佳用戶體驗的、經(jīng)過了大量實驗的功能。因為它是一項事關(guān)客戶體驗的非常重要的業(yè)務(wù)功能,并且經(jīng)常需要修改,所以它是一個進(jìn)行解耦的良好目標(biāo)。
For example in an online retail system, ‘customer personalization’ is a capability that goes under a lot of experimentation to provide the best experience to the customer and is a good candidate for decoupling. It is a capability that matters to business a lot, customer experience, and gets modified frequently.
圖6:找到最重要的業(yè)務(wù)功能并解耦:在定期迭代中創(chuàng)建最多的業(yè)務(wù)和用戶價值
Figure 6: Identify and decouple the capability that matters most: creates most value for business and customer, while changing regularly.
利用社交代碼分析工具(如CodeScene)來查找代碼中最活躍更改的組件。如果自動構(gòu)建系統(tǒng)會在每次提交時觸發(fā)或自動生成代碼,請確保過濾掉這些噪聲。將經(jīng)常發(fā)生更改的代碼與產(chǎn)品路線圖上預(yù)計進(jìn)行的更改疊加,并找到進(jìn)行解耦的交點。解耦業(yè)務(wù)功能,不是解耦業(yè)務(wù)代碼Use social code analysis tools such as CodeScene to find the most lively components. Make sure to filter signal from the noise if the build system happens to touch or auto-generate code on every commit. Overlay the frequently changed code with the product roadmap upcoming changes and find the intersection to decouple.
無論何時,只要開發(fā)人員想要從系統(tǒng)中解耦一項服務(wù)出來,他們都有兩種可行的辦法:提取代碼,或是直接重寫。
Whenever developers want to extract a service out of an existing system, they have two ways to go about it: extract code or rewrite capability.
一般來說,服務(wù)的提取,或是單體系統(tǒng)的解構(gòu)都默認(rèn)被設(shè)想為是通過重用現(xiàn)有的實現(xiàn)方案并將其提取到多帶帶的服務(wù)中來實現(xiàn)。其中的部分原因是因為,我們對我們所親自設(shè)計和編寫的出來的代碼有著認(rèn)知偏好。對于我們所辛苦付出了勞動而收獲的結(jié)果,無論過程有多么痛苦,結(jié)果有多么不完善,我們都會懷有熱愛。這實際上被稱為“宜家效應(yīng)”。不幸的是,這種認(rèn)知偏好會阻礙我們在分解單體系統(tǒng)所付出的努力。它使得開發(fā)人員和更重要的技術(shù)管理人員忽略了提取和重用現(xiàn)有代碼的高成本和低價值。
Often by default the service extraction or monolith decomposition is imagined as a case of reusing the existing implementation as-is and extracting it into a separate service. Partly because we have a cognitive bias towards the code we design and write. The labor of building, no matter how painful the process or imperfect the result, make us grow love for it. This is in fact known as the IKEA Effect. Unfortunately this bias is going to hold the monolith decomposition effort back. It causes the developers and more importantly technical managers to disregard the high cost and low value of extracting and reusing the code.
比如說,在零售系統(tǒng)中,“定價和促銷”功能是一段相當(dāng)復(fù)雜的高難度代碼,它可以動態(tài)地配置和應(yīng)用促銷規(guī)則,并且根據(jù)各種參數(shù)(例如客戶行為,忠誠度,產(chǎn)品捆綁等)提供折扣和優(yōu)惠。
For example in the retail system, the ‘pricing and promotion’ capability is an intellectually complex piece of code. It enables dynamic configuration and application of pricing and promotion rules, providing discounts and offers based on a variety of parameters such as customer behavior, loyalty, product bundles, etc.
像“定價和促銷”這樣的業(yè)務(wù)功能,就是進(jìn)行“重用及提取”的完美選擇。而相比之下,“客戶信息”就是一個很簡單的CRUD功能點,主要也就包括序列化、數(shù)據(jù)存儲以及相關(guān)配置之類的一些模板功能,因此,它是進(jìn)行“重寫和淘汰”的理想對象。
This capability is arguably a good candidate for reuse and extraction. In contrast, ‘customer profile’ is a simple CRUD capability that is mostly composed of boilerplate code for serialization, handling storage and configuration, hence, it is a good candidate for rewrite and retire.
根據(jù)我的經(jīng)驗,在大多數(shù)系統(tǒng)解構(gòu)的情況中,考慮到重用的高成本和低價值,交付團(tuán)隊?wèi)?yīng)當(dāng)將業(yè)務(wù)功能重寫為新的服務(wù)并淘汰掉舊代碼。理由如下:
老代碼中存在大量的“模板代碼”,用于處理一些諸如“在運行時獲取應(yīng)用程序配置”、“訪問數(shù)據(jù)存儲”、“數(shù)據(jù)緩存”之類的環(huán)境依賴,并且這些代碼還都是用的老框架寫的。而大部分的這些模板代碼,都需要重寫。承載微服務(wù)的新基礎(chǔ)架構(gòu)與這些跑了好幾十年的老代碼有很大不同,并且需要相差甚遠(yuǎn)的新的模板代碼來做這些事。
現(xiàn)有的業(yè)務(wù)功能,很有可能并不是圍繞著一個清晰的領(lǐng)域概念而構(gòu)建的。這導(dǎo)致了進(jìn)行傳輸和存儲的數(shù)據(jù)結(jié)構(gòu)并不能直接反映新的領(lǐng)域模型,并且需要進(jìn)行徹底的重構(gòu)。
經(jīng)歷了許多變化和迭代而長期存在的遺留代碼,很有可能具有很高的“代碼毒性”,并且重用價值很低。
In my experience, in majority of the decomposition scenarios, the teams are better off to rewrite the capability as a new service and retire the old code. This is considering the high cost and low value of reuse, due to reasons such as below:
There is a large amount of boilerplate code that deals with environmental dependencies, such as accessing application configuration at runtime, accessing data stores, caching, and is built with old frameworks. Most of this boilerplate code needs to be rewritten. The new infrastructure to host a microservice is very different from the decades old application runtime and will require a very different kind of boilerplate code.
It is very likely that the existing capabilities are not built around clear domain concepts. This results in transporting or storing data structures that are not reflecting the new domain models and require undergoing a big restructuring.
A long lived legacy code that has gone through many iterations of change could have a high code toxicity level and low value for reuse.
除非待解耦的業(yè)務(wù)功能與清晰的領(lǐng)域概念相一致,并且邏輯精巧復(fù)雜,否則我強烈建議進(jìn)行重寫,并淘汰老代碼。
Unless the capability is relevant, aligned with a clear domain concept and has high intellectual property, I strongly recommend a rewrite and retiring of the old code.
圖7:重用和提取低毒性的高價值代碼,重寫和淘汰高毒性的低價值代碼
Figure 7: Reuse and Extract high value code with low toxicity, Rewrite and Retire low value code with high toxicity
使用代碼毒性分析工具(如CheckStyle)來做出關(guān)于重寫與重用的決策。先宏觀,再微觀Use code toxicity analysis tools such as CheckStyle to make decisions around rewrite vs. reuse.
從遺留單體系統(tǒng)中尋找領(lǐng)域邊界,既是一門藝術(shù),又是一門科學(xué)。而作為一個具有普適性的規(guī)則,利用領(lǐng)域驅(qū)動設(shè)計的概念來尋找“上下文邊界”來定義微服務(wù)的邊界是一個很好的開始。我承認(rèn),我經(jīng)常看到一些單體系統(tǒng)的解耦在粒度上“矯枉過正”,將一個過大的系統(tǒng)切分為了過小的服務(wù),而這些過小的服務(wù)常常是從現(xiàn)有數(shù)據(jù)的視角上出發(fā)的。而這種的微服務(wù)識別方法,幾乎總是會導(dǎo)致大量與資源的CURD直接相關(guān)的貧血服務(wù)的“寒武紀(jì)生物大爆發(fā)”。對于新的微服務(wù)架構(gòu)來說,這將創(chuàng)建一個高度摩擦的環(huán)境,并且最終無法獨立發(fā)布測試,也無法提供服務(wù)。這種方式創(chuàng)建了一個難以調(diào)試的分布式系統(tǒng),一個跨越事務(wù)邊界的分布式系統(tǒng),因此很難保持一致,這一切對于運維來說,太復(fù)雜了。盡管存在一些關(guān)于微服務(wù)粒度的“啟發(fā)方法”,諸如團(tuán)隊大小,重寫服務(wù)的時間,必須包括多少行為等等,我的建議是,微服務(wù)的規(guī)模將取決于運維團(tuán)隊可以獨立發(fā)布、監(jiān)控和運營的服務(wù)數(shù)量。首先,圍繞領(lǐng)域概念構(gòu)建一個大型服務(wù),待團(tuán)隊的微服務(wù)運維能力提升之后再將其分解為多個小服務(wù)。
Finding the domain boundaries in a legacy monolith is both an art and science. As a general rule applying domain driven design techniques to find the bounded contexts defining microservices boundaries is a good place to start. I admit, far too often I see an overcorrection from large monolith to really small services, really small services whose design is inspired and driven by the existing normalized view of the data. This approach to identifying service boundaries almost always leads to a cambrian explosion of large number of anemic services for CRUD resources. For many new to the microservices architecture, this creates a high friction environment that ultimately fails the test of independent release and execution of the services. It creates a distributed system that is hard to debug, a distributed system that is broken across transactional boundaries and hence difficult to keep consistent, a system that is too complex for the operational maturity of the organization. Though there are some heuristics on how ‘micro’ should be the microservice: the size of the team, the time to rewrite the service, how much behavior it must encapsulate, etc. My advice is that the size depends on how many services the delivery and operation teams can independently release, monitor and operate. Start with larger services around a logical domain concept, and break the service down into multiple services when the teams are operationally ready.
例如,在解耦零售系統(tǒng)的過程中,開發(fā)者可以從“購買”服務(wù)開始解耦?!百徺I”服務(wù)內(nèi)封裝了“購物袋”的功能,還封裝了購買一個真正的購物袋的功能,即“結(jié)賬”。而隨著他們組建團(tuán)隊并發(fā)布更多服務(wù)的能力不斷增強,他們可以將“購物袋”功能從“結(jié)賬”中解耦出來,成為一個多帶帶的服務(wù)。
For example, on the journey decoupling the retail system, developers may start with one service ‘buy’ that encapsulates both the content of a ‘shopping bag’ as well as capability of buying the shopping bag, i.e ‘check out’. As their ability to form smaller teams and release larger number of services grows then they can decouple ‘shopping bag’ from ‘check out’ into a separate service.
圖8:圍繞豐富的領(lǐng)域概念從宏觀角度上進(jìn)行解耦,在準(zhǔn)備就緒后,將服務(wù)細(xì)分為更小的領(lǐng)域概念
Figure 8: Decouple macro services around rich domain concepts and when ready, breakdown services to smaller domain concepts
使用Richardson L3成熟度模型和超鏈接,可以在不影響調(diào)用方的情況下實現(xiàn)未來的服務(wù)分離,即調(diào)用方可以發(fā)現(xiàn)如何結(jié)賬,并且需要提前知道。原子性地按步遷移Use Richardson Maturity Model L3 and hyperlinks to enable future decoupling of services without impacting callers, i.e. caller discovers how to checkout and does not know in advanced.
想直接將一個巨型單體系統(tǒng)解構(gòu)成一個精心設(shè)計的微服務(wù)系統(tǒng),從而讓其在空氣中消失,是一件是不可能的事。任何一個經(jīng)驗豐富的工程師都可以分享一些關(guān)于嘗試進(jìn)行遺留系統(tǒng)的升級和遷移的故事。而這些嘗試,在最開始都是以非常樂觀的方式被計劃和啟動的,然而它們最好的結(jié)局一般都是“在一個足夠好的時間點被及時放棄”。由于一個宏觀條件的變化,這些系統(tǒng)遷移的長期計劃會被放棄,比如項目資金耗盡、高層的關(guān)注重點轉(zhuǎn)移,或者支持這個項目的領(lǐng)導(dǎo)走人了。所以,現(xiàn)實是,團(tuán)隊?wèi)?yīng)該如何做規(guī)劃,來開始這場龐大的微服務(wù)解耦之旅。這種方法,我稱之為“架構(gòu)演進(jìn)的原子性按步遷移”。遷移的每一步,都應(yīng)使架構(gòu)更接近其目標(biāo)狀態(tài)。而每一次的架構(gòu)演進(jìn),都應(yīng)該是原子性的——無論是一個小步驟還是一個大飛躍,要么完成,要么回滾。這是非常重要的,因為我們正在采取迭代和漸進(jìn)的方法來改進(jìn)整體架構(gòu)和進(jìn)行服務(wù)解耦。就完成架構(gòu)目標(biāo)而言,每個改變都必須讓我們處于更好的位置。使用“演進(jìn)架構(gòu)”的適應(yīng)度函數(shù)進(jìn)行隱喻的話,演進(jìn)的每個原子步驟之后的架構(gòu)適應(yīng)度函數(shù)都應(yīng)該對架構(gòu)的目標(biāo)產(chǎn)生更高價值。
The idea of vanishing a legacy monolith into thin air by decoupling it into beautifully designed microservices is somewhat of a myth and arguably undesirable. Any seasoned engineer can share stories of legacy migration and modernization attempts that got planned and initiated with over optimism of total completion, and at best got abandoned at a good enough point in time. Long term plans of such endeavors get abandoned because the macro conditions change: the program runs out of money, the organization pivots its focus to something else or leadership in support of it leaves. So this reality should be designed in how the teams approach the monolith to microservices journey. I call this approach "migration in atomic steps of architecture evolution", where every step of the migration should take the architecture closer to its target state. Each unit of evolution might be a small step or a large leap but is atomic, either completes or reverts. This is specially important as we are taking an iterative and incremental approach to improving the overall architecture and decoupling services. Every increment must leave us in a better place in terms of the architecture goal. Using the evolutionary architecturefitness function metaphor, the architecture fitness function after every atomic step of migration should generate a closer value to the architecture’s goal.
讓我舉一個例子來說明這一點。 想象一下,微服務(wù)架構(gòu)的目標(biāo)是提高開發(fā)人員的修改系統(tǒng)以提供價值的速度。團(tuán)隊決定將最終用戶身份驗證解耦為一個基于OAuth 2.0協(xié)議的多帶帶服務(wù)。此服務(wù)旨在替代現(xiàn)有老架構(gòu)中的用戶身份驗證系統(tǒng),并以微服務(wù)的新體系結(jié)構(gòu)來進(jìn)行身份驗證。 我們將這種增量改變稱為“導(dǎo)入鑒權(quán)服務(wù)”。導(dǎo)入這項新服務(wù)的一種方法是:
(1)構(gòu)建鑒權(quán)服務(wù),實現(xiàn)OAuth 2.0協(xié)議。
(2)在老系統(tǒng)后端添加一個新的認(rèn)證路徑,以調(diào)用鑒權(quán)服務(wù)來認(rèn)證最終用戶。
Let me illustrate this point with an example. Imagine the microservice architecture goal is to increase the speed of developers modifying the overall system to deliver value. The team decides to decouple the end user authentication into a separate service based on OAuth 2.0 protocol. This service is intended to both replace how the existing (old architecture) client application authenticates the end user, as well as new architecture microservices validate the end user. Let"s call this increment in the evolution, ‘Auth service introduction’. One way to introduce the new service is to go through these steps first:
(1) Build the Auth service, implementing OAuth 2.0 protocol.
(2) Add a new authentication path in the monolith back end to call Auth service for authenticating the end user on whose behalf it is processing a request.
如果團(tuán)隊至此而止,轉(zhuǎn)而去開發(fā)一些其他服務(wù)或功能的話,他們會將整體架構(gòu)置于熵增狀態(tài)。因為在此狀態(tài)下,有兩種實現(xiàn)用戶身份驗證的方式,即新的OAuth 2.0路徑和舊客戶端的基于密碼/會話的路徑。在這一點上,團(tuán)隊實際上遠(yuǎn)離了實現(xiàn)更快更改的總體目標(biāo)。對于新來的老系統(tǒng)開發(fā)人員來說,現(xiàn)在需要處理兩條代碼路徑了。這實際上增加了熟悉代碼的工作量,還導(dǎo)致更改和測試代碼的過程變得更慢了。
If the team stops here and pivots into building some other service or feature, they leave the overall architecture in a state of increased entropy. In this state there are two ways of authenticating the user, the new OAuth 2.0 base path, and old client’s password/session based path. At this point the teams are actually further away from their overall goal of making changes faster. Any new developer to the monolith code needs to deal with two code paths, increased cognitive load of understanding the code, and slower process of changing and testing it.
相反,團(tuán)隊可以在我們的原子演進(jìn)單元中包含以下步驟:
(3)將舊客戶端的基于密碼/會話的身份驗證替換為OAuth 2.0路徑
(4)從老系統(tǒng)中刪除舊的驗證路徑
Instead the team can include the following steps in our atomic unit of evolution:
(3) Replace old client’s password/session based authentication with OAuth 2.0 path
(4) Retire the old authentication code path from the monolith
在這一點上,我們可以爭辯說,在這一點上,團(tuán)隊已經(jīng)接近目標(biāo)架構(gòu)。
At this point we can argue that the teams have gotten closer to the target architecture.
圖9:使用原子演進(jìn)步驟將體系結(jié)構(gòu)演進(jìn)為微服務(wù)。即使中間代碼更改可能會導(dǎo)致其遠(yuǎn)離其目標(biāo),但每步完成后,整體體系結(jié)構(gòu)都朝向目標(biāo)方向改進(jìn)
Figure 9: Evolve the architecture towards microservices with atomic steps of architecture evolution where after each step the overall architecture is improved towards its goal even though intermediary code changes might take it further away from its fitness objective
單體系統(tǒng)解構(gòu)的原子單元包括:
解耦新服務(wù)
將調(diào)用方重定向至新服務(wù)
從單體系統(tǒng)中刪除老代碼
反模式:解耦新服務(wù)給新的調(diào)用方調(diào)用,但是卻不淘汰舊服務(wù)
The atomic unit of monolith decomposition includes:
Decouple the new service
Redirect all consumers to new service
Retire the old code path in the monolith.
The anti-pattern: Decouple the new service, use for new consumers and never retire the old.
我經(jīng)常發(fā)現(xiàn),團(tuán)隊完成了一個業(yè)務(wù)功能的遷移,并且在新的業(yè)務(wù)功能建立之后立即宣告勝利,而不淘汰舊的代碼路徑,即上述的反模式。論其主要原因,一是注重引入新功能帶來的短期收益,二為淘汰舊實現(xiàn)所需的總體工作量,同時還面臨著開發(fā)新功能的優(yōu)先級競爭。為了做正確的事情,我們需要使演進(jìn)的原子步驟盡可能小。
I often find teams end migration of a capability out of the monolith and claim victory as soon as the new capability is built without retiring the old code path, the anti-pattern described above. The main reasons for this are (a) the focus on short-term benefits of introducing a new capability and (b) the total amount of effort required to retire the old implementations while facing competing priorities for building new features. In order to do the right thing, we need to strive for making the atomic steps as small as possible.
通過這種方式進(jìn)行遷移,我們可以在解耦的旅途中小憩,也可以安全地停下來休養(yǎng)生息,并在這漫長的旅程中幸存下來,徹底消滅舊系統(tǒng)。
Migrating with this approach we can break up the journey to shorter trips. We can safely stop, revive and survive this long journey, slaying the monolith.
文章版權(quán)歸作者所有,未經(jīng)允許請勿轉(zhuǎn)載,若此文章存在違規(guī)行為,您可以聯(lián)系管理員刪除。
轉(zhuǎn)載請注明本文地址:http://systransis.cn/yun/69562.html
摘要:從單體系統(tǒng)到微服務(wù)的正確打開方式原文標(biāo)題原文鏈接注每一段譯文后跟作者原文,原文中可能包含著作者所提到的內(nèi)容的跳轉(zhuǎn)超鏈接。從一整塊單體系統(tǒng)遷移到微服務(wù)生態(tài)系統(tǒng)簡直是一段史詩般的旅程。 從單體系統(tǒng)到微服務(wù)的正確打開方式 原文標(biāo)題:How to break a Monolith into Microservices原文鏈接:https://martinfowler.com/arti...注:...
摘要:本篇博客通過小強開飯店的通俗易懂的故事,帶你了解后端服務(wù)是如果從單體應(yīng)用演變到微服務(wù)的。小強開飯店有一天,小強為了早日奔赴小康生活,打算開一個飯店來幫他快速的實現(xiàn)這個目標(biāo)。于是小強開始給服務(wù)盡量的無狀態(tài)化,然后在一個服務(wù)器上啟動了幾個實例。 本篇博客通過小強開飯店的通俗易懂的故事,帶你了解后端服務(wù)是如果從單體應(yīng)用演變到微服務(wù)的。如果有說的不對的地方,歡迎各位大佬強勢懟。 小強開飯店 有...
摘要:相反,它由單體中的適配器和使用一個或多個進(jìn)程間通信機制的服務(wù)組成。因為微服務(wù)架構(gòu)的本質(zhì)是一組圍繞業(yè)務(wù)功能組織的松耦合服務(wù)。如果你嘗試將此類功能實現(xiàn)為服務(wù),則通常會發(fā)現(xiàn),由于過多的進(jìn)程間通信而導(dǎo)致性能下降。這是快速展示微服務(wù)架構(gòu)價值的好方法。你很有可能正在處理大型復(fù)雜的單體應(yīng)用程序,每天開發(fā)和部署應(yīng)用程序的經(jīng)歷都很緩慢而且很痛苦。微服務(wù)看起來非常適合你的應(yīng)用程序,但它也更像是一項遙不可及的必殺...
摘要:本篇博客主要介紹了自動化工具這個概念,在微服務(wù)集群當(dāng)中的作用,算拋磚引玉,歡迎大家提出自己的見解。而在微服務(wù)中,單個服務(wù)重新部署的代價明顯要小的多。 本篇博客主要介紹了自動化工具這個概念,在微服務(wù)集群當(dāng)中的作用,算拋磚引玉,歡迎大家提出自己的見解。 寫在前面 在了解自動化工具的概念之前,我們先了解一下微服務(wù)和集群的概念。 什么是微服務(wù) 這個概念其實有些廣泛,而我的知識廣度也有限,我會盡...
摘要:使用異步編程,有一個事件循環(huán)。它作為面向?qū)ο缶幊痰奶娲桨福渲袘?yīng)用狀態(tài)通常與對象中的方法搭配并共享。在用面向?qū)ο缶幊虝r遇到不同的組件競爭相同的資源的時候,更是如此。 翻譯:瘋狂的技術(shù)宅原文:https://www.indeed.com/hire/i... 本文首發(fā)微信公眾號:jingchengyideng歡迎關(guān)注,每天都給你推送新鮮的前端技術(shù)文章 不管你是面試官還是求職者,里面...
閱讀 1978·2021-11-22 15:33
閱讀 3009·2021-11-18 10:02
閱讀 2622·2021-11-08 13:16
閱讀 1633·2021-10-09 09:57
閱讀 1378·2021-09-30 09:47
閱讀 2013·2019-08-29 13:05
閱讀 3076·2019-08-29 12:46
閱讀 1013·2019-08-29 12:19