We are thinking of moving to an upgraded version of WSO2 Enterprise Integrator, this is doing a migration of WSO2, an installation with 100 integrations which was found in WSO2 ESB 4.9.0, Api Manager, and Message Broker. It has continuous CI/CD integration and services managed by Kubernetes displayed in the cloud, as well as a number of components custom-designed as connectors and custom mediators, among others.
It includes the latest developments, is a manageable size and follows simple and repeatable patterns. Other parts were developed years ago by different teams and had a size and complexity that made both its maintenance and evolution particularly difficult.
Additionally, multiple installation and deployment scripts were made available based on gradle and jenkins. However, they also had to be taken into account in our plan in the process of migration of WSO2 to gitlab-ci.
The project also had a deadline constraint that forced us to be agile in terms of the migration of WSO2 process from a system that was already in production.
First dilemma: Which version should we do the migration of WSO2?
The two upgraded versions that were put forward as options for the WSO2 were Enterprise Integrator 6.6.0 y 7.1.0
In order to make a decision with realistic information on the effort and difficulty that each option could entail, a series of tests were carried out, including both the adaptation of tools for automated deployment and integrations.
The conclusions of this study were as follows.
In general, the CAR application files we used in ESB490 deployments were highly consistent with El 7.1.0, including elements of registry, transformations, payload json management, DSS services, backend calls, among others. This gave us confidence that our process of building integrations would not require major changes.
However, EI 7.1.0 is a significant change as it is microservices oriented, and the management API used by our automated deployment process has changed significantly. By definition, microservices should be immutable and, therefore, the management API should not be able to change them. Instead, the microservice should be rebranded and redeployed. That was a huge change in terms of deployment processes.
Another problem with this microservices orientation is that the development carried out thus far was not completely oriented towards this architecture, since there were certain shared sequences or templates, as well as certain elements defined in the registry (templates, endpoints, configurations, etc.) that were also shared between different integrations.
In WSO2 ESB 4.9.0 and EI 6.6.0 there is a shared global context that enables the sharing of some elements commonly used for integrations. Nevertheless, moving it to EI 7.1.0 would have required some redesigning to accommodate these shared elements in a microservices-oriented context. These shared elements would have had to have been deployed in each microservice, which would have required either the redeployment of all microservices every time a change to these common elements is needed or making it a microservice in its own right to be used by the different microservices that consume it.
Because of these differences in the EI 7.1.0 that were a greater risk and a migration effort and considering that we were time constrained, we decided to migrate from ESB 4.9.0 to EI 6.6.0. In short, it is not always the best option to migrate to the latest version. It is necessary to analyse the pros and cons of each option considering the circumstances from the point of view of a qualified technical team.
Preparations for the migration of WSO2 integrations
A study of the projects was carried out to determine the typology according to the design pattern they followed, and the dependencies between them in order to define the order to be followed.
With this information, a list of projects was drawn up, grouped according to their typology and defined structure. It was decided to start working on a project of each of these typologies. In this way, we were able to detect problems or setbacks in earlier phases of the migration of wso2 and then replicate the solutions in the rest of the projects in an easier and faster way.
A further study was carried out on the dependencies between the different projects. Common integrations existed that were transversal and used by all other integrations. There were also direct dependencies between integrations. After reviewing these dependencies, a correct order was established between them, which enabled us to determine a strategy to follow in order to undertake each integration and the roadmap for its implementation in a more secure way, and thus avoid any conflicts derived directly from it.
The next step was reviewing the WS02 documentation concerning the version update. In our case, as we were on ESB version 4.9.0, an old version, and migrated to a recent Enterprise Integrator version 6.6.0, we were reviewing first the migration from our ESB version to a later version, and then, the migration from that ESB version to the Enterprise Integrator.
- This page guides you through the process of upgrading to ESB 5.0.0 from ESB 4.9.0
Upgrading from a Previous Release – Enterprise Service Bus 5.0.0 – WSO2 Documentation
- This page guides you through the process of upgrading to WSO2 Enterprise Integrator (WSO2 EI) 6.6.0 from WSO2 Enterprise Service Bus (WSO2 ESB) 5.0.0
With this information we were able to determine the changes that had been introduced through the different versions; for example, ESB components that had been deprecated in the new versions, and different functionalities or behaviours, among others. This helped us to adapt our integrations to the new environment.
A review of the components that had been custom-developed for ESB 4.9.0 (connectors, custom mediators, libraries, database drivers, etc.) was also carried out in order to determine what changes were needed to adapt them to the new environment.
Another task was to review our CI/CD continuous integration, to check which changes needed to be made in order for the automatic deployment of our integrations in the new environment. For this, we had a first test environment of Enterprise Integrator 6.6.0, where we were able to make the necessary modifications to our deployment scripts and test it through our continuous integration.
Minimising disruption
Since this is a system that is in production and many of the integrations are business-critical, a priority objective of the migration was to not cause any disruption to the service.
On the one hand, this was facilitated by having the infrastructure in the cloud, managed with Kubernetes. On the other hand, we carried out various reliability tests of the deployment operations to verify that there would be no impact from performing the migration while the business was conducting its activity. Except on rare occasions, and more out of precaution than real need, almost all deployments have taken place during working hours.
Most integrations followed the pattern of a publishing service exposing an API. This service writes messages to a topic that are received by various subscribers, and, eventually, it transforms the message and sends it, often by calling a backend to its destination. There were other integrations that followed a different pattern such as exposing an API, and reading or writing from a DSS, but these were less frequent cases, and we handled them for deployments as a publisher.
For the purposes of analysis, we differentiate between publisher and subscriber. To analyse the impact, we prepared a jmeter project that generated a continuous sequence of requests, each request including a sequential ID in order to verify possible missing or duplicate messages. These were verified by checking the event logs generated in Kibana. We also tested different deployment operations to analyse their impact.
For the publisher, the most reliable method found was to follow our automated deployment process, first deploying the integration in EI 6.6.0, and then deploying the API Manager, so that the endpoint would change from what was deployed in ESB 4.9.0 to EI 6.6.0. This transition didn’t take much time and resulted in no impact of lost or duplicate messages, at most there was a minimal delay in changing the endpoint in API Manager.
Subsequently, once normal operation was checked in EI 6.6.0, the CAR file that was deployed in ESB 4.9.0 was deleted. The same procedure was followed for DSS-based projects.
Dangers to avoid
With regard to subscribers, our tests revealed two of pitfalls to avoid:
- If we deployed in EI 6.6.0 while deploying in ESB 4.9.0, at the end there would be one topic with two subscribers, and the received messages would be split between them.
- As subscribers follow a Store and Forward pattern based on a message store, this caused a critical problem. A message serialised in one version of the ESB could not be interpreted by the EI and vice versa. The message store is a shared element between the ESB and the EI, so we had no control over who read and wrote which messages in the message store, if both subscribers were running simultaneously.
The solution
The solution we found to avoid this problem was:
- Disable topic subscription in the ESB 4.9.0 subscriber and wait until the message store has sent all messages to the backend and is empty to avoid problems with serialisation.
- Delete the CAR file of the ESB 4.9.0 subscriber. In this way, the subscription to the topic would become inactive and it would continue to receive the topic messages that accumulated there.
- Deploy the subscription in the EI 6.6.0. Immediately, the subscription was reactivated, and the messages accumulated there started to be processed without serialisation problems. At no time did ESB 4.9.0 and EI 6.6.0 use the message store concurrently.
With these operations, it was possible to avoid serialisation issues, with the minimal inconvenience of delaying message delivery during the time it took to deploy the new subscriber in EI 6.6.0.
Problems detected
During the migration of WSO2 integrations we have encountered various problems to adapt and deploy the integrations in the new EI 6.0.0 environment. Some of these issues stem from the migration from ESB 4.9.0 to the new EI 6.6.0 release, and others from the new container-oriented infrastructure with Kubernetes. The main problems encountered are described in detail below:
When trying to move a file to a SFTP using File Connector, an exception occurs if the system cannot identify the group/owner permissions.
- Enrich Mediator is changing the nature of the payload from JSON to XML jsonObject although it is not related to the body.
- PayloadFactory Mediator fails if inline functions are used directly in the format section.
- In ESB490 when a request has an accept: application/xml header and the response is text, it is running text; while in EI 6.6.0, when a request has an accept: application/xml header and the response is text, it is returning a <text> xml node with the content.
- Using the xpath expression without the text() text function, an xml node is returned instead of the contents of the node. This pattern of behaviour differs from version to version.
- Error processing path parameters in DELETE DSS request.
- In some integrations we find that JSON body disappears after a call to the call mediator and does not continue in the sequence flow.
- As part of the new infrastructure, an Istio proxy has been configured in Kubernetes to handle incoming and outgoing request traffic. In some of the integrations we have a proxy service to call a backend through Istio, and when the backend is unreachable, it sends and RST + ACK as a connection reestablishment. This causes a fatal error when handling the response in EI 6.6.0.
- The Istio proxy changes the headers to lower case on outgoing traffic as in the http-2 protocol standard. This causes EI 6.6.0 to fail when trying to handle the payload received in response because it is case sensitive.
- We have encountered the problem that messages stored in a queue in the Message Store in ESB 4.9.0 generate a failure when trying to consume them from the EI600 due to a serialisation format problem. This only occurs at the time of deployment if both subscribers are operational.
- Problems with configuration of the time zone. In ESB 4.9.0, the time zone is in local format, while in the new platform the EI 6.6.0 is set to UTC format.
- The admin services API changes from version ESB 4.9.0 to version EI 6.6.0.We encountered the problem that the service we use in the ESB490 LogViewer Admin Service to check logs from deployment scripts is no longer valid for EI 6.6.0. This is because this version uses version 2 of the Java Library log4j2 instead of version 1.
A detailed study has been necessary for many of the issues identified above, and the release of WUM updates of EI 6.6.0 by the WSO2 product team has been required for their resolution. Other issues have required workarounds, and settings and changes to the Kubernetes platform configuration.
Conclusions after the migration of WSO2 integrations
It is extremely important to analyse in detail the dependencies between generations at the beginning of the project, and to generate a migration plan from there. In our case, we had a tool that processes projects, discovers significant elements, and looks for relationships between them, so it was relatively easy for us to obtain these dependencies and prepare a plan accordingly. We found ourselves with an integration with a complex design that generated many dependencies and that caused us a bottleneck in the plan that we had to manage carefully. This plan must include that some of the integrations are in the old environment (ESB 4.9.0), while others have been moved across to the new environment (EI 6.6.0).
In a project like this, there can be a significant number of unforeseen events that are very difficult to anticipate. A good way to reduce uncertainty is to find a representative set of integrations of different types and make a first approximation with them in order to have an initial estimate of the complexity by type of integration. In any case, unforeseen events are always going to occur, and there should be a time allowance for this concept so that the project is protected against unforeseeable circumstances.
Simultaneously migrating the integration tool (WSO2) and the platform on which it sits further complicates matters, as there are too many moving parts in the project. It is best to avoid this, but as it is often difficult to achieve, the change of version is used as an opportunity to improve the infrastructure. In which case, a very close cooperation with the infrastructure managers is required.
Although the versions involve changes that, in theory maintain compatibility, and there are documents making the changes made in each version, in reality there are some subtle changes in behaviour that may require the integrations to be reworked and adapted.
It is very important to have a systematic working method that documents the adaptations to be made for known changes or problems. Eventually, there comes a time when most problems are known, and an integration can be migrated by applying a checklist as long as care is taken to generate good documentation. In our case, towards the end of the project it was necessary to dedicate more resources to the migration process, and it was practically enough to give them a checklist to get them up and running quickly. Good documentation was essential for this.
