Banco do Brasil extracts Open Banking investment data with Quarkus and Kafka - Quarkus

Blog Banco do Brasil extracts Open Banking investment data with Quarkus and Kafka
Banco do Brasil S.A. is a Brazilian financial services company headquartered in Brasília, Brazil. The oldest bank in Brazil, and among the oldest banks in continuous operation in the world, it was founded by John VI, King of Portugal, in 1808. It is the second largest banking institution in Brazil, as well as the second largest in Latin America, and the seventy-seventh largest bank in the world. Banco do Brasil is controlled by the Brazilian government and is listed on the B3 stock exchange in São Paulo. [1]
There’s an Investment Portfolio application in Banco do Brasil where all the client’s investments are grouped together. This helps our financial specialists to give advice and also allows the customer to see all of their investments in one place. This information is stored on a daily basis, thus it is possible to create graphics showing the changes in the client’s investments over time.
With Brazilian Open Banking’s creation, now it is possible to retrieve our customers investments' data from other financial institutions.
According to Brazilian Central Bank:
The Banco Central do Brasil (BCB - Central Bank of Brazil) and the National Monetary Council (CMN) define the Brazilian Open Banking environment as the sharing of data, products and services between regulated entities — financial institutions, payment institutions and other entities licensed by BCB — at the customers' discretion, as far as their own data is concerned (individuals or legal entities).[2]
The business leaders envisioned the opportunity to improve customer experience by aggregating all investments, from others financial institutions, into our brand new and powerful Investment Portfolio solution.
The task was to extract this data in two different ways:
Daily - on every work day.
On demand - when the data is requested by the financial specialists or directly by the customer.
There were some difficulties in accomplishing this task.
We have millions of customers, so the application must be able to scale up to handle all the processing. At first there were not so many clients using this feature, but this number is growing daily.
The challenge is that we have a throttling rule for using the APIs from the other financial institutions. So, we can scale up our application to cope with the demand, but we need to respect the maximum number of API calls we can do per minute.
The on demand requests cannot be put at the end of the processing queue concurrently with the daily process.
This would be a problem, because the daily process can take hours and our clients and financial specialists cannot wait all this time when they make a request.
The daily processing must accurately store the data of every work day.
The difficulty is that sometimes some financial institutions have problems, so it is not possible to be sure that the Open Banking information will be available every day.
It is necessary to create some rules, where we can reprocess information from previous days.
We need to have some metrics to be able to know if the system is working correctly or if there is a problem.
For instance, "why is the data extraction taking so long today?" "Is it some financial institution that is having a problem?" "Is it our system?" "Our database?"
Other questions we must answer are "Has the daily process already run today?" "How many API calls had a problem in the last processing?"
The legacy Investment Portfolio backend runs in a mainframe environment, with COBOL programs, JCL procedures and DB2 database.
Daily, this application receives huge files with investment data from other internal systems of our bank, like Investment Funds, Accounts and Shares.
It is important to recognize that the mainframe environment thrives with this kind of massive processing, reading large amounts of data files and storing data on the database using specialized processes that don’t execute individual SQLs. This means that we can generate a file with millions of records and execute one DB2 procedure that stores all this data very quickly.
Banco do Brasil technology now has a private cloud orchestrated with Kubernetes, where we are able to run everything that can be built in a container. The most common languages used are Java, TypeScript, Python and Go.
Quarkus was chosen by our Development Supporting Team as the official Java framework to be used. The main reasons were:
Very good development experience with fast startup time and live reload.
Quarkus implements the Eclipse MicroProfile specification meaning that we are not locked in one specific solution, so theoretically we can change to another MicroProfile implementation if needed.
The Quarkus community is very active, bugs are solved quickly and new features are released frequently.
Quarkus is cloud-native.
Therefore, Quarkus is being massively used by us to create microservices. We have all sorts of applications that use it, the most common are simple API CRUDs, but we also have batch processes, integration with other institutions using REST APIs, integration with B3, the Brazilian stock exchange, using the FIX (Financial Information eXchange) protocol, low latency applications, and much more.
A new system was created in our company to provide integration with the other financial institutions that participate in the Brazilian Open Banking environment. It’s our Open Banking Integrator and it was built with Quarkus.
At this point, we needed to decide how we would overcome all the project challenges. Would we stick with the mainframe, where it would be possible to communicate with our Open Banking Integrator, or should we try something new?
As with all decisions in software architecture, we always have pros and cons. In the end, we decided to solve this problem running microservices written in Quarkus in our private cloud. The main reasons were:
It is easier to scale the process horizontally.
Increase possibilites to expose metri