Message broker and workers in investments project

Why solution architecture is important for investments projects? #

Investments projects are very special in terms of non-functional requirements (aka Quality Attributes). This means that in addition to the features themselves, you need to make sure they are:

1. Business Location specific - it should be possible to decompose the system into separate parts handling different countries, regulations, and compliance.
2. Scalable under the load - analyze and make sure the system will be capable to handle estimated amounts of users/data.
3. Flexible - build in such a way, that it is easy to extend the system in the future.
4. Real-time - usually, users expect financial data/operations to be accurate and represent the latest state of things almost immediately.
5. Fault-tolerant - no financial data should be lost in case of system failure. Maximum functionality should continue to work in case of disaster.
6. Secured - How to store sensitive data? What communication protocols to use? Etc.

Case study #

A project from consumer sector, which simplifies investing by providing analytical tools. Investment is all about information. The right information allows consumers to make correct investments and achieve optimal returns.

Core functional modules:

1. Authentication - login, register, etc.
2. Portfolio - manage personal investment portfolio and ability to see a single consolidated portfolio view.
3. Watch lists - create complex rules to receive alerts about financial market data, performance, price history changes.
4. Financial data - receive financial data (Equities, FX, etc.) updates in real time.
5. Basic administrator panel.

Service-oriented architecture #

To prevent the philosophical dispute about SOA vs microservices, I assume that all real-life systems are actually hybrids. This means they will have services of different size together with monolith-like core(s).

Here are the main considerations:

1. Constant financial data input from 3rd party service in reality is a series of events. It is better to use service, when you expect many events in the system.
2. Publisher/Subscriber & Message Broker patterns will allow different services to react to events they only need.
3. Usage of events-based approach allows later to expand the system more easily because modules are not tightly coupled like in monolith.
4. Decouple critical business logic into separate services, which are easy to scale.
5. Frequent financial updates might overwhelm the system in case we check watch lists for any incoming piece of data. Queues and workers are required to process data. They allow queuing and regulating system resource consumption.
6. It is important to have both persistent storage for user data like portfolio and alerts and data-lake for historical data.

Component and connector view #

The central player of this diagram is a Message Broker. By publishing and consuming events from different services, we decouple them from each other. This fact allows making the overall system more robust (a failure/error inside one service will not influence other services) and flexible (we can easily add more services later and consume already existing events or publish new ones).

A special supervisor service is used to control the resizable workers pool, which could by dynamically scaled. The idea behind this trick is to periodically create jobs, which create alerts based on User provided rules. The job is placed into the queue and performed by workers. It allows controlling the load. In case we have too many incoming events with financial data, we are not reacting to each of them, but periodically perform the jobs.

Summary #

Using the event-driven approach with Message Broker and SOA architecture, we can achieve main Quality Attributes, required by our showcase. The resulting architecture allows scaling individual parts of the system, is ready for future features, and handles real-time data input from 3rd party services.