Redis PubSub and Microservices

I’m developing a Microservice-based system in predominantly (there are a few components written in Python) AdonisJS.

I just had a little question. To help make things clear, here is a simpler structure of my application.

I’ll focus on two components. The first is my Accounts Service. The Accounts Service is the “Gateway” Service to my system. It handles creating new users, authentication, request routing to other services in the system, and so on.

The second service is a Profile Service, handling user profile information and payments information.

These two services are connected via a Redis Event Store and pub/sub pattern amongst the services, which is made easy because Redis has first-class support. When a new user is created in the Accounts Service, the Accounts Service publishes an event to the Event store with user info. The Profile Service then consumes that event and creates a profile for the newly created user. And, of course, the other services in that system also consume the event and do what they need with it.

Now, suppose, my Profile Service goes down. While it’s down, the Accounts Service publishes three new user events. When the Profile Service goes back up, umm, it won’t be in sync with the rest of the system.

So, that is where I am having trouble.

I realize i’d need some kind of hook after the Http Server starts. But, how would I go about querying Redis for the unconsumed events? The Docs on the Adonis site don’t seem to have anything on it.

Any tips?

Thank you.

Why not running a specific task every hour?

FIRST IDEA :
Let’s say you have a boolean (called “created”) for every new profile event, this boolean is default to 0 (0 means the profile not been created yet, 1 means the profile has been created). The task will check if all the values of each profile events “created” property are equal to 1. If it’s equal to 0 then it will trigger the new profile action.

SECOND IDEA :
Same as first one but instead of a boolean, when a profile is created, you remove the event from the store and the task will check wether there’s an event or not. If true, then it will trigger the new profile action, if not, then everything is OK.

1 Like

The AdonisJs docs doesn’t talk about it, coz your questions are too subjective with service to service communication and not the framework itself.

I am not sure, if you are working with Micro services or it’s just 2 services. In both cases, I will never use Redis pub/sub for communicating, since Redis doesn’t guarantee the delivery of messages when subscribers are not listening, so you are using the wrong tool for the job.

I would simply go with plain HTTP requests, since you get a lot of free stuff with it.

  1. Reliable, if service is down, your HTTP requests will fail and you can bake some logic to handle that or retry.
  2. It’s easier to test HTTP calls.
  3. Your services just need to expose one API.

Now, if you are doing micro services and deeply concerned about the delivery of your messages + speed. Then you need to invest a lot of time to understand the space and build tooling around it.

Extras

The company I am working in, also uses micro services and this is how we handle it.

  1. Using Istio for service to service communication. It handles the retries for us.
  2. Making sure all REST API’s are idempotent
  3. There is a transaction logic with in the services. Listen to this talk and you will get an idea.
2 Likes

I believe the microservice term is “eventual consistancy” and I would use something like AWS SNS/SQS or Rabbit MQ… etc with the pub/sub model. If the profile service goes down the messages/events will just sit in the queue waiting to be processed.

For example: Create a worker class for the profile service that consumes the messages in the queue using a scheduled job.

PS> Two microservices should never use the same datastore. I have never used a Redis Pub/Sub implementation but I “think” you maybe coupling your services which defeats the object.

Oh okay then. That makes things easier.

Thanks