question

Upvotes
Accepted
kostix avatar image
7 ●3 ●2 ●4

Multiple SQS queues with the same news subscription

We intend to deploy our news-handling service which will pull news from AWS SQS queue provided by RDP (called "alerts delivery" by the "News Service on Refinitiv Data Platform - User and Design Guide", v2.0 document).

For durability, we will have two instances (nodes) of this service deployed.
Furthermore, we have two deployments — production and staging, — and our staging deployment must be continuously available as per our internal requirements, so we intend to have four concurrently working instances of our news-handling service deployed.
Please also note that the staging and production deployments are completely separated and do not "know" about each other, so they cannot use the single SQS queue to fetch news as in this case it would be impossible to synchronize reading of the same news updates from the queue, and deleting them from it.

Hence, we have to use at least two SQS queues — one for the staging deployment and one for the production, — or four of them: with each of the two nodes in each deployment using its own queue.

Please advise us on the following questions:

  • Is it possible to use concurrently two or more SQS queues with exactly the same subscription filter? If yes, will they all receive the same alerts? Will they work completely independently of each other? What is the system-imposed limit on the number of SQS queues (or, put it another way, can we have two or four queues)? Are there any session quota limits related to OAuth2 authentication/authorization grants?
  • What do you think would be the best architectural approach to implement durability within our concept of the two identical nodes fetching the same news? Would it be better for the nodes to access separate SQS queues or each pair should share a single one?
rdp-apirefinitiv-data-platform
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

1 Answer

Upvote
Accepted
nick.zincone avatar image
9.7k ●49 ●38 ●60

Hi @khomutov,

When you create a queue, it will act as an independent channel you can pull messages off. You can create multiple queues, with the exact same criteria and both will deliver the same content independently.

That being said, if you plan on running multiple instances, you will need to acquire multiple IDs. An ID today only has the ability to manage one session. That is, you can't use the same ID in multiple applications.

Regarding best practices, I don't see any issue with creating multiple identical queues. Because a queue is associated with a User ID/Machine ID, I don't believe you would be able to create one queue and share it with multiple applications for the reason I state above - an ID has a max session count of 1. You will likely need to acquire multiple IDs and have each create a queue.

I would suggest you reach out to your account manager regarding your situation and the need for multiple IDs. I would also suggest you contact Refinitiv Support to find out if there are any system-imposed limitations on the queues you can create.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Click below to post an Idea Post Idea