11 Feb 23 How to create persistent Queues, Exchanges, and DLXs on RabbitMQ to avoid loosing messages

What happens when you publish a message to an exchange in RabbitMQ with the wrong topic, or better, routing key? What happens if you send a message to the broker in a queue with a TTL policy or the TTL property is in the message itself and that TTL expires? What happens when a consumer discard your message got from the queue with no republish? What if a queue overflows due to a policy?

It’s simple, the broker will simply discard your message forever.

If this thing will make you mad like it does to me this blog article is for you. Here I will tell you how to create a simple tree of queues, DLX and policies to overcome this problem.

I think that starting with commands and examples is better than 10000 written words, and since I don’t have any ads on my blog I don’t have to write a long article to get money from the ads, so here we are.

I consider your RabbitMQ installation and your admin account ready, so we start with the commands.

# Create your VirtualHost rabbitmqctl add_vhost vhtest --description "Your VH" --default-queue-type classic

# Give your admin user permissions to to everything on your virtualhost rabbitmqctl set_permissions --vhost vhtest admin '.*' '.*' '.*'

# Create the user that will publish messages to the exchange rabbitmqctl add_user testuserpub yourpassword1

# Create the user that will subscribe to your queue to read messages rabbitmqctl add_user testusersub yourpassword2

Now, we have 3 users (admin, testuserpub and testusersub) and a virtualhost (vhtest). We are ready to create 2 DLX, one to handle overflow, expired TTL and discarded messages, the other to handle messages sent with the wrong routing key. A DLX (or Dead Letter Exchange) is a particular exchange that is designed to handle dead lettered (discarded) messages.

# Create the DLX to handle overflowed, expired or discarded by consumers rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQoverfloworttl type=headers internal=true

# Create the DLX to handle messages with wrong routing key rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQwrongtopic type=fanout internal=true

We’ll now declare and bind queues to the first DLX using three different policies

rabbitmqadmin declare queue --vhost=vhtest name=DLXquQoverflow rabbitmqadmin declare queue --vhost=vhtest name=DLXquQttl rabbitmqadmin declare queue --vhost=vhtest name=DLXquQrejected rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQoverflow arguments='{"x-first-death-reason": "maxlen", "x-match": "all-with-x"}' rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQttl arguments='{"x-first-death-reason": "expired", "x-match": "all-with-x"}' rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQrejected arguments='{"x-first-death-reason": "rejected", "x-match": "all-with-x"}'

And now we’ll declare and bind queues to the second DLX to handle messages with wrong topic (routing key)

rabbitmqadmin declare queue --vhost=vhtest name=DLXquQwrongtopic rabbitmqadmin declare binding --vhost=vhtest source=DLXexQwrongtopic destination=DLXquQwrongtopic

Now we have 1 DLX with 3 queues and another DLX with 1 queue bound. The first will route expired, discarded and overflowed messages to the respective queues (DLXquQttl, DLXquQoverflow, DLXquQrejected), the second will route messages with invalid routing key to the respective queue (DLXquQwrongtopic).

Now we are going to create our main queue and the normal Exchange that will send message to it

rabbitmqadmin declare queue --vhost=vhtest name=quQ rabbitmqadmin declare exchange --vhost=vhtest name=exQ type=direct

In this example, we want to route all messages with routing key NBE

rabbitmqadmin declare binding --vhost=vhtest source=exQ destination=quQ routing_key=NBE

We now want to create the policy that is needed to associate the wrong topic DLX to our main exchange

rabbitmqctl set_policy --vhost vhtest wrongtopicQ1 "^exQ$" '{"alternate-exchange":"DLXquQwrongtopic"}' --apply-to exchanges

This is an example policy to set limits to 100 messages, 1073741824 bytes, 30 seconds TTL to the quQ queue.

rabbitmqctl set_policy --vhost vhtest shorttimedqunbe '^quQ$' '{"max-length":100,"max-length-bytes":1073741824,"message-ttl":30000,"overflow":"reject-publish-dlx","dead-letter-exchange":"DLXexQoverfloworttl"}' --priority 0 --apply-to queues

Going to give proper permissions to our publish and subscriber users. The user testuserpub can only write to its exchange, while testusersub can read from its queue. No other permissions here.

rabbitmqctl set_permissions --vhost vhtest testuserpub '' '^exQ$' '' rabbitmqctl set_permissions --vhost vhtest testusersub '' '' '^quQ$'

Mission complete. Please try this at home and write to the comments below! Happy RabbitMQ hacking!

Tags: Linux, opensource, RabbitMQ, server, sysadmin
Filed in General, Linux OS, Open source, Sysadmin pills, What I'm doing with 0 Comments

13 Mag 21 How to create a Certification Authority with CRL, OCSP and SAN on OpenSSL

Anyone knows that OpenSSL is a very cool full featured, free and open source SSL/TLS framework and toolkit but few people use it to create a custom Private Certification Authority.

The reasons to create a Private CA are many, but they are out of scope here, so I’ll just say how to achieve the goal.

First you must create a personalized version of my openssl.conf configuration file. You can safely do it modifing this labels below and running this one, on a single line:

# C="IT"; ST="Italy"; L="Rome"; O="My org"; OU="My Unit"; CN="My CA"; eml="my@email.com"; CABASEDIR="/tmp/B"; DD=730; mkdir -p "$CABASEDIR"; cd "$CABASEDIR"; echo 'H4sIAGM8nWACA7VVbYvjNhD+rl8hCIYuhOx2r3fQFMOlyRbChc2SXD4cIQRFmsTq
2ZJPkjeb+/UdyXZsX5JCW/rJmmdG8/povOZsQwTsWZG6LWfxeLStJLJuzhui4Ljl
YJzdCmniaDz6fbR8mkwXERHMsR2z0Abvg+lA7M6uMxHbhD2+/0BynUp+irMThtuW
ArFgJEuveLDwjXCTqiLbgenqTVre8vofNYMcMvL2/uHXLbw5UFZqZePCmlAC4To/
tRVeJoplsNW5izGrugPe+kesLkiwk42jySRqumfSEn3387WyDWC+rxArXbeg/Gwz
5nhC1m1pgzkWypnTM2YVlwbWMQdz82L0q1QcWppUc5ZK1zbW5sCU/M4cFngDZulK
SReUWGIAMGqWVRdskeepBEEgYzIdCYH528ZybeAbEkdaJ9WhkDYBsfUtjBHfXsKE
OWfkrnBgg0UjXozp9R0Sg3ijLogI9sCgX+yR/UpjqnS2QyTkciVmt4nj8ky9QH96
pCk4B4ZyLeCubVgPGikVdfBMqvixi7A3RK7NZekxqg2t0SrsvkhT6nO7u3atCb38
HHWnOquEyg8c+pSjeNexau7PIvIwuODAvAW0HeksZwp9XV5pPM6jW+SZd2Dq8ZZz
C9wr7m7cbgVYRW36jcOx5ejLfLWoWteYhRF8+KXL0Scv0ErqqJpokKVRV1V5WnfZ
uSFr/6yRiRvCCpdog73+BKepAOXkXuJS+gonKYYsPeLTJ2vNbb4huA8lHyNxnWFS
OYtrdfjHaLZ8IrbY/Ykt6bpImE3+1n1fWluAISisLDtAzNFQ4uT7Qh6kY+lSHhRz
hQESHo0A8enCdD5evng7fCVkXS/D67nuGS7XJprSagF5IWSY3kXMPho+KS7zBEyG
iff9L6ENXCYVVqEZYcl9jltGOX/8j80pP6O0pGUpDcNuxxFOpC1nigW86FDnRz9Z
qfaa9M7upyiOOPc0+uhHWerXYSdt/mV+FTeqNGumXGu7M0Wr63wx8y323R3jpPy5
zqAuMjC4rPF/rL623JAe/aILQ8eLGbWJLlJBd0DzYpeGrYuvXVOeaG2Bnrwdzjz3
0chqMR08xIlz+fD+PjtxNsA3fP5Rr8+xmggj6tkZmIub9EowjJXBEQkGhLNpKNf+
diMOC2Eq1/4ZUHzyuUZGmn9Wh7/bDeJTH5wjBaaEP2OPftaUCUGXo2daKL+ykBXU
JUB7lClBc8zXb0L5HW5PtdefPC+Hx+NxkGjrfIgA1AL5CyDty6++CQAA' | openssl base64 -d | gunzip |sed "s/%C%/$C/g"|sed "s/%ST%/$ST/g"|sed "s/%L%/$L/g"|sed "s/%O%/$O/g"|sed "s/%OU%/$OU/g"|sed "s/%eml%/$eml/g"|sed "s|%CABASEDIR%|$CABASEDIR|g"|sed "s/%DD%/$DD/g" > openssl.conf

So, those labels must be modified to your needs:

C="IT"; ST="Italy"; L="Rome"; O="My org"; OU="My Unit"; eml="my@email.com" CABASEDIR="/tmp/B"; DD=730

where C is your Country, ST is your State or Province name, L is locality, O is organization, OU is organization unit, eml is your CA email (if any), CABASEDIR is the directory that will hold all your CA stuff (private keys, certificates, config files, certificate serials and ca db) and DD is your default certificate validity in days.

At this time you should have a file called openssl.conf into your CABASEDIR directory.

As an alternative, you could directly copy and modify the openssl.conf file here.

Next, you obviously need to create the private key and self sign the certificate of your brand new CA, in this example we’ll create a clear RSA private key with 4096 bit encryption length, and a CA certificate that is valid for about 10 years. You mileage may vary, feel free to customize things:

# openssl genrsa -out ca.key 4096
# openssl req -new -x509 -days 3650 -key ca.key -out ca.crt -extensions 'v3_ca' -config openssl.conf

This way you’ll get a clear (not encrypted) private key, so a password is not needed when you’ll going to use it to sign things, generate new certificates, etc. btw, for security reasons, you may need to encrypt your PK with a passphrase. In that case, add -des attribute to your openssl genrsa command. Also you’ll have your precious CA certificate.
Choose a cool CN for your CA Name.

Now it’s time to create a large random number that will be used by OpenSSL as a starting point for your certificates’ Serial Numbers. You can create a large random number with this one (you could also create one by hand…):

# hexdump -n 20 -e '20/1 "%02X" 1 "\n"' /dev/random > certs.seq
C17FA21B11EE604633317891658DCC421F2EDFD5

Perfect. Now proceed creating an empty file called certs.db:

# touch certs.db

Also create a starting serial number for revoked certificates and an empty CRL:

# echo 00 > crlserial
# openssl ca -config openssl.conf -keyfile ca.key -cert ca.crt -gencrl -out crl.pem 
Using configuration from openssl.conf

At this point, you should have something like this:

# ls -lrth
totale 16K
-rw-r--r-- 1 dino dino 1,5K mag 5 20:06 openssl.conf
-rw------- 1 dino dino 3,2K mag 5 20:06 ca.key
-rw-r--r-- 1 dino dino 2,0K mag 5 20:07 ca.crt
-rw-r--r-- 1 dino dino 41 mag 5 20:07 certs.seq
-rw-r--r-- 1 dino dino 3 mag 5 20:08 crlserial
-rw-r--r-- 1 dino dino 999 mag 5 20:08 crl.pem
-rw-r--r-- 1 dino dino 0 mag 5 20:08 certs.db

Well. Now we can start creating our server (or client) certificates.
We start from its private key (here at 2048 bit but you can choose your own key length):

# openssl genrsa -out server.key 2048

And now the certificate:

# openssl req -new -key server.key -out server.csr -extensions 'v3_req' -config openssl.conf
# openssl ca -cert ca.crt -keyfile ca.key -in server.csr -out server.crt -config openssl.conf

If you don’t want to use the default certificates expiry days setted into openssl.conf (param default_days), you can pass the -days attribute to the last command, for example -days 365.
When asked, pay attention to correctly set all the requested attributes, principally the Common Name. Press Y when asked to sign and commit.

If everything gone OK you’ll have your brand new key and certificate:

# ls -lrth server.*
-rw------- 1 dino dino 1,7K mag 5 22:26 server.key
-rw-r--r-- 1 dino dino 1,2K mag 5 22:26 server.csr
-rw-r--r-- 1 dino dino 5,7K mag 5 22:28 server.crt

You can check the certificate with this command:

# openssl x509 -in server.crt -noout -text

Please note that the new certificate is signed by our CA, and also has the following useful properties:

    X509v3 extensions:
        X509v3 Key Usage: 
            Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
        X509v3 Basic Constraints: 
            CA:FALSE
        X509v3 Extended Key Usage: 
            TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha256WithRSAEncryption

Also note that the generated certificate is valid for both Server and Client purposes and has a X509v3 CRL Distribution Points extension that points to URI:http://myca.com/crl.pem: please change this value inside your openssl.conf before generating the CA certificate. This is where you’ll publish your CRL.

As an added bonus, you could also add SANs (Subject Alternative Names) to your certificate, if you like. This permits you to have cool certificates like wildcard domains, multiple domains and IP address on your certificates.

To add a SAN you can modify the last line of openssl.conf configuration file, so that you can include your SANs. For example:

...
[v3_req]
# To add SAN uncomment the # and personalize
subjectAltName=email:copy,DNS:www.host.com,DNS:host.com

and create your CSR and certificate like done above (but first remember to revoke or manually remove the certificate from certs.db or you’ll get the “ERROR:There is already a certificate for…” error):

# openssl req -new -key server.key -out server.csr -extensions 'v3_req' -config openssl.conf
# openssl ca -cert ca.crt -keyfile ca.key -in server.csr -out server.crt -config openssl.conf

To revoke a certificate (in this example is called 792FCB9AE9BBBFFAE33796CF3D1D0D7B6AF399DF.pem) you can simply do this (this will set the given certificate as revoked into the certs.db file):

# openssl ca -config openssl.conf -keyfile ca.key -cert ca.crt -revoke 792FCB9AE9BBBFFAE33796CF3D1D0D7B6AF399DF.pem -crl_reason unspecified
Using configuration from openssl.conf
Revoking Certificate 792FCB9AE9BBBFFAE33796CF3D1D0D7B6AF399DF.
Data Base Updated

After that you need to update the CRL with all the revoked certificates inside. Also, remember to refresh the CRL with the same command almost every default_crl_days (check openssl.conf) even if no certificates are revoked or your CRL will expire:

# openssl ca -config openssl.conf -keyfile ca.key -cert ca.crt -gencrl -updatedb -out crl.pem

At this poin you might want to arrange your OCSP responder with its key and certificate.

Please note that the configuration of OCSP Stapling or responder is out of scope in this article, we just realized how to create its certificates with OpenSSL. If you don’t need OCSP on your certificates, left commented out the authorityInfoAccess attribute in openssl.conf and skip this last step, btw I can tell you, as a testing purposes, how to create a OCSP test responder:

# openssl genrsa -out ocsp.key 2048
# openssl req -new -key ocsp.key -out ocsp.csr -extensions 'v3_req' -config openssl.conf
# openssl ca -cert ca.crt -keyfile ca.key -in ocsp.csr -out ocsp.crt -extensions ocsp -config openssl.conf

When you create the OCSP certificate, keep in mind that the common name must match the OCSP;URI.0 attribute defined into the [ocsp_info] section of your openssl.conf.

# openssl ocsp -index certs.db -port 9999 -rsigner ocsp.crt -rkey ocsp.key -CA ca.crt
ocsp: waiting for OCSP client connections...

And then, to test:

# openssl ocsp -issuer ca.crt -CAfile ca.crt -cert server.crt -url http://ocsp:9999
Response verify OK
server.crt: good
	This Update: May 13 15:10:17 2021 GMT

Now, we try to revoke the server certificate, just for test:

openssl ca -config openssl.conf -keyfile ca.key -cert ca.crt -revoke server.crt -crl_reason unspecified
Using configuration from openssl.conf
Adding Entry with serial number 2DE87D684C64D0BB4B23D0BC9959B8EB23AF932F to DB for /C=IT/ST=Italy/L=Rome/O=My org/OU=My Unit/CN=myserver/emailAddress=my@email.com
Revoking Certificate 2DE87D684C64D0BB4B23D0BC9959B8EB23AF932F.
Data Base Updated

We must also update our CRL and check if the revoked certificate is inserted into our CRL:

# openssl ca -config openssl.conf -keyfile ca.key -cert ca.crt -gencrl -updatedb -out crl.pem                            
Using configuration from openssl.conf
# openssl verify -crl_check -CAfile ca.crt -CRLfile crl.pem server.crt 
C = IT, ST = Italy, L = Rome, O = My org, OU = My Unit, CN = myserver, emailAddress = my@email.com
error 23 at 0 depth lookup: certificate revoked
error server.crt: verification failed

If we recheck now our OCSP responder:

openssl ocsp -issuer ca.crt -CAfile ca.crt -cert server.crt -url http://ocsp:9999 
Response verify OK
server.crt: revoked
	This Update: May 13 15:23:34 2021 GMT
	Reason: unspecified
	Revocation Time: May 13 15:22:14 2021 GMT

Finaly, the mission is complete!!!!

I spent hours getting this things done!! It’s now time to collect and share back to everybody. Bye!!! Ciao, Dino 🙂

openssl.conf.zip Download