Blog di Dino Ciuffetti (Bernardino in realtà)

11 Feb 23 How to create persistent Queues, Exchanges, and DLXs on RabbitMQ to avoid loosing messages

What happens when you publish a message to an exchange in RabbitMQ with the wrong topic, or better, routing key? What happens if you send a message to the broker in a queue with a TTL policy or the TTL property is in the message itself and that TTL expires? What happens when a consumer discard your message got from the queue with no republish? What if a queue overflows due to a policy?

It’s simple, the broker will simply discard your message forever.

If this thing will make you mad like it does to me this blog article is for you. Here I will tell you how to create a simple tree of queues, DLX and policies to overcome this problem.

I think that starting with commands and examples is better than 10000 written words, and since I don’t have any ads on my blog I don’t have to write a long article to get money from the ads, so here we are.

I consider your RabbitMQ installation and your admin account ready, so we start with the commands.

# Create your VirtualHost
rabbitmqctl add_vhost vhtest --description "Your VH" --default-queue-type classic

# Give your admin user permissions to to everything on your virtualhost
rabbitmqctl set_permissions --vhost vhtest admin '.*' '.*' '.*'

# Create the user that will publish messages to the exchange
rabbitmqctl add_user testuserpub yourpassword

# Create the user that will subscribe to your queue to read messages
rabbitmqctl add_user testusersub yourpassword2

Now, we have 3 users (admin, testuserpub and testusersub) and a virtualhost (vhtest). We are ready to create 2 DLX, one to handle overflow, expired TTL and discarded messages, the other to handle messages sent with the wrong routing key. A DLX (or Dead Letter Exchange) is a particular exchange that is designed to handle dead lettered (discarded) messages.

# Create the DLX to handle overflowed, expired or discarded by consumers
rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQoverfloworttl type=headers internal=true

# Create the DLX to handle messages with wrong routing key
rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQwrongtopic type=fanout internal=true

We’ll now declare and bind queues to the first DLX using three different policies

rabbitmqadmin declare queue --vhost=vhtest name=DLXquQoverflow
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQttl
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQrejected
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQoverflow arguments='{"x-first-death-reason": "maxlen", "x-match": "all-with-x"}'
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQttl arguments='{"x-first-death-reason": "expired", "x-match": "all-with-x"}'
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQrejected arguments='{"x-first-death-reason": "rejected", "x-match": "all-with-x"}'

And now we’ll declare and bind queues to the second DLX to handle messages with wrong topic (routing key)

rabbitmqadmin declare queue --vhost=vhtest name=DLXquQwrongtopic
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQwrongtopic destination=DLXquQwrongtopic

Now we have 1 DLX with 3 queues and another DLX with 1 queue bound. The first will route expired, discarded and overflowed messages to the respective queues (DLXquQttl, DLXquQoverflow, DLXquQrejected), the second will route messages with invalid routing key to the respective queue (DLXquQwrongtopic).

Now we are going to create our main queue and the normal Exchange that will send message to it

rabbitmqadmin declare queue --vhost=vhtest name=quQ
rabbitmqadmin declare exchange --vhost=vhtest name=exQ type=direct

In this example, we want to route all messages with routing key NBE

rabbitmqadmin declare binding --vhost=vhtest source=exQ destination=quQ routing_key=NBE

We now want to create the policy that is needed to associate the wrong topic DLX to our main exchange

rabbitmqctl set_policy --vhost vhtest wrongtopicQ1 "^exQ$" '{"alternate-exchange":"DLXquQwrongtopic"}' --apply-to exchanges

This is an example policy to set limits to 100 messages, 1073741824 bytes, 30 seconds TTL to the quQ queue.

rabbitmqctl set_policy --vhost vhtest shorttimedqunbe '^quQ$' '{"max-length":100,"max-length-bytes":1073741824,"message-ttl":30000,"overflow":"reject-publish-dlx","dead-letter-exchange":"DLXexQoverfloworttl"}' --priority 0 --apply-to queues

Going to give proper permissions to our publish and subscriber users. The user testuserpub can only write to its exchange, while testusersub can read from its queue. No other permissions here.

rabbitmqctl set_permissions --vhost vhtest testuserpub '' '^exQ$' ''
rabbitmqctl set_permissions --vhost vhtest testusersub '' '' '^quQ$'

Mission complete. Please try this at home and write to the comments below! Happy RabbitMQ hacking!

23 Ott 18 How to disable Diffie-Hellman ciphers on apache

If you are getting errors like “DH key too small” you can avoid using DH ciphersuites on apache.
You can obtain that using Perfect forward secrecy, or disabling all DH ciphersuites like this:


06 Lug 18 How to setup xtables GeoIP for iptables on OpenWRT

I’m not racist, but sometimes you may need to keep some people out of your network based on its geographical region… in reality I’m referring to its public source IP address.

Requests are coming from the big internet, so you can check if the source IP of the peers that are trying to connect to your powerful OpenWRT Linux router are coming from a particular region and you may want to stop all those requests accordingly.

If you are using iptables and ip6tables (if you are on linux this is the standard) you can setup GeoIP based packet filtering with iptables: it’s an iptables extension that make use of the glorious and free MaxMind GeoLite Legacy Downloadable Database (that is now dying…).

On OpenWRT this is supported automatically, but generally your router does not have all the disk space to host the entire GeoLite GeoIP database. So, you need to prepare all the records of the geographical regions that you want to block on your firewall.

Take a linux workstation or server (your notebook, a raspberry pi you own always on, one of your client’s server… better not IMHO…) and run those commands:

root@firegate2:~# mkdir /tmp/temp
root@firegate2:~# cd /tmp/temp
root@firegate2:/tmp/temp# /usr/share/xt_geoip/xt_geoip_dl
–2018-07-05 23:54:40–
Risoluzione di (…,, 2400:cb00:2048:1::6810:262f, …
Connessione a (||:80… connesso.
Richiesta HTTP inviata, in attesa di risposta… 200 OK
Lunghezza: 1715246 (1,6M) [application/octet-stream]
Salvataggio in: “GeoIPv6.csv.gz”

GeoIPv6.csv.gz 100%[================================================================>] 1,64M –.-KB/s in 0,1s

2018-07-05 23:54:40 (10,9 MB/s) – “GeoIPv6.csv.gz” salvato [1715246/1715246]

–2018-07-05 23:54:40–
Riutilizzo della connessione esistente a
Richiesta HTTP inviata, in attesa di risposta… 200 OK
Lunghezza: 2540312 (2,4M) [application/zip]
Salvataggio in: “” 100%[================================================================>] 2,42M 11,3MB/s in 0,2s

2018-07-05 23:54:40 (11,3 MB/s) – “” salvato [2540312/2540312]

TERMINATO –2018-07-05 23:54:40–
Tempo totale: 0,4s
Scaricati: 2 file, 4,1M in 0,4s (11,1 MB/s)
inflating: GeoIPCountryWhois.csv
root@firegate2:/tmp/temp# /usr/share/xt_geoip/xt_geoip_build -D . *.csv
209370 entries total
0 IPv6 ranges for A1 Anonymous Proxy
32 IPv4 ranges for A1 Anonymous Proxy
0 IPv6 ranges for A2 Satellite Provider
36 IPv4 ranges for A2 Satellite Provider
3 IPv6 ranges for AD Andorra
26 IPv4 ranges for AD Andorra
51 IPv6 ranges for AE United Arab Emirates
302 IPv4 ranges for AE United Arab Emirates
… and continues …
11 IPv6 ranges for ZM Zambia
64 IPv4 ranges for ZM Zambia
14 IPv6 ranges for ZW Zimbabwe
59 IPv4 ranges for ZW Zimbabwe

The script /usr/share/xt_geoip/xt_geoip_build it’s part of xtables-addons package that you may or may not find on your distribution. Search on google for that.

This will create LE and BE directories. Now you want to extract only the geo regions that you need (ex: CN,UA,TW,VN,VG,KP,VI,KR), like that:

root@firegate2:/tmp/temp# du -csh BE/CN.iv? BE/UA.iv? BE/TW.iv? BE/VN.iv? BE/VG.iv? BE/KP.iv? BE/VI.iv? BE/KR.iv? LE/CN.iv? LE/UA.iv? LE/TW.iv? LE/VN.iv? LE/VG.iv? LE/KP.iv? LE/VI.iv? LE/KR.iv?
36K BE/CN.iv4
48K BE/CN.iv6
24K BE/UA.iv4
16K BE/UA.iv6
8,0K BE/TW.iv4
4,0K BE/TW.iv6
4,0K BE/VN.iv4
4,0K BE/VN.iv6
4,0K BE/VG.iv4
4,0K BE/VG.iv6
4,0K BE/KP.iv4
0 BE/KP.iv6
4,0K BE/VI.iv4
4,0K BE/VI.iv6
8,0K BE/KR.iv4
4,0K BE/KR.iv6
36K LE/CN.iv4
48K LE/CN.iv6
24K LE/UA.iv4
16K LE/UA.iv6
8,0K LE/TW.iv4
4,0K LE/TW.iv6
4,0K LE/VN.iv4
4,0K LE/VN.iv6
4,0K LE/VG.iv4
4,0K LE/VG.iv6
4,0K LE/KP.iv4
0 LE/KP.iv6
4,0K LE/VI.iv4
4,0K LE/VI.iv6
8,0K LE/KR.iv4
4,0K LE/KR.iv6
352K totale

So, in this case we will need 352 Kb of router disk space. After we created the /usr/share/xt_geoip/BE/ and /usr/share/xt_geoip/LE/ directories on our router’s filesystem:

root@dam2ktplinkrouter:~# mkdir -p /usr/share/xt_geoip/BE /usr/share/xt_geoip/LE

We are going to copy those files on our OpenWRT router:

root@firegate2:/tmp/temp# scp LE/CN.iv? LE/UA.iv? LE/TW.iv? LE/VN.iv? LE/VG.iv? LE/KP.iv? LE/VI.iv? LE/KR.iv? root@
root@’s password:
CN.iv4 100% 35KB 109.8KB/s 00:00
CN.iv6 100% 47KB 76.0KB/s 00:00
UA.iv4 100% 23KB 108.4KB/s 00:00
UA.iv6 100% 15KB 87.9KB/s 00:00
TW.iv4 100% 4704 47.5KB/s 00:00
TW.iv6 100% 3104 77.8KB/s 00:00
VN.iv4 100% 3888 98.9KB/s 00:00
VN.iv6 100% 3584 87.9KB/s 00:00
VG.iv4 100% 536 40.1KB/s 00:00
VG.iv6 100% 224 20.0KB/s 00:00
KP.iv4 100% 40 3.9KB/s 00:00
KP.iv6 100% 0 0.0KB/s 00:00
VI.iv4 100% 392 30.6KB/s 00:00
VI.iv6 100% 160 14.4KB/s 00:00
KR.iv4 100% 8128 105.8KB/s 00:00
KR.iv6 100% 3616 87.0KB/s 00:00
root@firegate2:/tmp/temp# scp BE/CN.iv? BE/UA.iv? BE/TW.iv? BE/VN.iv? BE/VG.iv? BE/KP.iv? BE/VI.iv? BE/KR.iv? root@
root@’s password:
CN.iv4 100% 35KB 114.4KB/s 00:00
CN.iv6 100% 47KB 66.0KB/s 00:00
UA.iv4 100% 23KB 115.2KB/s 00:00
UA.iv6 100% 15KB 74.9KB/s 00:00
TW.iv4 100% 4704 93.8KB/s 00:00
TW.iv6 100% 3104 75.7KB/s 00:00
VN.iv4 100% 3888 108.8KB/s 00:00
VN.iv6 100% 3584 76.2KB/s 00:00
VG.iv4 100% 536 40.3KB/s 00:00
VG.iv6 100% 224 20.0KB/s 00:00
KP.iv4 100% 40 4.1KB/s 00:00
KP.iv6 100% 0 0.0KB/s 00:00
VI.iv4 100% 392 33.1KB/s 00:00
VI.iv6 100% 160 15.4KB/s 00:00
KR.iv4 100% 8128 111.1KB/s 00:00
KR.iv6 100% 3616 71.8KB/s 00:00

OK, now we have our pieces of geo ip informations. We now need to install the iptables-mod-geoip from the LuCI web interface (or by hand if you like).
Now you can create your Firewall Traffic Rules. Go to Network -> Firewall -> Traffic Rules on the router’s LuCI web interface and add a custom traffic rule. In my case I created 2 that I called “CinamerdaMuoriUDPeTCP” and “CinamerdaMuoriICMP”, the first to block TCP and UDP and the second to block ICMP traffic.

After that you can setup your custom rule setting your protocol and address families, set “source zone” to WAN, “destination zone” to “Any zone (forward)”, action to DROP, and the “Extra arguments” field like this:

-m geoip –source-country CN,UA,TW,VN,VG,KP,VI,KR

You can check the image below.

OpenWRT Firewall geoip example

Now you can say hello to most chinaspammers and something like that, but don’t abuse, this is not ethic if you set up this on a server that offers some sort of public service.

21 Apr 17 How to create a sparse file from a block device

This is how to create a sparse file dump from a block device:
cp --sparse=always <(dd if=/dev/vg0/vmservice1 bs=8M iflag=direct | pv -pre --size=20G) /opt/backups/vmservice1.dat

14 Gen 17 Come creare un gateway IPv6 su Fastweb

Fastweb ha deciso per il momento di non fornire IPv6 nativo ai propri clienti, e inoltre da qualche giorno ha disabilitato il tunnel TSP ( il quale non risulta piu’ raggiungibile.

Visto che nel mio caso ho un router fastweb Argo 55+ su fibra 100, e il suddetto router non supporta IPv6, quando vado sulla MyFastPage e cerco di attivare il protocollo IPv6 il sistema mi dice che devo sostituire il router con un nuovo modello. Io NON voglio cambiare router perche’ lo considero estremamente stabile e performante.

Come posso quindi attivare IPv6 nella mia rete domestica senza dover cambiare router?

Fastweb porta IPv6 ai suoi utenti tramite 6rd ( Questo significa che e’ probabilmente possibile ottenere la subnet in tunnel anche su linux.

Ho preso uno dei miei raspberry pi con raspbian e ci ho installato il pacchetto radvd (sudo apt-get install radvd), poi nel mio /etc/network/interfaces ho messo questo:

iface eth0 inet6 static
address 2001:b07:27b:7b7b::1
netmask 64

auto ipv6fastweb
iface ipv6fastweb inet6 v4tunnel
netmask 64
up ip -6 route add default dev ipv6fastweb
down ip -6 route del default dev ipv6fastweb

Invece di usare come indirizzo ip 2001:b07:27b:7b7b::1 devi calcolarti il tuo a partire dal tuo IP pubblico fisso che ti ha fornito fastweb. Puoi ottenere il tuo ip pubblico ad esempio da qui:

Facciamo finta che tu abbia l’IP, devi convertirlo in esadecimale, ad esempio cosi:

printf “%x%02x:%x%02x::\n” `echo |tr . ” “`

Quello che otterrai, ad esempio 27b:7b7b:: va accodato al prefisso di fastweb (2001:b07:) e come suffisso accodi il numero 1.

In questo caso, quindi, l’IP diventa:

2001:b07:27b:7b7b::1 che e’ stato costruito da [2001:b07]:[b07:27b:7b7b]::[1]. La prima e’ fissa, la seconda dipende dal tuo IP pubblico e infine 1. Questo andra’ messo nella riga “address” nel file /etc/network/interfaces che ti dicevo prima e va anche messo nella direttiva “prefix” del radvd.conf, ma in ques’ultimo caso senza l’1 finale.

Crea il file /etc/radvd.conf e mettici dentro questo:

interface eth0
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;

prefix 2001:b07:27b:7b7b::/64
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;

RDNSS 2001:4860:4860::8888
AdvRDNSSLifetime 20;

Riavvia il raspberry pi e se tutto va bene, sempre nel raspberry pi dovrai avere una scheda di rete virtuale chiamata ipv6fastweb senza IP usabili ma che serve per creare il tunnel con il border gateway di fastweb ( Se non funziona, prova a cercare un altro border gateway, magari chiedendo al numero verde o cercando su internet. A me funziona con questo. Metti quello giusto alla direttiva “endpoint” del file “interfaces”.

Poi avrai l’IP pubblico IPv6 che ti sei calcolato (nel caso di esempio 2001:b07:27b:7b7b::1/64) sulla scheda eth0 e avrai il tuo radvd che invia i router advertisement ipv6 alla tua rete.

Ogni PC nella tua rete che supporta IPv6 otterrà un IP pubblico nella subnet che ti sei calcolato, e sara’ raggiungibile direttamente da internet tramite IPv6.

Bello no? Ovviamente se abilitate questo dovete disabilitare IPv6 sul router di fastweb perche’ sara il vostro raspberry pi a fare da router ipv6.

Commentate sotto, mi raccomando! Fatemi sapere.

A me funziona perfettamente e sono molto felice. Credo che Fastweb dovrebbe creare una guida ufficiale su questo per il bene degli utenti. Ci ho messo 2 ore a farlo funzionare, con una guida ci avrei messo 2 minuti.



10 Ago 16 Shared memory and huge pages

This script can be used to determine how many shared memory is used by “oracle” user’s processes, and how many huge page memory is used:

t=0; for i in `ipcs -m  | grep oracle | awk ‘{print $5}’`; do echo “Oracle consumed other $i bytes”; let t=t+$i; done

echo “Consumed by oracle as shared memory segments: $t bytes”
let t=t/1024
let g=t/1024/1024;
echo “Memory conversions: ~ $t kbytes | ~ $g GB”

hugepagetot=`cat /proc/meminfo | grep HugePages_Total | awk -F’:’ ‘{print $2}’`
hugepagefree=`cat /proc/meminfo | grep HugePages_Free | awk -F’:’ ‘{print $2}’`
let usedhugepages=hugepagetot-hugepagefree
let totkbinhigepage=usedhugepages*2048

echo “Total hugepage usage in kb: $totkbinhigepage”

04 Ago 16 Change date format to tomcat log catalina.out

If you need to change your catalina.out date and time format, you can add this line to your tomcat/conf/[%1$td.%1$tm.%1$tY %1$tH:%1$tM:%1$tS,%1$tL] %4$s [%2$s] %5$s %6$s %n


06 Mag 16 How to convert URI to query string parameters with mod_rewrite

You may need to convert URI levels to query string parameters, for example if want to be RESTful compliant with PHP.

Try this one:

RewriteEngine on
RewriteRule ^/(\w+)/(\w+)$ /path_of_index.php?lev1=$1&lev2=$2 [QSA,L]

In this case the first URI level will be converted to a query string parameter called lev1, while the second will be converted to a query string parameter called lev2, each one with the respective values.

For example, the uri /user/list will be passed to index.php and will become index.php?lev1=user&lev2=list

An eventual query string will be passed, eventually overriding lev1 and lev2 parameters.

09 Apr 16 Getting in love with Docker Machine!!

Docker is becoming the “today standard” of lxs linux containers.
I think I will avoid learning Kubernetes to handle dockerized hosts, and I will study Docker Engine, Docker Swarm and Docker Machine and its REST APIs instead.

I started from here:

24 Nov 15 HTTP request header from a query string parameter on apache reverse proxy

Suppose you have a apache httpd server working as a reverse proxy. Now suppose that this server has to set a HTTP request header called “token” to be attached to every request made to the backends, and that the header’s value must be copied from a query string parameter called “querytoken”.

This can be simply done with the help of mod_headers + mod_rewrite.

RewriteEngine On
RewriteCond %{QUERY_STRING} (?:^|&)querytoken=([^&]+)
RewriteRule (.*) - [E=QS_TOKEN:%1]

RequestHeader set token %{QS_TOKEN}e env=QS_TOKEN
ProxyPass / http://your_backend/

Will your external client make a request like this:

GET /?querytoken=somestuff HTTP/1.0
Host: example

The request header that apache will do to the backend will be something like this:

GET /?token=somestuff HTTP/1.1
token: somestuff
X-Forwarded-Host: example
X-Forwarded-Server: myserver.linux
Connection: Keep-Alive

The request header “token” with value “somestuff” is added to the request made to the backend.