11 Feb 23 How to create persistent Queues, Exchanges, and DLXs on RabbitMQ to avoid loosing messages
What happens when you publish a message to an exchange in RabbitMQ with the wrong topic, or better, routing key? What happens if you send a message to the broker in a queue with a TTL policy or the TTL property is in the message itself and that TTL expires? What happens when a consumer discard your message got from the queue with no republish? What if a queue overflows due to a policy?
It’s simple, the broker will simply discard your message forever.
If this thing will make you mad like it does to me this blog article is for you. Here I will tell you how to create a simple tree of queues, DLX and policies to overcome this problem.
I think that starting with commands and examples is better than 10000 written words, and since I don’t have any ads on my blog I don’t have to write a long article to get money from the ads, so here we are.
I consider your RabbitMQ installation and your admin account ready, so we start with the commands.
# Create your VirtualHost
rabbitmqctl add_vhost vhtest --description "Your VH" --default-queue-type classic
# Give your admin user permissions to to everything on your virtualhost
rabbitmqctl set_permissions --vhost vhtest admin '.*' '.*' '.*'
# Create the user that will publish messages to the exchange1
rabbitmqctl add_user testuserpub yourpassword
# Create the user that will subscribe to your queue to read messages
rabbitmqctl add_user testusersub yourpassword2
Now, we have 3 users (admin, testuserpub and testusersub) and a virtualhost (vhtest). We are ready to create 2 DLX, one to handle overflow, expired TTL and discarded messages, the other to handle messages sent with the wrong routing key. A DLX (or Dead Letter Exchange) is a particular exchange that is designed to handle dead lettered (discarded) messages.
# Create the DLX to handle overflowed, expired or discarded by consumers
rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQoverfloworttl type=headers internal=true
# Create the DLX to handle messages with wrong routing key
rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQwrongtopic type=fanout internal=true
We’ll now declare and bind queues to the first DLX using three different policies
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQoverflow
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQttl
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQrejected
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQoverflow arguments='{"x-first-death-reason": "maxlen", "x-match": "all-with-x"}'
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQttl arguments='{"x-first-death-reason": "expired", "x-match": "all-with-x"}'
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQrejected arguments='{"x-first-death-reason": "rejected", "x-match": "all-with-x"}'
And now we’ll declare and bind queues to the second DLX to handle messages with wrong topic (routing key)
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQwrongtopic
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQwrongtopic destination=DLXquQwrongtopic
Now we have 1 DLX with 3 queues and another DLX with 1 queue bound. The first will route expired, discarded and overflowed messages to the respective queues (DLXquQttl, DLXquQoverflow, DLXquQrejected), the second will route messages with invalid routing key to the respective queue (DLXquQwrongtopic).
Now we are going to create our main queue and the normal Exchange that will send message to it
rabbitmqadmin declare queue --vhost=vhtest name=quQ
rabbitmqadmin declare exchange --vhost=vhtest name=exQ type=direct
In this example, we want to route all messages with routing key NBE
rabbitmqadmin declare binding --vhost=vhtest source=exQ destination=quQ routing_key=NBE
We now want to create the policy that is needed to associate the wrong topic DLX to our main exchange
rabbitmqctl set_policy --vhost vhtest wrongtopicQ1 "^exQ$" '{"alternate-exchange":"DLXquQwrongtopic"}' --apply-to exchanges
This is an example policy to set limits to 100 messages, 1073741824 bytes, 30 seconds TTL to the quQ queue.
rabbitmqctl set_policy --vhost vhtest shorttimedqunbe '^quQ$' '{"max-length":100,"max-length-bytes":1073741824,"message-ttl":30000,"overflow":"reject-publish-dlx","dead-letter-exchange":"DLXexQoverfloworttl"}' --priority 0 --apply-to queues
Going to give proper permissions to our publish and subscriber users. The user testuserpub can only write to its exchange, while testusersub can read from its queue. No other permissions here.
rabbitmqctl set_permissions --vhost vhtest testuserpub '' '^exQ$' ''
rabbitmqctl set_permissions --vhost vhtest testusersub '' '' '^quQ$'
Mission complete. Please try this at home and write to the comments below! Happy RabbitMQ hacking!
23 Ott 18 How to disable Diffie-Hellman ciphers on apache
If you are getting errors like “DH key too small” you can avoid using DH ciphersuites on apache.
You can obtain that using Perfect forward secrecy, or disabling all DH ciphersuites like this:
SSLCipherSuite ALL:!EXP:!NULL:!DH:!LOW
06 Lug 18 How to setup xtables GeoIP for iptables on OpenWRT
I’m not racist, but sometimes you may need to keep some people out of your network based on its geographical region… in reality I’m referring to its public source IP address.
Requests are coming from the big internet, so you can check if the source IP of the peers that are trying to connect to your powerful OpenWRT Linux router are coming from a particular region and you may want to stop all those requests accordingly.
If you are using iptables and ip6tables (if you are on linux this is the standard) you can setup GeoIP based packet filtering with iptables: it’s an iptables extension that make use of the glorious and free MaxMind GeoLite Legacy Downloadable Database (that is now dying…).
On OpenWRT this is supported automatically, but generally your router does not have all the disk space to host the entire GeoLite GeoIP database. So, you need to prepare all the records of the geographical regions that you want to block on your firewall.
Take a linux workstation or server (your notebook, a raspberry pi you own always on, one of your client’s server… better not IMHO…) and run those commands:
root@firegate2:~# mkdir /tmp/temp
root@firegate2:~# cd /tmp/temp
root@firegate2:/tmp/temp# /usr/share/xt_geoip/xt_geoip_dl
–2018-07-05 23:54:40– http://geolite.maxmind.com/download/geoip/database/GeoIPv6.csv.gz
Risoluzione di geolite.maxmind.com (geolite.maxmind.com)… 104.16.37.47, 104.16.38.47, 2400:cb00:2048:1::6810:262f, …
Connessione a geolite.maxmind.com (geolite.maxmind.com)|104.16.37.47|:80… connesso.
Richiesta HTTP inviata, in attesa di risposta… 200 OK
Lunghezza: 1715246 (1,6M) [application/octet-stream]
Salvataggio in: “GeoIPv6.csv.gz”GeoIPv6.csv.gz 100%[================================================================>] 1,64M –.-KB/s in 0,1s
2018-07-05 23:54:40 (10,9 MB/s) – “GeoIPv6.csv.gz” salvato [1715246/1715246]
–2018-07-05 23:54:40– http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
Riutilizzo della connessione esistente a geolite.maxmind.com:80.
Richiesta HTTP inviata, in attesa di risposta… 200 OK
Lunghezza: 2540312 (2,4M) [application/zip]
Salvataggio in: “GeoIPCountryCSV.zip”GeoIPCountryCSV.zip 100%[================================================================>] 2,42M 11,3MB/s in 0,2s
2018-07-05 23:54:40 (11,3 MB/s) – “GeoIPCountryCSV.zip” salvato [2540312/2540312]
TERMINATO –2018-07-05 23:54:40–
Tempo totale: 0,4s
Scaricati: 2 file, 4,1M in 0,4s (11,1 MB/s)
Archive: GeoIPCountryCSV.zip
inflating: GeoIPCountryWhois.csv
root@firegate2:/tmp/temp#
root@firegate2:/tmp/temp# /usr/share/xt_geoip/xt_geoip_build -D . *.csv
209370 entries total
0 IPv6 ranges for A1 Anonymous Proxy
32 IPv4 ranges for A1 Anonymous Proxy
0 IPv6 ranges for A2 Satellite Provider
36 IPv4 ranges for A2 Satellite Provider
3 IPv6 ranges for AD Andorra
26 IPv4 ranges for AD Andorra
51 IPv6 ranges for AE United Arab Emirates
302 IPv4 ranges for AE United Arab Emirates
… and continues …
11 IPv6 ranges for ZM Zambia
64 IPv4 ranges for ZM Zambia
14 IPv6 ranges for ZW Zimbabwe
59 IPv4 ranges for ZW Zimbabwe
root@firegate2:/tmp/temp#
The script /usr/share/xt_geoip/xt_geoip_build it’s part of xtables-addons package that you may or may not find on your distribution. Search on google for that.
This will create LE and BE directories. Now you want to extract only the geo regions that you need (ex: CN,UA,TW,VN,VG,KP,VI,KR), like that:
root@firegate2:/tmp/temp# du -csh BE/CN.iv? BE/UA.iv? BE/TW.iv? BE/VN.iv? BE/VG.iv? BE/KP.iv? BE/VI.iv? BE/KR.iv? LE/CN.iv? LE/UA.iv? LE/TW.iv? LE/VN.iv? LE/VG.iv? LE/KP.iv? LE/VI.iv? LE/KR.iv?
36K BE/CN.iv4
48K BE/CN.iv6
24K BE/UA.iv4
16K BE/UA.iv6
8,0K BE/TW.iv4
4,0K BE/TW.iv6
4,0K BE/VN.iv4
4,0K BE/VN.iv6
4,0K BE/VG.iv4
4,0K BE/VG.iv6
4,0K BE/KP.iv4
0 BE/KP.iv6
4,0K BE/VI.iv4
4,0K BE/VI.iv6
8,0K BE/KR.iv4
4,0K BE/KR.iv6
36K LE/CN.iv4
48K LE/CN.iv6
24K LE/UA.iv4
16K LE/UA.iv6
8,0K LE/TW.iv4
4,0K LE/TW.iv6
4,0K LE/VN.iv4
4,0K LE/VN.iv6
4,0K LE/VG.iv4
4,0K LE/VG.iv6
4,0K LE/KP.iv4
0 LE/KP.iv6
4,0K LE/VI.iv4
4,0K LE/VI.iv6
8,0K LE/KR.iv4
4,0K LE/KR.iv6
352K totale
So, in this case we will need 352 Kb of router disk space. After we created the /usr/share/xt_geoip/BE/ and /usr/share/xt_geoip/LE/ directories on our router’s filesystem:
root@dam2ktplinkrouter:~# mkdir -p /usr/share/xt_geoip/BE /usr/share/xt_geoip/LE
We are going to copy those files on our OpenWRT router:
root@firegate2:/tmp/temp# scp LE/CN.iv? LE/UA.iv? LE/TW.iv? LE/VN.iv? LE/VG.iv? LE/KP.iv? LE/VI.iv? LE/KR.iv? root@192.168.10.1:/usr/share/xt_geoip/LE
root@192.168.10.1’s password:
CN.iv4 100% 35KB 109.8KB/s 00:00
CN.iv6 100% 47KB 76.0KB/s 00:00
UA.iv4 100% 23KB 108.4KB/s 00:00
UA.iv6 100% 15KB 87.9KB/s 00:00
TW.iv4 100% 4704 47.5KB/s 00:00
TW.iv6 100% 3104 77.8KB/s 00:00
VN.iv4 100% 3888 98.9KB/s 00:00
VN.iv6 100% 3584 87.9KB/s 00:00
VG.iv4 100% 536 40.1KB/s 00:00
VG.iv6 100% 224 20.0KB/s 00:00
KP.iv4 100% 40 3.9KB/s 00:00
KP.iv6 100% 0 0.0KB/s 00:00
VI.iv4 100% 392 30.6KB/s 00:00
VI.iv6 100% 160 14.4KB/s 00:00
KR.iv4 100% 8128 105.8KB/s 00:00
KR.iv6 100% 3616 87.0KB/s 00:00
root@firegate2:/tmp/temp# scp BE/CN.iv? BE/UA.iv? BE/TW.iv? BE/VN.iv? BE/VG.iv? BE/KP.iv? BE/VI.iv? BE/KR.iv? root@192.168.10.1:/usr/share/xt_geoip/BE/
root@192.168.10.1’s password:
CN.iv4 100% 35KB 114.4KB/s 00:00
CN.iv6 100% 47KB 66.0KB/s 00:00
UA.iv4 100% 23KB 115.2KB/s 00:00
UA.iv6 100% 15KB 74.9KB/s 00:00
TW.iv4 100% 4704 93.8KB/s 00:00
TW.iv6 100% 3104 75.7KB/s 00:00
VN.iv4 100% 3888 108.8KB/s 00:00
VN.iv6 100% 3584 76.2KB/s 00:00
VG.iv4 100% 536 40.3KB/s 00:00
VG.iv6 100% 224 20.0KB/s 00:00
KP.iv4 100% 40 4.1KB/s 00:00
KP.iv6 100% 0 0.0KB/s 00:00
VI.iv4 100% 392 33.1KB/s 00:00
VI.iv6 100% 160 15.4KB/s 00:00
KR.iv4 100% 8128 111.1KB/s 00:00
KR.iv6 100% 3616 71.8KB/s 00:00
root@firegate2:/tmp/temp#
OK, now we have our pieces of geo ip informations. We now need to install the iptables-mod-geoip from the LuCI web interface (or by hand if you like).
Now you can create your Firewall Traffic Rules. Go to Network -> Firewall -> Traffic Rules on the router’s LuCI web interface and add a custom traffic rule. In my case I created 2 that I called “CinamerdaMuoriUDPeTCP” and “CinamerdaMuoriICMP”, the first to block TCP and UDP and the second to block ICMP traffic.
After that you can setup your custom rule setting your protocol and address families, set “source zone” to WAN, “destination zone” to “Any zone (forward)”, action to DROP, and the “Extra arguments” field like this:
-m geoip –source-country CN,UA,TW,VN,VG,KP,VI,KR
You can check the image below.
OpenWRT Firewall geoip example
Now you can say hello to most chinaspammers and something like that, but don’t abuse, this is not ethic if you set up this on a server that offers some sort of public service.
21 Apr 17 How to create a sparse file from a block device
This is how to create a sparse file dump from a block device:
cp --sparse=always <(dd if=/dev/vg0/vmservice1 bs=8M iflag=direct | pv -pre --size=20G) /opt/backups/vmservice1.dat
14 Gen 17 Come creare un gateway IPv6 su Fastweb
Fastweb ha deciso per il momento di non fornire IPv6 nativo ai propri clienti, e inoltre da qualche giorno ha disabilitato il tunnel TSP (tsp-auth.ipv6.fastweb.it) il quale non risulta piu’ raggiungibile.
Visto che nel mio caso ho un router fastweb Argo 55+ su fibra 100, e il suddetto router non supporta IPv6, quando vado sulla MyFastPage e cerco di attivare il protocollo IPv6 il sistema mi dice che devo sostituire il router con un nuovo modello. Io NON voglio cambiare router perche’ lo considero estremamente stabile e performante.
Come posso quindi attivare IPv6 nella mia rete domestica senza dover cambiare router?
Fastweb porta IPv6 ai suoi utenti tramite 6rd (https://en.wikipedia.org/wiki/IPv6_rapid_deployment). Questo significa che e’ probabilmente possibile ottenere la subnet in tunnel anche su linux.
Ho preso uno dei miei raspberry pi con raspbian e ci ho installato il pacchetto radvd (sudo apt-get install radvd), poi nel mio /etc/network/interfaces ho messo questo:
iface eth0 inet6 static
address 2001:b07:27b:7b7b::1
netmask 64auto ipv6fastweb
iface ipv6fastweb inet6 v4tunnel
netmask 64
endpoint 81.208.50.214
up ip -6 route add default dev ipv6fastweb
down ip -6 route del default dev ipv6fastweb
Invece di usare come indirizzo ip 2001:b07:27b:7b7b::1 devi calcolarti il tuo a partire dal tuo IP pubblico fisso che ti ha fornito fastweb. Puoi ottenere il tuo ip pubblico ad esempio da qui: http://whatismyipaddress.com/
Facciamo finta che tu abbia l’IP 2.123.123.123, devi convertirlo in esadecimale, ad esempio cosi:
printf “%x%02x:%x%02x::\n” `echo 2.123.123.123 |tr . ” “`
Quello che otterrai, ad esempio 27b:7b7b:: va accodato al prefisso di fastweb (2001:b07:) e come suffisso accodi il numero 1.
In questo caso, quindi, l’IP diventa:
2001:b07:27b:7b7b::1 che e’ stato costruito da [2001:b07]:[b07:27b:7b7b]::[1]. La prima e’ fissa, la seconda dipende dal tuo IP pubblico e infine 1. Questo andra’ messo nella riga “address” nel file /etc/network/interfaces che ti dicevo prima e va anche messo nella direttiva “prefix” del radvd.conf, ma in ques’ultimo caso senza l’1 finale.
Crea il file /etc/radvd.conf e mettici dentro questo:
interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;prefix 2001:b07:27b:7b7b::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};RDNSS 2001:4860:4860::8888
{
AdvRDNSSLifetime 20;
};
};
Riavvia il raspberry pi e se tutto va bene, sempre nel raspberry pi dovrai avere una scheda di rete virtuale chiamata ipv6fastweb senza IP usabili ma che serve per creare il tunnel con il border gateway di fastweb (81.208.50.214). Se non funziona, prova a cercare un altro border gateway, magari chiedendo al numero verde o cercando su internet. A me funziona con questo. Metti quello giusto alla direttiva “endpoint” del file “interfaces”.
Poi avrai l’IP pubblico IPv6 che ti sei calcolato (nel caso di esempio 2001:b07:27b:7b7b::1/64) sulla scheda eth0 e avrai il tuo radvd che invia i router advertisement ipv6 alla tua rete.
Ogni PC nella tua rete che supporta IPv6 otterrà un IP pubblico nella subnet che ti sei calcolato, e sara’ raggiungibile direttamente da internet tramite IPv6.
Bello no? Ovviamente se abilitate questo dovete disabilitare IPv6 sul router di fastweb perche’ sara il vostro raspberry pi a fare da router ipv6.
Commentate sotto, mi raccomando! Fatemi sapere.
A me funziona perfettamente e sono molto felice. Credo che Fastweb dovrebbe creare una guida ufficiale su questo per il bene degli utenti. Ci ho messo 2 ore a farlo funzionare, con una guida ci avrei messo 2 minuti.
10 Ago 16 Shared memory and huge pages
This script can be used to determine how many shared memory is used by “oracle” user’s processes, and how many huge page memory is used:
t=0; for i in `ipcs -m | grep oracle | awk ‘{print $5}’`; do echo “Oracle consumed other $i bytes”; let t=t+$i; done
echo
echo “Consumed by oracle as shared memory segments: $t bytes”
let t=t/1024
let g=t/1024/1024;
echo “Memory conversions: ~ $t kbytes | ~ $g GB”hugepagetot=`cat /proc/meminfo | grep HugePages_Total | awk -F’:’ ‘{print $2}’`
hugepagefree=`cat /proc/meminfo | grep HugePages_Free | awk -F’:’ ‘{print $2}’`
let usedhugepages=hugepagetot-hugepagefree
let totkbinhigepage=usedhugepages*2048echo “Total hugepage usage in kb: $totkbinhigepage”
06 Mag 16 How to convert URI to query string parameters with mod_rewrite
You may need to convert URI levels to query string parameters, for example if want to be RESTful compliant with PHP.
Try this one:
RewriteEngine on RewriteRule ^/(\w+)/(\w+)$ /path_of_index.php?lev1=$1&lev2=$2 [QSA,L]
In this case the first URI level will be converted to a query string parameter called lev1, while the second will be converted to a query string parameter called lev2, each one with the respective values.
For example, the uri /user/list will be passed to index.php and will become index.php?lev1=user&lev2=list
An eventual query string will be passed, eventually overriding lev1 and lev2 parameters.
09 Apr 16 Getting in love with Docker Machine!!
Docker is becoming the “today standard” of lxs linux containers.
I think I will avoid learning Kubernetes to handle dockerized hosts, and I will study Docker Engine, Docker Swarm and Docker Machine and its REST APIs instead.
I started from here: https://docs.docker.com/machine/overview/
28 Nov 15 How to enable HTML5 video playback in firefox
If you use Mozilla Firefox and you want to view video streaming with HTML5 embedded players, you may need to enable some video functionality into the firefox configuration.
For some reason obscure to me Mozilla Firefox will not play some HTML5 based video streaming, and the embedded player will probably fall back to Adobe Flash based streaming (that is what I don’t want since it’s closed source and it’s a very insecure application).
For example, youtube has a HTML5 based player that you could choose to use instead of flv. You can verify and enable HTML5 based streaming functionality of your browser navigating to https://www.youtube.com/html5.
In my case (Firefox 64 bit on Linux Debian) the tabs “Media Source Extensions”, “MSE & H.264” and “MSE & WebM VP9” are disabled by default.
If you want to enable those functionalities simply write “about:config” into the firefox navigation bar.
You will be alerted to pay attention to not modify things or your warranty will be broken.
Just ignore the warning and proceed.
Now, if you did not already, switch the following configuration parameters to true:
media.fragmented-mp4.exposed
media.fragmented-mp4.ffmpeg.enabled
media.fragmented-mp4.gmp.enabled
media.mediasource.webm.enabled
media.mediasource.enabled
If you try to refresh the HTML5 based youtube player, you should hopefully have all video streaming tabs ready and enabled.
If you want, you can now switch the youtube default player to HTML5.
Follow me!
Stack Overflow
IPv6 Certified (he.net)