msgbartop
Blog di Dino Ciuffetti (Bernardino in realtà)
msgbarbottom

11 Feb 23 How to create persistent Queues, Exchanges, and DLXs on RabbitMQ to avoid loosing messages

What happens when you publish a message to an exchange in RabbitMQ with the wrong topic, or better, routing key? What happens if you send a message to the broker in a queue with a TTL policy or the TTL property is in the message itself and that TTL expires? What happens when a consumer discard your message got from the queue with no republish? What if a queue overflows due to a policy?

It’s simple, the broker will simply discard your message forever.

If this thing will make you mad like it does to me this blog article is for you. Here I will tell you how to create a simple tree of queues, DLX and policies to overcome this problem.

I think that starting with commands and examples is better than 10000 written words, and since I don’t have any ads on my blog I don’t have to write a long article to get money from the ads, so here we are.

I consider your RabbitMQ installation and your admin account ready, so we start with the commands.

# Create your VirtualHost
rabbitmqctl add_vhost vhtest --description "Your VH" --default-queue-type classic

# Give your admin user permissions to to everything on your virtualhost
rabbitmqctl set_permissions --vhost vhtest admin '.*' '.*' '.*'

# Create the user that will publish messages to the exchange
rabbitmqctl add_user testuserpub yourpassword
1

# Create the user that will subscribe to your queue to read messages
rabbitmqctl add_user testusersub yourpassword2

Now, we have 3 users (admin, testuserpub and testusersub) and a virtualhost (vhtest). We are ready to create 2 DLX, one to handle overflow, expired TTL and discarded messages, the other to handle messages sent with the wrong routing key. A DLX (or Dead Letter Exchange) is a particular exchange that is designed to handle dead lettered (discarded) messages.

# Create the DLX to handle overflowed, expired or discarded by consumers
rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQoverfloworttl type=headers internal=true

# Create the DLX to handle messages with wrong routing key
rabbitmqadmin declare exchange --vhost=vhtest name=DLXexQwrongtopic type=fanout internal=true

We’ll now declare and bind queues to the first DLX using three different policies

rabbitmqadmin declare queue --vhost=vhtest name=DLXquQoverflow
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQttl
rabbitmqadmin declare queue --vhost=vhtest name=DLXquQrejected
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQoverflow arguments='{"x-first-death-reason": "maxlen", "x-match": "all-with-x"}'
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQttl arguments='{"x-first-death-reason": "expired", "x-match": "all-with-x"}'
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQoverfloworttl destination=DLXquQrejected arguments='{"x-first-death-reason": "rejected", "x-match": "all-with-x"}'

And now we’ll declare and bind queues to the second DLX to handle messages with wrong topic (routing key)

rabbitmqadmin declare queue --vhost=vhtest name=DLXquQwrongtopic
rabbitmqadmin declare binding --vhost=vhtest source=DLXexQwrongtopic destination=DLXquQwrongtopic

Now we have 1 DLX with 3 queues and another DLX with 1 queue bound. The first will route expired, discarded and overflowed messages to the respective queues (DLXquQttl, DLXquQoverflow, DLXquQrejected), the second will route messages with invalid routing key to the respective queue (DLXquQwrongtopic).

Now we are going to create our main queue and the normal Exchange that will send message to it

rabbitmqadmin declare queue --vhost=vhtest name=quQ
rabbitmqadmin declare exchange --vhost=vhtest name=exQ type=direct

In this example, we want to route all messages with routing key NBE

rabbitmqadmin declare binding --vhost=vhtest source=exQ destination=quQ routing_key=NBE

We now want to create the policy that is needed to associate the wrong topic DLX to our main exchange

rabbitmqctl set_policy --vhost vhtest wrongtopicQ1 "^exQ$" '{"alternate-exchange":"DLXquQwrongtopic"}' --apply-to exchanges

This is an example policy to set limits to 100 messages, 1073741824 bytes, 30 seconds TTL to the quQ queue.

rabbitmqctl set_policy --vhost vhtest shorttimedqunbe '^quQ$' '{"max-length":100,"max-length-bytes":1073741824,"message-ttl":30000,"overflow":"reject-publish-dlx","dead-letter-exchange":"DLXexQoverfloworttl"}' --priority 0 --apply-to queues

Going to give proper permissions to our publish and subscriber users. The user testuserpub can only write to its exchange, while testusersub can read from its queue. No other permissions here.

rabbitmqctl set_permissions --vhost vhtest testuserpub '' '^exQ$' ''
rabbitmqctl set_permissions --vhost vhtest testusersub '' '' '^quQ$'

Mission complete. Please try this at home and write to the comments below! Happy RabbitMQ hacking!


09 Apr 16 Getting in love with Docker Machine!!

Docker is becoming the “today standard” of lxs linux containers.
I think I will avoid learning Kubernetes to handle dockerized hosts, and I will study Docker Engine, Docker Swarm and Docker Machine and its REST APIs instead.

I started from here: https://docs.docker.com/machine/overview/

08 Feb 15 Never say the truth to job interviews

After years of TI technical support middleware expert (and any offers received after a job interview that I had in june) it’s now time to explore new horizons, so I’m searching some new job.
In the meantime I’ll do the very minimum here at TI: I’ll turn my phone off and no more extra help to anyone.

NOTE1: never say the truth to job interviews;
NOTE2: never help heads and collegues;
NOTE3: never say you know something more or you’ll get new work for free;
NOTE4: always be cynic

02 Apr 14 Adafruit 4-Digit 7-Segment Display Backpack on raspberry pi in C

In my previous blog post I published a TSL2561 light sensor driver in C for Raspberry PI. In this article I will publish a user space C driver for Adafruit 4-digit 7-segment display.

This is based on a HT16K33 led driver IC, that it’s a I2C driven RAM mapping 16*8 LED controller driver.

The driver I’m posting it’s valid for the adafruit circuit only, since it’s completely based on the electronic schematic they realized.
Don’t use the driver with other circuits, since the display could not function properly.
Basically the adafruit 7-segment backpack (http://www.adafruit.com/products/879) uses 8 (rows) * 5 (columns) HT16K33 lines to drive its leds. The column number 1 is dedicated to the first digit, the second column is dedicated to the second digit, the third column is attached to the colon sign in the middle of the 4 digits, the fourth column is attached to the third digit, and the fifth colum to the fourth display digit.

adafruit_7seg_schematic

While each row drives a single led of the given column.

The display columns 0, 1, 3, 4 can show numbers and some letters (A-F, n, o, i, l, L, etc…) plus a decimal point, while the column 2 can only show a colon sign (:).
A number or a letter for each digit is composed by 7 led segments, so the possibilities are few… but not so few after all (check 7seg.txt file attachment for more details on letter composition).

So, now comes the fun. How can I access the led driver memory to light display digits in C? Adafruit releases proof of concept libraries in C and python, but they don’t seem to run on my raspberry pi.
Since I am too lazy to port their code with external dependencies, I decided to write my own library in C.

#include "7seg_bp_ada.h"

/* prepare the backpack driver
(the first parameter is the raspberry pi i2c master controller attached to the HT16K33, the second is the i2c selection jumper)
The i2c selection address can be one of HT16K33_ADDR_01 to HT16K33_ADDR_08
*/
HT16K33 led_backpack1 = HT16K33_INIT(1, HT16K33_ADDR_01);

/* initialize the backpack */
rc = HT16K33_OPEN(&led_backpack1);

/* power on the ht16k33 */
HT16K33_ON(&led_backpack1);

/* make it shining bright */
HT16K33_BRIGHTNESS(&led_backpack1, 0x0F);

/* make it not blinking */
HT16K33_BLINK(&led_backpack1, HT16K33_BLINK_OFF);

/* power on the display */
HT16K33_DISPLAY(&led_backpack1, HT16K33_DISPLAY_ON);

/* Say hello */
HT16K33_UPDATE_DIGIT(&led_backpack1, 0, 'H', 0); // first digit
HT16K33_UPDATE_DIGIT(&led_backpack1, 1, 'E', 0); // second digit
// turn off the colon sign in the middle of the 4 digits
HT16K33_UPDATE_DIGIT(&led_backpack1, 2, HT16K33_COLON_OFF, 0);
HT16K33_UPDATE_DIGIT(&led_backpack1, 3, '#', 0); // third digit
HT16K33_UPDATE_DIGIT(&led_backpack1, 4, 'o', 0); // fourth digit
HT16K33_COMMIT(&led_backpack1); // commit to the display memory

// call this if you want to shut down the device (power saving mode)
// HT16K33_OFF(&led_backpack1);

/* close things (the display remains in the conditions left) */
HT16K33_CLOSE(&led_backpack1);

I decided to release the software with the liberal apache 2 license, so feel free to use this software inside your commercial, non free software / firmware.

Below you will find the files .c and .h that you can embed into your project.
It’s helpful for me, and I hope it will be helpful for you.

Ciao, Dino.

Note: on Raspberry PI OS (and debian) you need libi2c-dev (apt install libi2c-dev) before compiling.

gcc -Wall -O2 -o 7seg_bp_ada.o -c 7seg_bp_ada.c
gcc -Wall -O2 -o 7seg_bp_ada_test.o -c 7seg_bp_ada_test.c
gcc -Wall -O2 -o 7seg_bp_ada_test 7seg_bp_ada.o -li2c 7seg_bp_ada_test.o

7seg_bp_ada.c
7seg_bp_ada.h
7seg_bp_ada_test.c

19 Mar 14 TSL2561 light sensor on Raspberry pi in C

After I bought a new TSL2561 digital light sensor from Adafruit, I found that the very cool and small device cannot be accessed directly from linux (rasbian doesn’t have it’s kernel module compiled). Since I didn’t want to cross recompile my whole raspberry pi kernel just to have the tsl2563.ko driver enabled, and since it seems that raspbian does not relase genuine kernel headers to just compile custom kernel modules, I decided to write a user space simple library driver in C.

I found out that Adafruit relases proof of concept libraries written in C++ and python to access its hardware devices, the problem is that the c++ version is ready for arduino but it was not so directly usable for my raspberry pi. It also makes use of an adafruit unified sensor library and other external stuff. Since I am too lazy I decided yesterday to write a new simple library in plain C without external dependencies, just ready for my raspberry pi.

This is the arduino version that inspired me: https://github.com/adafruit/TSL2561-Arduino-Library
This is another cool blog post that inspired me (it now seems dead!!): http://russelldavis.org/2013/03/23/raspberryhunt-part-2/

This is an example:

/* prepare the sensor
(the first parameter is the raspberry pi i2c master controller attached to the TSL2561, the second is the i2c selection jumper)
The i2c selection address can be one of: TSL2561_ADDR_LOW, TSL2561_ADDR_FLOAT or TSL2561_ADDR_HIGH
*/
TSL2561 light1 = TSL2561_INIT(1, TSL2561_ADDR_FLOAT);

/* initialize the sensor */
rc = TSL2561_OPEN(&light1);

/* sense the luminosity from the sensor (lux is the luminosity taken in "lux" measure units)
the last parameter can be 1 to enable library auto gain, or 0 to disable it */
rc = TSL2561_SENSELIGHT(&light1, &broadband, &ir, &lux, 1);

TSL2561_CLOSE(&light1);

Compile:

gcc -Wall -O2 -o TSL2561.o -c TSL2561.c
gcc -Wall -O2 -o TSL2561_test.o -c TSL2561_test.c
gcc -Wall -O2 -o TSL2561_test TSL2561.o TSL2561_test.o

The output is like this:

root@rasponi:~/test/gpio# ./TSL2561_test
Test. RC: 0(Success), broadband: 141, ir: 34, lux: 12

As you can see it’s very easy at this point to get the light measures in C. Just include TSL2561.c and TSL2561.h inside your project and use the public APIs to setup and sense the IC.

I decided to release the code with the liberal apache v2 license, so feel free to include it into your commercial projects if you like.

It’s useful for me, and I hope that it can be useful to you too. Obviously it comes with absolutely no warranty.

p.s.1: I left the hardware stuff out of this article (just attach +vcc, gnd and i2c bus to the sensor
p.s.2: you have to load two kernel modules to get i2c bus working on you Raspberry pi:

modprobe i2c_bcm2708
modprobe i2c_dev

Ciao, Dino.

TSL2561.c
TSL2561.h
TSL2561_test.c

This is an example on how to use all 3 sensors on the same i2c bus:

#include <stdio.h>
#include <string.h>
#include "TSL2561.h"

int main() {
	int i;
	int rc;
	uint16_t broadband, ir;
	uint32_t lux=0;
	TSL2561 lights[3]; // we can handle 3 sensors
	
	// prepare the sensors
	// (the first parameter is the raspberry pi i2c master controller attached to the TSL2561, the second is the i2c selection jumper)
	// The i2c selection address can be one of: TSL2561_ADDR_LOW, TSL2561_ADDR_FLOAT or TSL2561_ADDR_HIGH
	
	// prepare all sensors
	/* cannot assign that way
	lights[0] = TSL2561_INIT(1, TSL2561_ADDR_LOW);
	lights[1] = TSL2561_INIT(1, TSL2561_ADDR_FLOAT);
	lights[2] = TSL2561_INIT(1, TSL2561_ADDR_HIGH);
	*/
	
	// initialize at runtime instead
	// FIRST SENSOR --> TSL2561_ADDR_LOW
	lights[0].adapter_nr=1;						// change this according to your i2c bus
	lights[0].sensor_addr=TSL2561_ADDR_LOW;				// don't change this
	lights[0].integration_time=TSL2561_INTEGRATIONTIME_402MS;	// don't change this
	lights[0].gain=TSL2561_GAIN_16X;				// don't change this
	lights[0].adapter_fd=-1;					// don't change this
	lights[0].lasterr=0;						// don't change this
	bzero(&lights[0].buf, sizeof(lights[0].buf));			// don't change this
	
	// SECOND SENSOR --> TSL2561_ADDR_FLOAT
	lights[1].adapter_nr=1;						// change this according to your i2c bus
	lights[1].sensor_addr=TSL2561_ADDR_FLOAT;			// don't change this
	lights[1].integration_time=TSL2561_INTEGRATIONTIME_402MS;	// don't change this
	lights[1].gain=TSL2561_GAIN_16X;				// don't change this
	lights[1].adapter_fd=-1;					// don't change this
	lights[1].lasterr=0;						// don't change this
	bzero(&lights[1].buf, sizeof(lights[1].buf));			// don't change this
	
	// THIRD SENSOR --> TSL2561_ADDR_HIGH
	lights[2].adapter_nr=1;						// change this according to your i2c bus
	lights[2].sensor_addr=TSL2561_ADDR_HIGH;			// don't change this
	lights[2].integration_time=TSL2561_INTEGRATIONTIME_402MS;	// don't change this
	lights[2].gain=TSL2561_GAIN_16X;				// don't change this
	lights[2].adapter_fd=-1;					// don't change this
	lights[2].lasterr=0;						// don't change this
	bzero(&lights[2].buf, sizeof(lights[2].buf));			// don't change this
	
	// initialize the sensors
	for(i=0; i<3; i++) {
		rc = TSL2561_OPEN(&lights[i]);
		if(rc != 0) {
			fprintf(stderr, "Error initializing TSL2561 sensor %i (%s). Check your i2c bus (es. i2cdetect)\n", i+1, strerror(lights[i].lasterr));
			return 1;
		}
		// set the gain to 1X (it can be TSL2561_GAIN_1X or TSL2561_GAIN_16X)
		// use 16X gain to get more precision in dark ambients, or enable auto gain below
		rc = TSL2561_SETGAIN(&lights[i], TSL2561_GAIN_1X);
		
		// set the integration time 
		// (TSL2561_INTEGRATIONTIME_402MS or TSL2561_INTEGRATIONTIME_101MS or TSL2561_INTEGRATIONTIME_13MS)
		// TSL2561_INTEGRATIONTIME_402MS is slower but more precise, TSL2561_INTEGRATIONTIME_13MS is very fast but not so precise
		rc = TSL2561_SETINTEGRATIONTIME(&lights[i], TSL2561_INTEGRATIONTIME_101MS);
	}
	
	// you can now sense each sensor when you like
	for(i=0; i<3; i++) {
		// sense the luminosity from the sensors (lux is the luminosity taken in "lux" measure units)
		// the last parameter can be 1 to enable library auto gain, or 0 to disable it
		rc = TSL2561_SENSELIGHT(&lights[i], &broadband, &ir, &lux, 1);
		printf("Test sensor %i. RC: %i(%s), broadband: %i, ir: %i, lux: %i\n", i+1, rc, strerror(lights[i].lasterr), broadband, ir, lux);
	}
	
	// when you have finisched, you can close things
	for(i=0; i<3; i++) {
		TSL2561_CLOSE(&lights[i]);
	}
	
	return 0;
}

05 Dic 13 As you can see, this site is reachable via IPv6!

If you are using IPv6 (like me) you can see that this blog is reachable via IPv6. Pretty cool!

ipv6 ready

02 Dic 12 New NuvolaBase Dashboard

Tonight at 03.00 GTM the NuvolaBase team publicly released the new NuvolaBase Dashboard.
As you may know, with NuvolaBase you can handle your private database on the cloud.

The new dashboard aims to be simple, stable and powerful. You can login using your google, twitter, facebook, linkedin account.
In the next days the NuvolaBase guys will release many new cool features like a powerful REST API to handle your databases in the cloud from your application.

This is the official article on the NuvolaBase blog: http://nuvolabase.blogspot.it/2012/12/nuvolabase-dashboard-upgrade.html

20 Apr 12 Avaaz urgent petitions to sign NOW and quickly!

Urgent Online Petitions to sign now and quicky!! Please wide spread!!

https://secure.avaaz.org/it/monti_save_our_internet/

In qualità di cittadini preoccupati, le chiediamo di fermare immediatamente tutte le iniziative del governo volte a dare all’Autorità per le Garanzie nelle Comunicazioni (AGCOM) il potere di censurare siti internet senza mandato del giudice. Le chiediamo inoltre di riaffermare pubblicamente che solo il Parlamento può approvare leggi che incidono sui nostri diritti fondamentali, inclusa la libertà di espressione, come affermato dalla Costituzione. Ci affidiamo a lei per proteggere la libertà di Internet in quanto pilastro fondamentale della nostra democrazia.

https://secure.avaaz.org/en/stop_cispa/

As concerned global citizens, we urge you to immediately drop the Cyber Intelligence Sharing and Protection Act (CISPA). Our democracy and civil liberties are under threat from the excessive and unnecessary Internet surveillance powers it grants. The Internet is a crucial tool for people around the world to exchange ideas and work collectively to build the world we all want. We urge you to show true global leadership and do all you can to protect our Internet freedom.

08 Mar 12 How to execute a HTTP/Rest Query to NuvolaBase distributed database with PHP

As previously said, nuvolabase.com is a great service that permits you to have a distributed nosql document database in the cloud. This is very cool: think each time you would had the need of a database always available in the cloud that you would access via simple HTTP/Rest queries. The possibilities are endless.

Here is a very simple but powerful PHP curl agent to submit commands (queries) to nuvolabase via HTTP.

<?php
/*
* Author: Dino Ciuffetti <dino@tuxweb.it>
* Object: Execute a remote query to a distributed database on nuvolabase.com (free account) using HTTP (OrientDB REST API)
*/

/* user configurable parameters */
$nuvolabasedb = ‘db$free$youruser$yourdb’;
$command = ‘select from yourclass’;
$user = ‘admin’;
$password = ‘qwerty’;
$useragent = “NuvolaBase PHP REST agent/v0.8 (compatible; Mozilla 4.0; MSIE 5.5; http://www.nuvolabase.com/)”;
/* END of user configurable parameters */

$nuvolabasehost = ‘studio.nuvolabase.com’;
$url = ‘http://’.$user.’:’.$password.’@’.’studio.nuvolabase.com/command/’.$nuvolabasedb.’/sql/’;

$ch = curl_init();

// set user agent
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);

// return the result or false in case of errors
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

// set the target url
curl_setopt($ch, CURLOPT_URL, $url);

// do basic login authentication
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

// howmany parameter to post
curl_setopt($ch, CURLOPT_POST, 1);

// the post data to send
curl_setopt($ch, CURLOPT_POSTFIELDS, $command);

// execute curl,fetch the result and close curl connection
$res = curl_exec ($ch);
curl_close ($ch);

// display result
if ($res !== FALSE);
print_r (json_decode($res));

?>

Please use the attached file.

test.php

12 Dic 11 How to quickly install courier-mta mail server from source on debian (for the impatients)

What it follows is a quick n’ dirty but working list of things to do to correctly install you brand new courier mta mail server from source on a linux debian system (or ubuntu). This is for the impatients that don’t want to read the courier installation manual page (http://www.courier-mta.org/install.html). You can download the last stable courier packages from here: http://www.courier-mta.org/download.php.
You only have to download those three software archive files:

  1. Courier
  2. Courier authentication library
  3. Courier unicode library

You’ll need to be root and have an internet connection to install dependencies debian packages.
First of all you have to enable the EN_US-utf8 locale, or “make check” will fail. You can do it with:

dpkg-reconfigure locales

Ready to install? Ok. This is how I have done:

groupadd courier
useradd -m -g courier courier
groupadd vmail
useradd -g vmail -d /opt/courier -m vmail

apt-get install build-essential
apt-get install libldap2-dev
apt-get install ldap-utils
apt-get install slapd
apt-get install libmysqlclient-dev
apt-get install libpcre++-dev libpcre3-dev
apt-get install libidn11-dev
apt-get install libgdbm-dev
apt-get install libdb-dev
apt-get install libgamin-dev
apt-get install libssl-dev
apt-get install libgnutls28-dev
apt-get install expect
apt-get install libperl-dev
apt-get install libltdl-dev
apt-get install libsqlite3-dev

tar jxvf courier-unicode-x.x.tar.bz2
cd courier-unicode-x.x

./configure –prefix=/opt/courier/unicode

make
make install

cd ..

tar jxvf courier-authlib-0.xx.0.tar.bz2
cd courier-authlib-0.xx.0

export CFLAGS=”-I/opt/courier/unicode/include”
export LDFLAGS=”-L/opt/courier/unicode/lib”

./configure \
–prefix=/opt/courier/authlib \
–without-authvchkpw \
–without-authpgsql \
–with-mysql-libs=/usr \
–with-mysql-includes=/usr \
–with-mailuser=vmail \
–with-mailgroup=vmail

make
make install

cd ..

tar jxvf courier-0.xx.0.tar.bz2
chown -R courier:courier courier-0..0

su – courier
cd courier-0.xx.0

export COURIERAUTHCONFIG=/opt/courier/authlib/bin/courierauthconfig
export CFLAGS=”-I/opt/courier/authlib/include -I/opt/courier/unicode/include”
export CPPFLAGS=”-I/opt/courier/authlib/include -I/opt/courier/unicode/include”
export LDFLAGS=”-L/opt/courier/authlib/lib -L/opt/courier/unicode/lib”

./configure \
–prefix=/opt/courier –with-locking-method=fcntl –without-explicitsync \
–with-qdircount=20 –with-random=/dev/urandom \
–without-ispell –disable-autorenamesent –enable-mimetypes

make
make check
exit
make install-strip
make install-configure >upgrade.log

/opt/courier/sbin/showmodules
/opt/courier/sbin/makesmtpaccess

OK. If everything gone OK, the installation stuff was terminated.
The configuration stuff is another complex thing that I cannot cover now here… may be another time.
Now, continue to read installation manual from here: http://www.courier-mta.org/install.html#aliases

This procedure was last tested with courier 0.75.0 on debian 8.4, but should be OK with newer versions too.
The software will be installed in /opt/courier.

I hope that it will help someone.
Ciao, Dino.