msgbartop
Blog di Dino Ciuffetti (Bernardino in realtà)
msgbarbottom

15 Nov 14 Dump and restore block device data on the fly by the network

Sometimes you may need to copy data from a block device (or LVM logical volume or snapshot) from one server to another., but you don’t want to dump the image to disk, move to the other server, then import. You may need (or just want) to copy on the fly, transfering data on the net.

To do this, and have ETA on the operation you need the pv executable. The command nc is used to stream data on the network, while pigz is used to compress data (gzip uses just one CPU, while pigz uses all available CPU, and it’s much faster).

On the origin server (server1) you have a block device (lvm logical volume in this case) called /dev/vg0/vm-111-disk-1, while on the destination server (server2) you want to overwrite a LVM logical volume called /dev/vg0/vm-112-disk-1 with data coming from the origin server.
To do this, assuming the device is big 20GB, you may run those commands:

Server side (destination server, server2, ip 192.168.0.2):

nc -l -n -p 2102 -q 2 | pigz -d | pv -pre –size=20G | dd iflag=fullblock bs=512k of=/dev/vg0/vm-112-disk-1

Client side (origin server, server1, 192.168.0.1):

dd if=/dev/vg0/vm-111-disk-1 bs=512k | pv -pre –size=20G | pigz | nc -q 2 192.168.0.2 2102

Data will be read, compressed, transfered on the network on (port TCP 2102 on our case, from 192.168.0.1 to 192.168.0.2), uncompressed on the destination server and restored on disk, and you’ll have ETA and progress indication:

Output server side (destination server, server2):

root@server2 ~ # nc -l -n -p 2102 -q 2 | pigz -d | pv -pre –size=20G | dd iflag=fullblock bs=512k of=/dev/vg0/vm-112-disk-1
[71.2MB/s] [=========================================================================================================================================>] 100%
40960+0 records in
40960+0 records out
21474836480 bytes (21 GB) copied, 296.436 s, 72.4 MB/s

Output client side (origin server, server1):

root@server1 ~ # dd if=/dev/vg0/vm-111-disk-1 bs=512k | pv -pre –size=20G | pigz | nc -q 2  192.168.0.2 2102
[72.2MB/s]
[=========================================================================================================================================>]
100%
40960+0 records
40960+0 records out
21474836480 bytes (21 GB) copied, 283.531 s, 75.7 MB/s

30 Ott 14 WPA2 connection without NetworkManager in Debian linux

This is a memo that I can use to remember how to enable WPA2 protected WiFI connections with debian without using NetworkManager.

All that you have to do is:

  • create the file /etc/wpa_supplicant/wpa_supplicant.conf with the following content:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=0
country=IT
ap_scan=2

network={
ssid=”My WiFI SSID”
psk=”mysupersecret password”
bssid=””
proto=RSN
key_mgmt=WPA-PSK
pairwise=CCMP TKIP
group=TKIP
}

  • if you want to set a static IP, add the following content to the file /etc/network/interfaces

iface wlan0 inet static
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
address 192.168.100.7
netmask 255.255.255.0
network 192.168.100.0
broadcast 192.168.100.255

  • if, instead, you want a dynamic address assigned by a DHCP server, add the following to /etc/network/interfaces

iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

  • now, setup the connection with this command by root (sudo before the command if you are on ubuntu):

ifup wlan0

  • when you have finisched with the connection, close it up with the command by root (sudo before the command if you are on ubuntu)

ifdown wlan0

 

28 Ott 14 Best rsync options to mirror a remote directory

If you want to mirror a remote directory via SSH, you may want to use the wonderful rsync command.

The rsync executable has many options, so, which is the correct option list to make an exact copy of a remote directory, maintaining permissions, ownerships, timestampts, copying only the modified files, and updating only the pieces of modified files?

Let me begin with an example. We want to full mirror the directory /mystuff on server 1.2.3.4 into /mystufflocal. The files deleted on 1.2.3.4 from the previous rsync will be removed locally too, so pay attention! If you don’t want to locally remove deleted files you can remove the “–delete” option.
If you want to compress the stream in transit you can add the “-z” option.
All we have to do is:

rsync -vartuh –inplace –delete –progress –stats -e “ssh -carcfour128” root@1.2.3.4:/mystuff/ /mystufflocal/

The trailing slashes are important because are used by rsync to understand precisely what should be transferred and where.

27 Ago 14 Squid: how to get rid of “All url_rewriter processes are busy”

If you check your squid forward (transparent or not) proxy log files you may found errors like those:

WARNING: All url_rewriter processes are busy.
WARNING: up to 6 pending requests queued

This is true if you use the directive “url_rewrite_program”, for example with SquidGuard.
In this case, squid tells you that it cannot spawn more helper processes to externally scan your requests in parallel, so it’s queuing your requests.
This is not a great problem, but you may be annoyed to see this stuff in your log files, or there are cases in which the default may be too low!

You may raise this limit with the parameter called url_rewrite_children.

To solve, add something like this to your squid.conf configuration file, and restart squid:

url_rewrite_children 32

Ciao, Dino.

21 Ago 14 How to enable apache NameVirtualHost with SSL

If you want to create name based virtualhosts in apache with SSL Certificates, you need openssl with SNI and TLS support (0.9.8f or better) and good apache 2.2.X version.

It’s a simple task, after you’ve read this official article: https://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI

12 Giu 14 How to declare a read only variable in bash

I didn’t know that it was possible to declare a read only variable in bash.

It’s as simple as to run the following statement:

declare -r a=10

This will create a read only variable called $a with value 10 that you cannot overwrite or unset.
Cool!!

02 Apr 14 Adafruit 4-Digit 7-Segment Display Backpack on raspberry pi in C

In my previous blog post I published a TSL2561 light sensor driver in C for Raspberry PI. In this article I will publish a user space C driver for Adafruit 4-digit 7-segment display.

This is based on a HT16K33 led driver IC, that it’s a I2C driven RAM mapping 16*8 LED controller driver.

The driver I’m posting it’s valid for the adafruit circuit only, since it’s completely based on the electronic schematic they realized.
Don’t use the driver with other circuits, since the display could not function properly.
Basically the adafruit 7-segment backpack (http://www.adafruit.com/products/879) uses 8 (rows) * 5 (columns) HT16K33 lines to drive its leds. The column number 1 is dedicated to the first digit, the second column is dedicated to the second digit, the third column is attached to the colon sign in the middle of the 4 digits, the fourth column is attached to the third digit, and the fifth colum to the fourth display digit.

adafruit_7seg_schematic

While each row drives a single led of the given column.

The display columns 0, 1, 3, 4 can show numbers and some letters (A-F, n, o, i, l, L, etc…) plus a decimal point, while the column 2 can only show a colon sign (:).
A number or a letter for each digit is composed by 7 led segments, so the possibilities are few… but not so few after all (check 7seg.txt file attachment for more details on letter composition).

So, now comes the fun. How can I access the led driver memory to light display digits in C? Adafruit releases proof of concept libraries in C and python, but they don’t seem to run on my raspberry pi.
Since I am too lazy to port their code with external dependencies, I decided to write my own library in C.

#include "7seg_bp_ada.h"

/* prepare the backpack driver
(the first parameter is the raspberry pi i2c master controller attached to the HT16K33, the second is the i2c selection jumper)
The i2c selection address can be one of HT16K33_ADDR_01 to HT16K33_ADDR_08
*/
HT16K33 led_backpack1 = HT16K33_INIT(1, HT16K33_ADDR_01);

/* initialize the backpack */
rc = HT16K33_OPEN(&led_backpack1);

/* power on the ht16k33 */
HT16K33_ON(&led_backpack1);

/* make it shining bright */
HT16K33_BRIGHTNESS(&led_backpack1, 0x0F);

/* make it not blinking */
HT16K33_BLINK(&led_backpack1, HT16K33_BLINK_OFF);

/* power on the display */
HT16K33_DISPLAY(&led_backpack1, HT16K33_DISPLAY_ON);

/* Say hello */
HT16K33_UPDATE_DIGIT(&led_backpack1, 0, 'H', 0); // first digit
HT16K33_UPDATE_DIGIT(&led_backpack1, 1, 'E', 0); // second digit
// turn off the colon sign in the middle of the 4 digits
HT16K33_UPDATE_DIGIT(&led_backpack1, 2, HT16K33_COLON_OFF, 0);
HT16K33_UPDATE_DIGIT(&led_backpack1, 3, '#', 0); // third digit
HT16K33_UPDATE_DIGIT(&led_backpack1, 4, 'o', 0); // fourth digit
HT16K33_COMMIT(&led_backpack1); // commit to the display memory

// call this if you want to shut down the device (power saving mode)
// HT16K33_OFF(&led_backpack1);

/* close things (the display remains in the conditions left) */
HT16K33_CLOSE(&led_backpack1);

I decided to release the software with the liberal apache 2 license, so feel free to use this software inside your commercial, non free software / firmware.

Below you will find the files .c and .h that you can embed into your project.
It’s helpful for me, and I hope it will be helpful for you.

Ciao, Dino.

Note: on Raspberry PI OS (and debian) you need libi2c-dev (apt install libi2c-dev) before compiling.

gcc -Wall -O2 -o 7seg_bp_ada.o -c 7seg_bp_ada.c
gcc -Wall -O2 -o 7seg_bp_ada_test.o -c 7seg_bp_ada_test.c
gcc -Wall -O2 -o 7seg_bp_ada_test 7seg_bp_ada.o -li2c 7seg_bp_ada_test.o

7seg_bp_ada.c
7seg_bp_ada.h
7seg_bp_ada_test.c

19 Mar 14 TSL2561 light sensor on Raspberry pi in C

After I bought a new TSL2561 digital light sensor from Adafruit, I found that the very cool and small device cannot be accessed directly from linux (rasbian doesn’t have it’s kernel module compiled). Since I didn’t want to cross recompile my whole raspberry pi kernel just to have the tsl2563.ko driver enabled, and since it seems that raspbian does not relase genuine kernel headers to just compile custom kernel modules, I decided to write a user space simple library driver in C.

I found out that Adafruit relases proof of concept libraries written in C++ and python to access its hardware devices, the problem is that the c++ version is ready for arduino but it was not so directly usable for my raspberry pi. It also makes use of an adafruit unified sensor library and other external stuff. Since I am too lazy I decided yesterday to write a new simple library in plain C without external dependencies, just ready for my raspberry pi.

This is the arduino version that inspired me: https://github.com/adafruit/TSL2561-Arduino-Library
This is another cool blog post that inspired me (it now seems dead!!): http://russelldavis.org/2013/03/23/raspberryhunt-part-2/

This is an example:

/* prepare the sensor
(the first parameter is the raspberry pi i2c master controller attached to the TSL2561, the second is the i2c selection jumper)
The i2c selection address can be one of: TSL2561_ADDR_LOW, TSL2561_ADDR_FLOAT or TSL2561_ADDR_HIGH
*/
TSL2561 light1 = TSL2561_INIT(1, TSL2561_ADDR_FLOAT);

/* initialize the sensor */
rc = TSL2561_OPEN(&light1);

/* sense the luminosity from the sensor (lux is the luminosity taken in "lux" measure units)
the last parameter can be 1 to enable library auto gain, or 0 to disable it */
rc = TSL2561_SENSELIGHT(&light1, &broadband, &ir, &lux, 1);

TSL2561_CLOSE(&light1);

Compile:

gcc -Wall -O2 -o TSL2561.o -c TSL2561.c
gcc -Wall -O2 -o TSL2561_test.o -c TSL2561_test.c
gcc -Wall -O2 -o TSL2561_test TSL2561.o TSL2561_test.o

The output is like this:

root@rasponi:~/test/gpio# ./TSL2561_test
Test. RC: 0(Success), broadband: 141, ir: 34, lux: 12

As you can see it’s very easy at this point to get the light measures in C. Just include TSL2561.c and TSL2561.h inside your project and use the public APIs to setup and sense the IC.

I decided to release the code with the liberal apache v2 license, so feel free to include it into your commercial projects if you like.

It’s useful for me, and I hope that it can be useful to you too. Obviously it comes with absolutely no warranty.

p.s.1: I left the hardware stuff out of this article (just attach +vcc, gnd and i2c bus to the sensor
p.s.2: you have to load two kernel modules to get i2c bus working on you Raspberry pi:

modprobe i2c_bcm2708
modprobe i2c_dev

Ciao, Dino.

TSL2561.c
TSL2561.h
TSL2561_test.c

This is an example on how to use all 3 sensors on the same i2c bus:

#include <stdio.h>
#include <string.h>
#include "TSL2561.h"

int main() {
	int i;
	int rc;
	uint16_t broadband, ir;
	uint32_t lux=0;
	TSL2561 lights[3]; // we can handle 3 sensors
	
	// prepare the sensors
	// (the first parameter is the raspberry pi i2c master controller attached to the TSL2561, the second is the i2c selection jumper)
	// The i2c selection address can be one of: TSL2561_ADDR_LOW, TSL2561_ADDR_FLOAT or TSL2561_ADDR_HIGH
	
	// prepare all sensors
	/* cannot assign that way
	lights[0] = TSL2561_INIT(1, TSL2561_ADDR_LOW);
	lights[1] = TSL2561_INIT(1, TSL2561_ADDR_FLOAT);
	lights[2] = TSL2561_INIT(1, TSL2561_ADDR_HIGH);
	*/
	
	// initialize at runtime instead
	// FIRST SENSOR --> TSL2561_ADDR_LOW
	lights[0].adapter_nr=1;						// change this according to your i2c bus
	lights[0].sensor_addr=TSL2561_ADDR_LOW;				// don't change this
	lights[0].integration_time=TSL2561_INTEGRATIONTIME_402MS;	// don't change this
	lights[0].gain=TSL2561_GAIN_16X;				// don't change this
	lights[0].adapter_fd=-1;					// don't change this
	lights[0].lasterr=0;						// don't change this
	bzero(&lights[0].buf, sizeof(lights[0].buf));			// don't change this
	
	// SECOND SENSOR --> TSL2561_ADDR_FLOAT
	lights[1].adapter_nr=1;						// change this according to your i2c bus
	lights[1].sensor_addr=TSL2561_ADDR_FLOAT;			// don't change this
	lights[1].integration_time=TSL2561_INTEGRATIONTIME_402MS;	// don't change this
	lights[1].gain=TSL2561_GAIN_16X;				// don't change this
	lights[1].adapter_fd=-1;					// don't change this
	lights[1].lasterr=0;						// don't change this
	bzero(&lights[1].buf, sizeof(lights[1].buf));			// don't change this
	
	// THIRD SENSOR --> TSL2561_ADDR_HIGH
	lights[2].adapter_nr=1;						// change this according to your i2c bus
	lights[2].sensor_addr=TSL2561_ADDR_HIGH;			// don't change this
	lights[2].integration_time=TSL2561_INTEGRATIONTIME_402MS;	// don't change this
	lights[2].gain=TSL2561_GAIN_16X;				// don't change this
	lights[2].adapter_fd=-1;					// don't change this
	lights[2].lasterr=0;						// don't change this
	bzero(&lights[2].buf, sizeof(lights[2].buf));			// don't change this
	
	// initialize the sensors
	for(i=0; i<3; i++) {
		rc = TSL2561_OPEN(&lights[i]);
		if(rc != 0) {
			fprintf(stderr, "Error initializing TSL2561 sensor %i (%s). Check your i2c bus (es. i2cdetect)\n", i+1, strerror(lights[i].lasterr));
			return 1;
		}
		// set the gain to 1X (it can be TSL2561_GAIN_1X or TSL2561_GAIN_16X)
		// use 16X gain to get more precision in dark ambients, or enable auto gain below
		rc = TSL2561_SETGAIN(&lights[i], TSL2561_GAIN_1X);
		
		// set the integration time 
		// (TSL2561_INTEGRATIONTIME_402MS or TSL2561_INTEGRATIONTIME_101MS or TSL2561_INTEGRATIONTIME_13MS)
		// TSL2561_INTEGRATIONTIME_402MS is slower but more precise, TSL2561_INTEGRATIONTIME_13MS is very fast but not so precise
		rc = TSL2561_SETINTEGRATIONTIME(&lights[i], TSL2561_INTEGRATIONTIME_101MS);
	}
	
	// you can now sense each sensor when you like
	for(i=0; i<3; i++) {
		// sense the luminosity from the sensors (lux is the luminosity taken in "lux" measure units)
		// the last parameter can be 1 to enable library auto gain, or 0 to disable it
		rc = TSL2561_SENSELIGHT(&lights[i], &broadband, &ir, &lux, 1);
		printf("Test sensor %i. RC: %i(%s), broadband: %i, ir: %i, lux: %i\n", i+1, rc, strerror(lights[i].lasterr), broadband, ir, lux);
	}
	
	// when you have finisched, you can close things
	for(i=0; i<3; i++) {
		TSL2561_CLOSE(&lights[i]);
	}
	
	return 0;
}

21 Feb 14 HOWTO generate a SAN (Subject Alternative Names) SSL CSR with OpenSSL

There is a cool SSLv3 protocol extension that’s called SAN (Subject Alternative Names). With this extension you can create a single SSL X509 certificate that is valid for several domain names, instead of a classic certificate that’s valid for one domain name only.

You can ofcourse create this kind of certificate with OpenSSL. We are now going to see how to do that.
Fist you have to create a file called openssl.cnf and put it for example into a temporary dir. The file should begin with:

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

This is to enable SSLv3 req extensions.
Now, you have to add your custom informations to the openssl.cnf file: those informations will be reflected on the next steps.
Add something like this to openssl.cnf:

[req_distinguished_name]
countryName = Country Name (2 letter code)
countryName_default = IT
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Italy
localityName = Locality Name (eg, city)
localityName_default = Rome
organizationName = Organization name
organizationName_default = My company name Srl
organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = System Techies
commonName = Common Name (eg, YOUR name)
commonName_max = 64
#commonName_default = www.myfirstdomain.it
emailAddress = Email Address
emailAddress_max = 40

The informations above are used by the “openssl req” command to ask you data to generate your certificate request.
Then, add this block of informations into the openssl.cnf file:

[v3_req]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

Those informations will enable some extra useful things on your certificate request that will hopefully became valid on your brand new SSLv3 certificate. For example you are requesting your Certification Authority to release a X509 SSLv3 certificate with server and client authentication purposes, plus other certificate goodies.

Now the cool part: this is where you are asking your CA to release a certificate with Alternative Names (certificate valid for several domains). Append this stuff in openssl.cnf:

[alt_names]
DNS.1   = www.myfirstdomain.it
DNS.2   = myfirstdomain.it
DNS.3   = www.myalternativedomain.it
# you could also specify IP addresses like this:
# IP.1 = 1.2.3.4

OK. You are almost ready to create your CSR, but first you have to generate your private key.
NOTE that many CA are now requesting a private key of 2048 bits or more. Warned: a key of 1024 bits is not recommended!
To generate a 2048 bits private key, as usual, execute this command:

openssl genrsa -out server.key 2048

Perfect. It’s time to create the Certificate Request (PKCS#10) with SSLv3 extensions:

openssl req -new -out server.csr -key server.key -config openssl.cnf

Now, send your new server.csr file to your Certification Authority that will hopefully accept the request and relase a valid X509 SSLv3 certificate with SAN.

Good luck and enjoy.

05 Dic 13 As you can see, this site is reachable via IPv6!

If you are using IPv6 (like me) you can see that this blog is reachable via IPv6. Pretty cool!

ipv6 ready