msgbartop
Blog di Dino Ciuffetti (Bernardino in realtà)
msgbarbottom

21 Feb 12 Mitigare attacchi di tipo Distributed Denial Of Service su apache con mod_evasive

Ciao.
C’e’ un modulo molto carino non official per apache che permette di bloccare o comunque mitigare facilmente attacchi di tipo DDOS.

Si scarica il modulo mod_evasive dal sito http://www.zdziarski.com/blog/?page_id=442, si decomprime il tar.gz e si compila con la seguente stringa:

<directory_di_installazione_apache>/bin/apxs -i -c mod_evasive20.c

Ora bisogna inserire due cosette in httpd.conf e successivamente riavviare gli apache:

1) LoadModule evasive20_module modules/mod_evasive20.so
2) La sua configurazione

Per quanto riguarda la numero 2 bisogna fare una considerazione importante, ovvero che il tuning del presente modulo con parametri errati o maldimensionati puo’ provocare dei falsi positivi che creano gravi disservizi per un periodo anche prolungato di tempo.

La configurazione e’ simile a questa (effettuo già un pre tuning che comunque dovrete ricontrollare e confermare voi che conoscete il vostro sistema):

<IfModule mod_evasive20.c>
DOSHashTableSize   196613
DOSLogDir “/tmp”

DOSPageCount         20
DOSPageInterval      10

DOSSiteCount        350
DOSSiteInterval      5

DOSBlockingPeriod   10
</IfModule>

Di seguito una breve spiegazione dei parametri:

DOSHashTableSize: e’ la dimensione della tabella di lookup interna usata dal modulo. Su server carichi il numero non deve essere troppo basso. Non andrebbe messo un numero casuale ma un numero primo presente nella struct dei numeri primi dentro mod_evasive20.c.

DOSLogDir: e’ una directory dove evasive appoggia dei dati che vengono utilizzati internamente per motivi di locking tra i processi child di apache

DOSPageCount: e’ la soglia per il numero di richieste effettuate verso una stessa url da uno stesso IP entro un determinato intervallo di tempo specificato dalla direttiva DOSPageInterval. Se il numero di richieste verso una stessa pagina/oggetto/uri viene superato nell’intervallo di tempo specificato apache blocchera’ l’IP del client per un determinato numero di secondi specificati dalla variabile DOSBlockingPeriod. Consiglio vivamente di non impostare questo parametro troppo basso per evitare falsi positivi all’interno di ambienti nattati o dietro proxy server (ad esempio clienti fastweb, utenti all’interno di grosse organizzazioni che usano un proxy server, ecc.)

DOSPageInterval: e’ l’intervallo di tempo espresso in secondi che attiva la soglia relativa al parametro precedente (DOSPageCount)

DOSSiteCount: e’ la soglia per il numero di richieste complessive verso tutto il sito effettuate da uno stesso IP entro un determinato intervallo di tempo specificato dalla direttiva DOSSiteInterval. Se il numero complessivo di richieste verso apache viene superato nell’intervallo di tempo specificato, apache blocchera’ l’IP del client per un determinato numero di secondi specificati dalla variabile DOSBlockingPeriod. Anche qui consiglio vivamente di non impostare questo parametro troppo basso per evitare falsi positivi all’interno di ambienti nattati o dietro proxy server (ad esempio clienti fastweb, utenti all’interno di grosse organizzazioni che usano un proxy server, ecc.)

DOSSiteInterval: e’ l’intervallo di tempo espresso in secondi che attiva la soglia relativa al parametro precedente (DOSSiteCount)

DOSBlockingPeriod: e’ l’intervallo di tempo espresso in secondi durante i quali tutti i client finiti in blacklist avendo superato i limiti imposti dai parametri specificati sopra non potranno collegarsi ad apache e navigare sul sito ottenendo invece un codice di errore HTTP/403 (Forbidden). Se i clients bloccati continuano ad effettuare nuove richieste durante il periodo di blocco, ogni richiesta resetta il timer percui il conteggio ricomincia. Consiglio quindi VIVAMENTE di non impostare questo parametro troppo alto, in genere non vanno mai superati i 20 secondi. Il motivo principale e’ la mitigazione di eventuali disservizi causati da falsi positivi. Il discorso e’ inoltre valido perche’ durante un attacco DDOS il timer viene continuamente resettato e l’attacco viene comunque mitigato anche impostando il parametro ad un valore molto basso.

Riavviate apache e buon divertimento.

Dino Ciuffetti.

15 Nov 11 How to compile apache httpd on HP-UX 11.11 PA-RISC

The first thing that I have to say, after more than 10 years working with different OSes, is that there is no better operative system than Linux. Any other OS that I’ve worked with is a pure shit, in my humble opinion off course. HP-UX is one of this. This is a closed box with custom patches here and there, not a true, modern os like linux or free bsd, and the like. The compiler is closed source and it’s not free.

The best way that I’ve found to compile apache with gcc on HP-UX 11.11 (pa-risc) using open source free software is:

  1. download the following software packages from HP-UX Porting Centre (http://hpux.connect.org.uk/) – your version may vary: zlib-1.2.5-hppa-11.11.depot.gz, make-3.82-hppa-11.11.depot.gz, libiconv-1.14-hppa-11.11.depot.gz, gettext-0.18.1.1-hppa-11.11.depot.gz, openssl-1.0.0e-hppa-11.11.depot.gz, libgcc-4.2.3-hppa-11.11.depot.gz, gcc-4.2.3-hppa-11.11.depot.gz
  2. gunzip each one of the downloaded depot, (eg: gunzip * from the directory where you downloaded)
  3. install each depot in the order given below (the first is zlib, the last is gcc) with the standard hpux command: swinstall -s [your_absolute_depot_path]
  4. once this boring operation mandatory only on non modern operative systems is terinated successfully, you can export the PATH variable setting /usr/local/bin in front of the PATH list: export PATH=”/usr/local/bin:$PATH”
  5. ok. We are now ready to compile apache. Download and uncompress the httpd tar.gz with “gunzip”, then “tar xf” (on a modern system you can do it in a single pass with tar xzvf …)
  6. the configure string to run is: ./configure –with-included-apr –with-expat=builtin –prefix=[YOUR_APACHE_INSTALLATION_PATH] –enable-mods-shared=most –enable-ssl –enable-proxy –enable-proxy-connect –enable-proxy-http –enable-proxy-balancer –enable-cache –enable-disk-cache –enable-mem-cache
  7. once finisced, run: “gmake“.

At this point, after some minute, you probably will end with a compiler error like this one:

/var/adm/crash/src/httpd-2.2.21/srclib/apr/libtool –silent –mode=link gcc -g -O2 -pthread     -L/usr/local/lib   -o htpasswd  htpasswd.lo   -lm /var/adm/crash/src/httpd-2.2.21/srclib/pcre/libpcre.la /var/adm/crash/src/httpd-2.2.21/srclib/apr-util/libaprutil-1.la /var/adm/crash/src/httpd-2.2.21/srclib/apr-util/xml/expat/libexpat.la -liconv /var/adm/crash/src/httpd-2.2.21/srclib/apr/libapr-1.la -lrt -lm -lpthread -ldld
libtool: link: warning: this platform does not like uninstalled shared libraries
libtool: link: `htpasswd’ will be relinked during installation
/usr/ccs/bin/ld: Unsatisfied symbols:
apr_generate_random_bytes (first referenced in .libs/htpasswd.o) (code)
collect2: ld returned 1 exit status
gmake[2]: *** [htpasswd] Error 1
gmake[2]: Leaving directory `/var/adm/crash/src/httpd-2.2.21/support’
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/var/adm/crash/src/httpd-2.2.21/support’
gmake: *** [all-recursive] Error 1

This means that the APR library cannot generate random numbers. I have to investigate why, probably the system is not capable/patched to generate PRN numbers at kernel level (/dev/random or /dev/urandom) and the APR library breaks. Not a problem. Simply skip the creation of the htpasswd executable. You will probably not need it.

  • cd support
  • touch htpasswd
  • cd ..

Now came back to compile:

  • gmake

when finished, simple “gmake install“, and you hopefully have done, thinking why you are still using a non modern os and becoming soon a happy new linux user..

😉 Hope this one will help some linux user fighting on HP as well like me!

Ciao, Dino.

07 Ott 11 Apache with Worker MPM (multi threaded), mem_cache and mod_deflate

When you have to publish mainly static contents, like static sites, the most powerful solution is to configure your apache http server to use the MPM Worker, mod_mem_cache e mod_deflate modules.

Why the MPM Worker
It implements a multi process / multi thread server model. The father process spawn processes, and each child process spawn threads. Each thread will handle a client connection.
This implementation can handle a large number of requests with fewer system resources than a standard prefork multi process server model.
Please note that you cannot use the MPM Worker in server environments that are not thread safe. For example, PHP, mod_perl, and other dynamic page processors do not ensure you that the environment it’s completely thread safe, so my advice is to NOT USE the MPM Worker with PHP, mod_perl and the like.
The Worker MPM can consume much less memory because the heap memory is shared among threads, while that’s not true for processes.
For more informations you can read the official page: http://httpd.apache.org/docs/2.2/mod/worker.html

Why the mod_mem_cache module
This module can be configured to cache open file descriptors and objects into the heap storage (memory).
If the same object (html, css, js, etc) it’s requested for the first time by a client, it get saved into the heap memory. The second time it got requested, the object got feeded directly from the memory cache. It can lower down CPU and disk I/O.
For more informations you can read the official page: http://httpd.apache.org/docs/2.2/mod/mod_mem_cache.html

Why the mod_deflate module
It can allows output from your server to be compressed before being sent to the client . The HTTP 1/1 protocol has a header called Accept-Encoding. This way a client can tell the server witch response encoding it can reads.
Any modern browsers today can handle page compression, so why not using it?
With it you can save bandwidth.
For more informations you can read the official page: http://httpd.apache.org/docs/2.2/mod/mod_deflate.html

Ok. Let’s begin to enable that stuff.

First step is to compile apache from source.
If you want to use the packages released by your linux distribution instead of compiling apache by yourself you can do it.
Always choose the latest apache stable version available.

To compile apache 2.2.X with most modules in shared form (*.so) you should run this configure:
$ ./configure –prefix=<YOUR_APACHE_DIR> –with-mpm=worker –with-included-apr –with-expat=builtin –enable-mods-shared=most –enable-ssl –enable-proxy –enable-proxy-connect –enable-proxy-http –enable-proxy-balancer –enable-cache –enable-disk-cache –enable-mem-cache –enable-nonportable-atomics=yes

Then, as usual, run:
$ make
$ make install

You hopefully end up with apache correctly installed with all needed modules in place.
Now configure your httpd.conf adding those lines:

# Compress on the fly HTML pages, TXT and XML files, CSS and JS.
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/x-js application/x-javascript

# Cache open file descriptors
CacheEnable fd /

# Enable memory caching
CacheEnable mem /

# Limit the size of the cache to 24 Megabyte
MCacheSize 25165824

# Minimum size of an object that can be cached: 1 Kbyte
MCacheMinObjectSize 1024

# Maximum size of an object that can be cached: 3 Mbyte
MCacheMaxObjectSize 3145728

# Spawn 10 child processes, spawning 100 threads for each child process.
# So, a pool of 1000 threads is left up and sleeping, ready to serve incoming requests.
# If more requests will come in, apache will spawn new child processes, each one spawning 100 threads,
# enlarging the thread pool until the total number of threads become 2000. In that case, apache begin
# to cleanly drop processes, trying to reach 1000 threads.
# New processes and its threads are spawned in case of a large spike of requests, until 4000 parallel
# client requests are reached, then apache will no longer accept new incoming connections.
# When the load calm down, and requests come back under 4000 parallel connections, apache will continue
# to accept connections. After 1,000,000 requests served by a child, q. 10,000 per thread, the process
# get closed by the father to ensure no memory leak is fired.
<IfModule mpm_worker_module>
ThreadLimit          100
ServerLimit         4000
StartServers          10
MaxClients          4000
MinSpareThreads      1000
MaxSpareThreads      2000
ThreadsPerChild      100
MaxRequestsPerChild   1000000
</IfModule>

Start apache.
Enjoy!!

 

15 Set 11 How to compile apache httpd 64 bits on Solaris 10 sparc

As usual we try to consider the “do it right (r)” way of doing configurations.
Today we will compile Apache HTTPD on Sun Solaris 10 OS (SPARC).

The first thing to do is to install the gcc c compiler if it is not already installed.
To do so, download and install the package from www.sunfreeware.com. Double read the package release notes.
You have to download the latest gcc package and its dependencies. You probably will need also libiconv and libintl.
Now download openssl-0-9.X package.

For each downloaded package install it with the command: dpkg -d <full_path>/your_package

When finished, go into your apache source directory and:

export LD_LIBRARY_PATH=/usr/sfw/lib/sparcv9:/usr/local/lib/sparcv9
export PATH=/usr/sfw/bin:/usr/ccs/bin:/usr/local/ccs/bin:/usr/local/bin:$PATH

# if you want it 64 bits:
export CFLAGS=”-m64″
# if you want it 32 bits:
# export CFLAGS=”-m32″
export LDFLAGS=”-L/usr/sfw/lib/sparcv9″

./configure –with-included-apr –with-expat=builtin –prefix=<your_installation_path> –enable-mods-shared=most –enable-ssl –with-ssl=/usr/sfw –enable-proxy –enable-proxy-connect –enable-proxy-http –enable-proxy-balancer

If the configure process terminated successfully, you can now call:

make

When finished, as usual, call:

make install

I recommend you to use gnu make. You can download it from sunfreeware.
Now, if everything gone ok, you can try to start your brand new 64 bits apache full of powerfull modules.
You may want to set your LD_LIBRARY_PATH variable into <apache>/bin/envvars file so that apachectl can find all the library it needs to start or stop the server.

Ciao, Dino.

12 Set 11 How to install the “Apache Tomcat Native” libtcnative module

The Apache Tomcat Native module, also called “TC-Native library” or “libtcnative”, is a library that implements HTTP, HTTPS and AJP connectors in tomcat using the Apache APR library. This ensure great scalability and performance because permits tomcat to access server native technologies like openssl, system calls like sendfile() or epoll(), advanced I/O, OS level functionality and native Inter Process Communication.

To install libtcnative you must first have a working C compiler environment, a valid “apr” and “openssl” installation with the development libraries, a working apache tomcat 6.0.X and a Java JDK.

On debian it’s as simple as to run:

apt-get install build-essential libapr1-dev libssl-dev

The libtcnative source software can be found in the Tomcat binary bundle, in the bin/tomcat-native.tar.gz archive, but if you want the latest version you can find it here: http://tomcat.apache.org/native-doc/

Untar the tomcat-native archive, then:

cd tomcat-native-1.*/jni/native
./configure –with-apr=`which apr-1-config` –with-java-home=$JAVA_HOME –with-ssl=yes –prefix=$CATALINA_HOME

If you want or need to, you can pass the correct path of APR and OpenSSL libraries to the –with-apr and –with-ssl parameters.
CATALINA_HOME and JAVA_HOME are the path of the Java JDK and Tomcat installations.

After the configure script succeeded, you have to:

make
make install

Now, the libtcnative library should be correctly installed into “$CATALINA_HOME/lib”.
If you want you can now configure tomcat with the new connectors parameters.

The official project page of libtcnative is here: http://tomcat.apache.org/native-doc/
The documentation page of the tomcat 6 APR native functionality is here: http://tomcat.apache.org/tomcat-6.0-doc/apr.html

Hope this help someone to speed installation.
Ciao a tutti, Dino Ciuffetti.

31 Ago 11 CVE-2011-3192 – Apache killer DOS vulnerability

Hi people. On August 2011 has been discovered that apache httpd server is vulnerable to a simple to perform DOS attack. A simple perl exploit has been released called apache killer that make a big number of parrallel crafted HTTP calls (HEAD method) with the “Range” header. This make possibile to the attacker to consume memory and cpu on the attacked server bringing apache and the system down in no time. The attacker does not need large bandwidth to perform the attack.

Anyone using apache httpd in production environment is encouraged to upgrade to the latest apache version that solve the security problem.

If you cannot upgrade to >= 2.2.20 you can use mod_rewrite to deny requests with the Range header like the rewrite regexp below. This is what you need on your httpd.conf:

RewriteCond %{HTTP:range} ^bytes=[^,]+(,[^,]+){0,4}$
RewriteRule .* – [F]

You can find the exploit script on google. I will not put it here.
Ciao, Dino.

12 Ago 11 Apache: [error] (28)No space left on device: Cannot create SSLMutex

When you work as a horizontal support tech consultant for a very big company you may have to do with people that has basic linux/opensource knowledge and commercial system stuff (win, unix) skills. You may even have to do (… !?!?) with very prepared people (…).

Someone here takes care of apache installations and apache itself is very robust and stable so we all can sleep quietly, the problem comes out when someone with less OS system skills fires “kill -9” on apache processes to stop it.

[Thu Aug 11 17:47:01 2011] [error] (28)No space left on device: Cannot create SSLMutex
Configuration Failed

And apache does no longer come up and running.

The trouble may be weird because one could read “No space left on device” without reading the real error message: “Cannot create SSLMutex”.
But the problem is really easy to understand if you try to figure out what is going on at syscall OS level when apache starts up. This can be done with “strace” command on linux (“truss” or “tusc” on other expensive unix environments…).

The problem is caused by IPC SysV semaphores still standing up on the system from the previous apache kill.

The definitive solution here is to stop apache with “apachectl” command or calling “kill -15” (and not kill -9 !!!!) on the apache father process. This way you are instructing apache to stop gracefully, the father kills his childs and cleanup semaphores and the like, the clean way.

To solve your problem you have to cleanup hanging semaphores. You could reboot linux, but this is avoided on any serious environment, so which is the magic command?

If apache runs with “apache” user you can call this command to clean up semaphores created by the “apache” user:

ipcs -s | grep apache | perl -e ‘while (<STDIN>) {@a=split(/\s+/); print `ipcrm sem $a[1]`}’

You could do “ipcs -s | grep apache” to see the semaphores first, and then call ipcrm on each to clean it up.
Try to start apache now and the problem would solve.

The other way is to change the apache serialization mechanism from semaphores to pthread mutexes or fcntl. To do so you have to:

1) set “AcceptMutex fcntl” on httpd.conf
2) set “SSLMutex pthread” on httpd-ssl.conf

Hope this help someone… 🙂

Ciao, Dino.

15 Mag 11 How to cleanly install apache httpd webserver from source

Apache is a great piece of software and it’s very powerful. You can here find how to install apache web server, “The right and quick way” ™ 🙂 Apache will be compiled with modules support.

We begin downloading source code from httpd.apache.org official website.

Important: for security reasons download apache only from the official website or by your distribution vendor (redhat, debian, etc)!
Important: choose the right version for you, but if you can pick the last stable version of 2.2.X.

You can then untar your fresh apache source distribution and beginning to play with configure:

tar jxvf httpd-2.2.??.tar.bz2
cd httpd-2.2.??
./configure –with-included-apr –with-expat=builtin –prefix=/your_apache_installation_path –enable-mods-shared=most –enable-ssl –enable-proxy –enable-proxy-connect –enable-proxy-http –enable-proxy-balancer

Here we are going to compile apache with most of its modules compiled as shared objects and obviously with the shared object support enabled.
May be that the configure process will not find something. If so, remember to download and install the development version of the needed library. I always use the development package of the distribution (ex. libssl-dev, openssl-devel, etc).
If the configure process fails because of a dependency (development!!!) library not found but you already installed it, it means that the apache building tool does not find it.
Sometimes, in not standard environments (hpux, etc) apache will not find the openssl dev library but it’s installed. For example if apache fails a dependency check on openssl, you can fix the problem passing the argument –with-ssl=/ssl_path to configure.

If the configure pass, you have to call:

make

If the make process pass without problems, you have to execute:

make install

If you have problems compiling apache, most probably the problem is that you failed to install building prerequisites: development libraries, or the building tool cannot find where they are installed. If the configure fails you can take a read on config.log or reading the configure file (it’s a generated shell script) searching for the reported error and finding the cause by yourself (wrong path of the library?). If the failing stuff fires on compiling (make) you have to find the problem going to the source directory reported by the make utility and searching for the source file that it’s failing. You may require a patch, may be you caught a apache bug or may be your compiling chain is not clean.

I have to go now, let me know if you have problems.
Ciao, Dino.

08 Mar 11 Modify JSESSIONID cookie path with apache and mod_headers

The question is: how can I change the path of the JSESSIONID cookie for a web application deployed in tomcat, jboss, or any other AS, and served by an apache reverse proxy (ProxyPass on mod_proxy, or jkMount on mod_jk) to add a trailing slash?

The answer is in mod_headers module. This module supports perl regular expression that you can use to substitute a string with another on any HTTP header, on the request or on the response.

We may want to add a trailing slash (mypath/) to the JSESSIONID cookie path, for example for security reasons.
This is the correct way (apache >= 2.2.4):

Header edit Set-Cookie "^(JSESSIONID=.*; Path=/YOUR_APP_PATH)(.*)$" "$1/$2"

eg:

Header edit Set-Cookie "^(JSESSIONID=.*; Path=/jsp-examples)(.*)$" "$1/$2"

The first attribute defines the regular expression that matches against the string that must be edited (the SESSIONID header in this case), the second attribute is the expression of the new string (the sessionid with the path modified with a trailing slash). Note that the expression begins with ‘^‘ character (it means: the string must begin with).
This kind of regexp defines that each match pattern is enclosed into brackets, so the first match is anything that begins with “JSESSIONID=”, have some kind of sub string (.*), and then contains “; Path=/jsp-examples”.
The second match is anything on the right of the path (.*).
The second argument implies that the string is composed by the first match, a slash, then the second match. So we have a cookie called JSESSIONID with a trailing slash added in the path.

If you don’t understand perl regular expression well, I advice you to get deeper into it, because it’s very very very useful for any sysadmin. There is very good documentation in internet, try to google “perl regular expression examples”.

Apache httpd is a very good and powerful piece of code, and it’s generally possible to do anything you can thinking of. You have only to know where to search, and the manual is generally the right place.

Ciao, Dino.

24 Ago 10 Why do not host sites from your own home

I would not recommend you to host sites that way, you have to be sure that your ISP give you public IP(s) and setup your router to port forward ports 80, 443, 53, and so on.
There are other problems too:
1) if you want to host more than one site with SSL you must have one public IP for each SSL site or use different SSL ports for each site, because name virtualhosting with SSL is not possible;
2) dsl lines are not designed to be stable. The connection can go down and make your site not visible. This is a major problem if you make the mistake to have your own DNS server on it!! The ISP assigned public IP address can change more than one time a day and you have to sync the DNS zone each time.
3) dsl ips are putted into DNS based blacklists zones. You may not be reached from various HTTP proxy servers around the world. For the same reason you cannot send mails, for example originated from your sites.
4) adsl lines are asymmetric (unbalanced for download). You have few kbytes per second in upload, that is just what you need to publish web sites, so this can be a problem when you have just more than 3 users.
5) you probably have problems with High-Availability and Load-Balancing on domestic hardware and you may have blackouts.
6) DNS subsystem may need primary and secondary DNS servers.

The best way (imho) is to use services like slicehost where you have a HA virtual server slice running linux, public IP addresses, free primary and secondary DNS hosting service, large public bandwidth, disk space… and not last your own root password that you can use to have maintenance on your own server for your own.

https://manage.slicehost.com/customers/new?referrer=af57db3020e04bb27352e271753a7a18