For some background you may want to read the Original Story leading to this write up.

The first thing that caught my attention was the fact Logwatch was reported login failures in the order of 1000’s from unassigned.psychz.net without an accompanying fail2ban email notifying me the offender had been banned.

And this as it would turn out was because the attack was clearly intended to defeat such protection methods, this is due to the logged host being unassigned.psychz.net, when the authentication failure is logged, a reverse lookup is made within vsftpd to resolve the host this PTR record returns unassigned.psychz.net, and as such is written into the log.

fail2ban no uses regex to extract the host from the logs, and attempts to make a forward lookup on unassigned.psychz.net (A/CNAME records required) to resolve the ip address, and ban the offending ip, this is where things go awry.

psychz.net maintains their own DNS servers,

1. DNS1.PSYCHZ.NET
2. DNS2.PSYCHZ.NET

These provide a PTR but no A/CNAME record, as such fail2ban can not resolve an IP and the attacking ip is left to run their attack unhindered, see this log file: fail2ban name resolution failure log

The only way therefor to gain the attacking ip was to match the ftp connection times to those of the reported login failures using iptables to log all accesses to ftp, quickly get a count of connecting ip’s using:

grep kernel /var/log/messages | awk '{print $9}' | sed 's/SRC=//' | uniq -c | sort

390 173.224.217.41

A complete log can be found here: iptables.log, and a whois can be found here: whois.txt

Disclosure steps taken:

1. 26/07/10 psychz support informed given deadline of 09/08/10 for resolution
2. Same day standard reply of “thanks for contacting support we are looking into this” …
3. 27/07/0 Attacks continue 173.224.208.0/20 network black holed as a result

   iptables -A INPUT -s 173.224.208.0/20 -j DROP
   

4. 09/08/10 deadline passes without update
5. 25/08/10 this blog post published

This blog entry here: https://rackerhacker.com/2010/08/25/a-nerds-perspective-on-cloud-hosting/ prompted me to write this blog post, after I realized I’d filled the comment field, without ending my “monologue”, anyway I thought it would be better to voice my opinions here, to you lot who are daft enough to read this blog.

I think the problem mainly is the term “cloud” has been massively over marketed and possibly long since lost it’s original meaning, with providers trying to jump on the marketing bandwagon.

I’ve not made the jump to “the Cloud” yet, as frankly I can’t see a benefit to them over properly configured HA installations, for example I would much rather be using several pre-configured servers using RHCS to handle the migration of critical services (mySQL etc..).

I begin to see the benefits for large hosting providers, where customers what the power of a dedicated server but only pay for what they actually use, in this instance a provider ensures up time through live migration,

Some other misconceptions through over marketing I’d like to point out,

1) The “cloud” is not always on

Don’t get me wrong it can be configured to be close, using distributed VM’s for your critical services (i.e. apache), coupling this with loadbalancing and clustering setups.

The misconception for most “end users” is that if you buy a single cloud instance, through magic/voodoo it will always be on 100% of the time!

Simply put if the hardware it was running on dies, it will go down, regardless of live migration measures in place, there will be downtime, do not pass go do not collect http 200 go directly to > /dev/null

2) The “cloud” is not secure

If you insist on putting your 5 year old joomla website on a cloud VM, it can and will become compromised quickly, security is only going to be as good as the configuration you have in place, you have mitigation measures such as

• selinux
• webapp updates/patches
• fail2ban/banhosts packages

Whilst in itself a VM is largely seen as secure as it protects the host machine should the VM become compromised, it is not always the case, for instance there have been several occurrences of VMWare ESXI servers allowing code execution on the host (long since patched Don’t panic!), allowing attackers who have compromised a VM on the cloud to root the host machine and as a cascading effect every other VM instace on the box.

Let me point out a worst case scenario here:

1. Hypervisor running on Host A with 30 Vm’s
2. Host A is part of a resilient set with live migration in place, Hosts B,C,D
3. VM A’s 5 year old joomla app is subject to an XSS bug, and an attacker places the r57 shell on the webapp,
4. attacker proceeds to deploy backdoors (i.e. meterpreter)
5. VM A is subject to remote code execution on host
6. Attacker compromises Host
7. Host A is now root’ed
8. Attacker forces Migration of VM A onto Host B
9. Host B rooted using same method
10. Rinse & repeat for C & D

In summary, if you are looking at a cloud solution and your web presence is important take an informed decision from one of the larger providers, and NEVER EVER go with the cheapest option you could find, probably on ebay …

The cloud is not some magical being created by the hosting fairies that will take all your hosting and maintenance woes away, it may or may not be the right thing for your business / web app, and in certain instances can lower TCO, I for one will be sticking with my Cluster services and high Availability designs for a while yet.

Time was when a photo was just a captured moment in time, /end nostalgia

Nowadays though what people do not realize is the shear amount of “extra” information is embedded in “that picture you just uploaded to flikr/facebook/photo bucket” especially if you are uploading from a “smart phone” as more and more people are now.

Most photos now contain GPS data embedded in them, this information will survive a resize / upload process, at the time of writing images tested from Facebook appear to have the exif data stripped out (thumbs up for facebook maybe), and it appears php GD by default replaces all EXIF data with it’s own (bug maybe?).

For non sanitized images however you can discern a wealth of information such as:

1. Make of camera
2. Model of camera
3. Software version
4. Unix timestamp of time taken
5. DateTime stamp of time taken
6. Focal length used
7. Shutter speed
8. if flash used

And if GPS is embedded:

1. Longitude
2. Latitude
3. Altitude
4. GPS timestamp
5. Direction facing when photo taken

There is yet more data such as the colour profile used, and image resolutions, in my tests photos taken from my iPhone 4 were within 10 meters of where I was actually standing when I took the picture, and in which direction I was facing when I took them.

So one more thing to note in your applications “data sanity” is to strip EXIF tags from uploaded images, lest your contributors private details be leaked from your application.

For example:

1. User uploads photo for competition
2. Site uses resized photo on competition page to allow visitor voting
3. malicious user, saves image from site (or just uses the copy from thier browser cache), gets gps data from photo
4. malicious user now knows exact whereabouts photo was taken aswell as the time.

And it doesn’t have to be a malicious user, it could be anyone/anything, if you want to check your images for EXIF data you can use my tool here: https://blog.oneiroi.co.uk/tools/exif_data.php

No data is stored, and images are deleted immediately after processing, you use this at your own risk however, if you misuse the tool you accept all liability for the legal action to follow, you have been warned.

ESP Ghostscript 815.02: Unrecoverable error, exit code 255

I got this issue today whilst running CentOS 5.4 x64 post investigation of images not being scaled when processing a specific PDF, the solution unfortunately is to build ghostscript and imagemagick from the latest sources.

wget https://ghostscript.com/releases/ghostscript-8.71.tar.gz
wget https://image_magick.veidrodis.com/image_magick/ImageMagick-6.6.3-0.tar.gz

Unpack, configure, make && make install

To fix compatibility with pear imagick

ln -s /usr/local/lib/libMagickCore.so /usr/lib64/libMagick.so.10
ln -s /usr/local/lib/libMagickWand.so /usr/lib64/libWand.so.10
ln -s /usr/local/bin/gs /usr/bin/gs

Most of the time when I review our log watches each morning I become enraged at the number of automated attacks,

But ever so occasional I find one that frankly intrigues me.

Today is just such an occasion where I have had multiple Brute force login attempts, the ingenious part is this attack has been designed to bypass tools such as fail2ban, blockhosts etc, and this is how

1. Attack is launched from XXX.XXX.XXX.XXX
2. has PTR set for sub.domain.tld
3. Failed login attempts record sub.domain.tld due to reverse lookup
4. There is no A record, attacker maintains their own nameservers for the domain.tld
5. fail2ban notes failed logins, attempts to resolve sub.domain.tld to an IP but fails, due to missing A record
6. Attacker can continue brute force attempts unhindered by being banned

I am still reading into how to counter this and will update this post as I figure out how to work around it, it’s a very sneaky and frankly quiet clever method of working around most automated blacklisting/banning tools.

Update 1: One method I am trialing is the “log target” feature of iptables, in an attempt to match login failure times to the iptables log, I’ll post back with results.

iptables -A INPUT -p tcp --dport ftp -j LOG

Outputs

Jul 23 11:45:57 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=64 TOS=0x00 PREC=0x00 TTL=55 ID=47423 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 23 11:45:57 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=52 TOS=0x00 PREC=0x00 TTL=55 ID=45370 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK URGP=0 
Jul 23 11:45:57 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=52 TOS=0x00 PREC=0x00 TTL=55 ID=46896 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK URGP=0 
Jul 23 11:46:01 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=63 TOS=0x00 PREC=0x00 TTL=55 ID=38502 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK PSH URGP=0 
Jul 23 11:46:02 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=52 TOS=0x00 PREC=0x00 TTL=55 ID=32551 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK URGP=0 
Jul 23 11:46:02 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=52 TOS=0x00 PREC=0x00 TTL=55 ID=59735 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK URGP=0 
Jul 23 11:46:04 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=66 TOS=0x00 PREC=0x00 TTL=55 ID=23116 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK PSH URGP=0 
Jul 23 11:46:07 132 kernel: IN=eth0 OUT= MAC=<mac addr> SRC=<connecitng ip> DST=<server ip> LEN=52 TOS=0x00 PREC=0x00 TTL=55 ID=40246 DF PROTO=TCP SPT=3865 DPT=21 WINDOW=65535 RES=0x00 ACK URGP=0 

Update 2: Defeating the hack

Now granted this would be a lot worse had the attacking IP been dynamic, fortunatly in this case it’s not

grep kernel /var/log/messages | awk '{print $9}' | sed 's/SRC=//' | uniq -c | sort

390   173.XXX.XXX.XXX
      4 195.XXX.XXX.XXX

Ip’s have been masked to prevent anyone complaining or threatening legal action (again) for inferring you should block their ip / network range … and me firing off the obligatory “Well if you policed your own network I wouldn’t have to post this no would I” email,

Maybe I am just being Cynical in my “old” age …

Any how as you may have guess I’m black holing the ip with the 390 connection entries.

Thanks

Being as I spoke to a load of people during the course of this I realy can not remember who contributed what to this solution, so I’ll just have to thank you all let me know if you want a crediting link.