I ment to note this down yesterday but everything is going ten to the dozen at the moment.

basically I have now authored a nagios addon for monitoring master-master replication between two servers, this carries out 4 stages of checks

1. Validates all required data is passed by servers
2. Slave IO is running on both servers
3. Seconds_Behind_Master check, args can be passed to vary warn and critical thresholds
4. (slave) Master_Log_File == (master) File

The 5th check was a comparison on the binlog positions themselves, comparing (slave) Read_Master_Log_Pos and (master) Position

Here in lies the problem, which took a while to track down, the problem is that no matter what I tried the slave was ALWAYS behind the master position … but why?

The reason is why I designed the High Availability solution in the first place … Very high traffic level, in the region of 20,800 transactions per second.

Why was this the problem? the two queries run to gather the data are done sequentially per server, using the python time library I was able to find that there is a 0.02s interval between gathering datasets (20 milliseconds) … in that time 416 transactions had take place.

i.e.

time: binlog pos

Slave A

0.000: 100

Master B

0.020: 516

This unfortunately has now lead to some 32 lines of code being commented out, as I can see no way to reliably use the binlog positions for monitoring the replication in this situation, if any delay occurs anywhere at any point during the dataset collection i.e. network latency, delay in query processing due to traffic peak on one server … etc. the collected samples will always be different

The only way I ever see this working is if you can validate that the datasets came from the same exact point in time down to the nanosecond, this however is again not possible, on the network the servers currently reside there is a 0.13 millisecond ping response time this works out to 13,000 nanoseconds (0.00013 * 10⁹⁾

If anyone has any theories on how to overcome this please let me know.

NOTE: At present due to the programming of this addon being done during working hours the nagios addons are not for public release at this time, this may be subject to change in the future should my employers allow their release.

I meant to write this up over a week ago now, basically the need arose for one of my Python scripts to use HTTP Basic authentication when grabbing the output from a URL.

An example script can be seen below:

Quick description, the script will attempt to connect to a URL and read the data supplied by the webserver, if a HTTP 401 error is returned (Authentication required) the script will then go on to attempt to authenticate using the credentials supplied.

Printing out to the console at each point.

Subversion: https://svn.blog.oneiroi.co.uk/branches/python/urllib2_httpbasic_auth.py

Highlighted source (at the time of writing)

#!/usr/bin/env python
"""
    Author: David Busby (https://blog.oneiroi.co.uk)
    Program: Python HTTP Basic Auth Exa
    Copyright: David Busby 2009. All rights reserved.
    License: https://creativecommons.org/licenses/by-sa/2.0/uk/
"""

import urllib2, base64

""" URL List """
urls = {
               0:{"url":"blog.oneiroi.co.uk/some/fictional/auth/area","user":"someuser","pass":"somepass"}
}

def main():
   ulen = len(urls)
   for i in range(0,ulen):
       url = "https://%s" % (urls[i]["url"])
       req = urllib2.Request(url)
       try:
           res = urllib2.urlopen(req)
           headers = res.info().headers
           data = res.read()
       except IOError, e:
            if hasattr(e, 'reason'):
                err = "%s ERROR(%s)" % (urls[i]["url"],e.reason)
                print err
            elif hasattr(e, 'code'):
                if e.code != 401:
                    err = "%s ERROR(%s)" % (urls[i]["url"],e.code)
                    print err
                #401 = auth required error
                elif e.code == 401:
                    base64string = base64.encodestring('%s:%s' % (urls[i]["user"], urls[i]["pass"]))[:-1]
                    authheader =  "Basic %s" % base64string
                    req.add_header("Authorization", authheader)
                    try:
                        res = urllib2.urlopen(req)
                        headers = res.info().headers
                        data = res.read()
                    except IOError, e:
                        if hasattr(e, 'reason'):
                            err = "%s:%s@%s ERROR(%s)" % (urls[i]["user"],urls[i]["pass"],urls[i]["url"],e.reason)
                            print err
                        elif hasattr(e, 'code'):
                            err = "%s:%s@%s ERROR(%s)" % (urls[i]["user"],urls[i]["pass"],urls[i]["url"],e.code)
                            print err
                    else:
                        err = "%s query complete" % (urls[i]["url"])
                        print err
       else:
            err = "%s query complete" % (urls[i]["url"])
            print err
                        
if __name__ == "__main__":
    main()

NOTES: This script does not check the authentication type, it always assumes it is HTTP BASIC, HTTP DIGEST for example is not compatible with this script, though there is no reason why you can not get the Auth type form the headers returned by the server and write in a Digest auth method.

As most have no doubt noticed the plugin is delayed … again

Too many projects, too few hours in the day.

Froomi - All in one HTTP Server, Search Engine, Pseudo Streaming Media server (Might add transcoding)

This project is up on google code: https://code.google.com/p/froomi/

So far the HTTP module is basically complete and already has GZIP compression support for browsers that send the appropriate header to indicate they support it.

I will being working on the Pseudo streaming component shortly, to recap this is in an Alpha state at the moment, I will be putting the server live on https://www.froomi.com shortly once the Pseudo streaming component is complete.

Flowplayer for wordpress, is still in development the code base has been completely re-written as per the previews you can now see the meta boxes in action, along with the license detection, I think I am going to cut a couple of features and just get the next release out the door (yes RTMP will still be in this release!).

Contributions, some people have shown an interest in contributing to these projects, contributions are very welcome and will carry the appropriate credits in the source code and license, I would go so far as to encourage contributions from developers as this will actually speed up releases for these projects.

So feel free if to contact me.

Contributor Requirements:

• Familiarity with the programming language used in the project you are contributing to (PHP,Python,etc..)
• Working knowledge of creating diff patches (not required bug a huge bonus)
• Ability to document your changes in English, sorry this is a requirements
• Skype / MSN / Ventrilo client

o

Because a picture is worth over 9000 internets … apparently

UPDATE: AKA “hayabusayuri” link … seriously who plays everquest? … maybe all that time playing everquest finally made the guy snap … and PHP & windows … never a good combination … infact Windows and internet is a bad combination

LINKY

Screencap incase it is removed:

(Thanks to the guys who forwarded me this)

Tonight I will be pushing to the development SVN a beta preview of the 2.1 release.

Here are some of the changes:

Poor tags, we barely knew them…

GOODBYE! inpost tags, (sort of), configuration will no longer be handled using the inpost tags, the old tag structure will be retired in favor of an anchor to place the player in your content [FLOWPLAYER], configuration of the player will now be handled by an admin menu box.

That’s quite a list you’ve got there…

(basic) Playlists support has been added, this is configurable from the admin menu for the post

Dude, where’s your config file?

The saiweb_wpfp.conf file has now been removed gasp, now reliant in internal wordpress magic for the storing of the plugin config.

Your media is a great big canvas, and you should throw all the paint you can on it

Fixed a bug with the canvas colour settings

Is that a logo in your pocket, or are you just pleased to see me?

The commercial version of flowplayer will now only be used if a license key is specified, the free version will now be used if no key is specified which has a reduced logo branding.

I Once Was Blind, But Now I See

Player embed causing issues with some navigation menus, this should be resolved with the wmode setting.

Details of how to get the preview version and install it along with screen casts of the new menus (time allowing) will be added to this post once everything is committed to subversion.

UPDATE: 15/04/2009 Got my hands on flowplayer 3.1 code is around 60% finished, went for a complete re-write.